Archive Caching For LXD: Update

Preface

Several months ago I wrote about caching the APT packages used on my computer/ within LXD. I'm pleased to announce that it's been a great success. This post is a follow-up which details some of the things I've discovered/ had to resolve.

We'll start by considering a couple of problems and then looking at some interesting metrics.

Problems

Start-up Ordering

As well as my LXD containers/ virtual machines proxying to the apt-cacher-ng instance running on the host computer, my computer does too.

+-------------------------------------------------------+
| HOST ---------------------------------+               |
|                                       |               |
|                                       V               |
| +-------------------------+     +----------------+    |
| | LXD                     |     | apt-cacher-ng  |    |
| |                         |     |                |    |
| | +-----------+           |     |                |    |
| | | Instance1 |---------------->|                |    |
| | +-----------+           |     |                |    |
| |                         |     |                |    |
| | +-----------+           |     |                |    |
| | | Instance2 |---------------->|                |    |
| | +-----------+           |     |                |    |
| |                         |     |                |    |
| | +-----------+           |     |                |    |
| | | Instance3 |---------------->|                |    |
| | +-----------+           |     |                |    |
| |                         |     |                |    |
| | +-----------+           |     |                |    |
| | | Instance4 |---------------->|                |    |
| | +-----------+           |     |                |    |
| |                         |     |                |    |
| +-------------------------+     +----------------+    |
|                                                       |
|                                                       |
+-------------------------------------------------------+

If apt-cacher-ng starts up beforethe the network interface lxdbr0, then the apt-cacher-ng will fail to start, as one would expect. This means that when I boot my computer and log in, I then have to run systemctl restart apt-cacher-ng at some point in order to get things back on track.

I knew this would be an issue when I decided to do this. Why did I not install apt-cacher-ng in a container and mitigate the issue up-front? If LXD or the container were to go down due to maintenance, a bug or a corruption, I would not be able to update my host pacakges without intervention.

A few options lay before me, though they all bear obvious similarities.

Create a new network

I could create a new NIC which is not managed by LXD, then bind apt-cacher-ng to that. This NIC would then have to be added to existing and new instances.

Use a host managed NIC

A second option would be to instead have the containers contact the host on a different network, e.g. the IPv4 address assigned to my physical NIC by the router in my home. The IP address would of course have to be static and firewall rules established to forward the traffic between networks.

Create an overlay network

A third solution might be to do something a bit radical which would involve extending the first option. I have toyed with the idea of setting up a secure network for my containers to transact information. I could set up a Wireguard network with the host acting as the Wireguard server!

This would require for each new instance:

• a new private key be created
• the server configuration be updated with the new instance
• the peer config being generated
• the wireguard configuration for all other peers be updated for peer-to-peer communication

This is a lot of bother, usually requiring third-party software to manage the scaling issues.

Move the cache into LXD

I could move the cache into a LXD container, and then bypass it in the APT configuration when necessary.

Do nothing

Doing nothing is an option. It's not a big hassle.

Choices

I think the simplest approach would be to do nothing, continuing with the current configuration. I have a script which I execute when I boot my computer, so it's a very small overhead to manage and it's obvious when I've forgotten to run the step as APT cannot connect to the cache.

Cache corruption

Today I updated my package list as normal however I was met with a error:

$ sudo apt update

Hit:1 http://gb.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:3 http://gb.archive.ubuntu.com/ubuntu noble-backports InRelease
Get:4 http://gb.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Hit:5 https://esm.ubuntu.com/apps/ubuntu noble-apps-security InRelease
Err:4 http://gb.archive.ubuntu.com/ubuntu noble-updates InRelease
  The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <[email protected]>
Hit:6 https://esm.ubuntu.com/apps/ubuntu noble-apps-updates InRelease
Hit:7 https://esm.ubuntu.com/infra/ubuntu noble-infra-security InRelease
Hit:8 https://esm.ubuntu.com/infra/ubuntu noble-infra-updates InRelease
Fetched 126 kB in 1s (193 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://gb.archive.ubuntu.com/ubuntu noble-updates InRelease: The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <[email protected]>
W: Failed to fetch http://gb.archive.ubuntu.com/ubuntu/dists/noble-updates/InRelease  The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <[email protected]>
W: Some index files failed to download. They have been ignored, or old ones used instead.

This is the pertinent line:

The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) [email protected]

I was surprised, in a good way, to discover the error was with apt-cacher-ng. I initially expected there to be some issue with the upstream archive or an unexpected GPG Key rotation. The solution was straight-forward.

• Go to http://your-apt-cacher-web-interface/acng-report.html
• Run "count data"
• In the Expiration section, click the "Start Scan" and/or "Expiration"
• Once the scan has finished select the "check all" checkbox at the bottom.
• Delete the selected files.

The problem went away after this.

HTTP vs HTTPS

It was pointed out to me today that it's surprising I transport my APT packages from the remote repo gb.archive.ubuntu.com using HTTP and not HTTPS. I do this because the InRelease file is signed with a GPG key which I can validate. I know if the file is correct or not. Inside the InRelease file is a list of the packages and their hashes. I can therefore have confidence that the packages I requested have not been tampered with.

The other aspect is privacy. When using HTTP my ISP, really anyone routing my packets, can see which packages I've requested. There is an argument to make that upgrading the host-to-remote-archive portion to HTTPS is a sensible choice and the instance-to-host portion remain HTTP. I think I will look into this.

Interesting things

This data is taken after a restart. As you can see, the data served is approximately 2.34 times that of data fetched.

One week of stats

History

Cache efficiency

Roundup

That's it for this week. apt-cacher-ng is still going strong and I have some improvements to effect in the near future, but nothing urgent.