The download took all night. It ended a little after 7 this morning, which pegs it at about 14 hours to download 65GB. My internet connection is a 10Mbit cable line, so there’s a bench mark of sorts.
When I embarked on this little project, my goal was to come up with a way to force apt-cacher-ng to update all of its cache on a schedule. For the native machine/architecture all that is required is to schedule an apt-get update which can be done by enabling it in the apt config files. But I couldn’t figure out how to force the same thing for a foreign architecture (ie- for another computer running i386 based stuff) so the files would already exist on the network.
The more I thought about it, I realized there wasn’t any bandwidth savings to be had because, though both machines run debian/testing, they both required different sets of package files because of the different architectures. Therefore, anything I was doing was going to be for creating a more convenient update process by having the files already present on the local network. The apt-mirror package isn’t ideal since it downloads an entire repository for a debian branch rather than caching and updating just the packages I need. However, it’s better than a full-blown mirror and, once it’s initially setup, the downloading is merely a matter of updating the local repository. In the end, it seemed like the best compromise.
For documentation purposes, here’s the mirror.list config file:
deb http://ftp.us.debian.org/debian/ testing main contrib non-free
deb-src http://ftp.us.debian.org/debian/ testing main contrib non-free
deb http://security.us.debian.org/ testing/updates main
deb-src http://security.us.debian.org/ testing/updates main
deb http://deb.opera.com/opera/ testing non-free
deb-amd64 http://ftp.us.debian.org/debian/ testing main contrib non-free
deb-amd64 http://security.us.debian.org/ testing/updates main
deb-amd64 http://deb.opera.com/opera/ testing non-free
The deb-amd64 is how the different architectures to mirror are specified. That sort of nugget isn’t documented or explained at all. I found help from this guy and also spent a little time looking at the source code to verify. For good measure, here’s another pretty good explanation.
With the config file squared away, here’s the magic command to run (as root):
su apt-mirror -c apt-mirror
I didn’t make any other changes to mirror.list so the directory tree is built under /var/spool/apt-mirror/mirror. When the command completed, I got a bunch of, apparently, harmless error messages. It’ll also specify if there is any cleanup to do and lists the command to run to achieve it.
From here, I had to modify my sources.list to point at localhost instead of the url locations. For instance, the line deb http://ftp.us.debian.org/debian/ testing main contrib non-free became deb http://localhost/debian/ testing main contrib non-free. I already had apache running so I just had to add a symbolic link from my document root to the directory tree. Like so:
ln -s /var/spool/apt-mirror/mirror/ftp.us.debian.org/debian/ /var/www/debian
I used the same technique for the opera and security repositories.
From here, I tested it out and aptitude found everything to its liking. So my server is good to go. I’ll just have to point my EEE’s sources.list at the server for the next step. Hopefully no issues there.
The last config step I took was to setup a cron job that runs apt-mirror daily. Fortunately, that was taken care of during the install and I just had to enable it in the /etc/cron.d/apt-mirror file. I changed the time to something in the early morning when the system is unlikely to be in use. I also adjusted my cron.daily job appropriately to minimize the opportunity for conflicts.
Now, the real test will be using it over an extended period of time. I’ll also remove apt-cacher-ng since I don’t need it anymore.
4 replies on “apt-mirror Update”
Right, you cannot “update the cache” because that refers to files in a frozen state. What you really want is actually mirroring the whole server in the scope of a certain arch, including the newer version of the cached packages.
And after all, you can do that kind of thing with a recent version of apt-cacher-ng. Example:
PrecacheFor: debrep/dists/unstable/*/binary-amd64/Packages*
PrecacheFor: debrep/dists/unstable/*/binary-i386/Packages*
And then trigger the process from the maintenance web UI or call that URL with wget in a cron job.
I’m not sure I understand the distinction you’re making. My (admittedly basic) understanding of how apt-cacher-ng works is that it sits between the repo and my computer and will serve up the more up-to-date version of a package, while also “caching” it locally for the next time an update is performed. Further, the only packages dealt with are the ones on my computer, rather than the entire repo.
Setting that aside, if I use apt-cacher-ng as you describe, do I still maintain the advantages of apt-cacher-ng? or will it still be downloading the entire repo?
Thanks for stopping by and offering the tip.
That’s a new feature available since December. Let’s call it precaching. You give it a list of index files (basically the same things you specified in mirror.list but expanded to the real paths used by apt) and it mirrors everything found in them on the server and keeps it in the cache. It’s accessible as usual, just like if a client had downloaded it before.
What I am going add RSN is an optional filter to this precaching function so it only downloads packages if some (older) version of them was in cache before.
By the way, your WordPress config has some problem, it mangled all paths with asterisks in my example above.
Actually, that wasn’t WordPress, it was my blogclient. It supports markdown and when I updated the comment it interpreted the asterisks as markdown syntax and added some HTML. I think I was able to recapture your original path string.