Arch Linux on Scaleway
I run Arch Linux on all of my physical computers, but I’d always opted for the ‘safe’ choice of Ubuntu for cloud servers. When I was revamping this website, I thought I’d take the opportunity to try out running Arch in the cloud.
I’d previously used Digital Ocean droplets for hosting, but they don’t support Arch Linux for new droplets. I’d heard good things about Scaleway, who do offically support Arch, so I gave it a go. The results were, ehhh, OK.
First, a little background. Arch Linux is a rolling release distro. This means that each package is updated regularly, and updates are available fairly soon after they land upstream. It also means there are no versioned releases of Arch Linux, just a collection of packages. This is distinct from say, Ubuntu, which has a new release every six months and a new LTS release every two years.
By upgrading the system early and often, issues with upgrades can largely be avoided. On the other hand, upgrading one LTS version of Ubuntu to the next involves applying two years’ worth of upgrades in one go, which can be a problem. I must admit to having maintained a few Ubuntu servers which have fallen out of the LTS period and haven’t had any updates. It’s true that upgrade to Arch systems can sometimes break things, but it’s very rare that my system has broken when I was not to blame.
Anyway, this was going to be about running Arch on Scaleway…
Upgrading the Scaleway image
When first logging into the newly created server, the first thing to do is to make sure all of the installed software is up to date. Sadly, it looks like Scaleway’s Arch Linux image was created in 2018, so it’s pretty out of date. The Arch Linux news page shows a lot of manual intervention steps between now and then!
We’re going to need to sync the mirrors and upgrade a few packages one at a time. Usually, partial upgrades like this are a bad idea but in this case we have no choice so let’s sync the mirrors:
The first big problem is that the arch keyring is out of date, so any attempts to upgrade any packages fail due to signature errors. No big deal, let’s upgrade the keyring:
# pacman -S archlinux-keyring
Uh oh. The next big problem is that the default format used to compress Arch packages changed from
.xz, and the installed version of libarchive doesn’t support this format. libarchive can’t be upgraded either, because it comes packaged in
.xz form. Luckily, the Arch maintainers foresaw this sort of thing happening so they distribute a binary of pacman which has built-in support for .xz packages. Let’s grab it now:
# curl -o pacman-static https://pkgbuild.com/~eschwartz/repo/x86_64-extracted/pacman-static # chmod +x pacman-static
Let’s upgrade libarchive, but first, we’ve got a bit of a catch-22 with the keyring that we need to resolve. Let’s disable signature verification for now, by setting
SigLevel = Never in
./pacman-static -S libarchive
In theory, we could stop using
pacman-static at this point and go back to regular pacman. However, there are a few hooks that our version of pacman doesn’t support, so we’ll keep using
pacman-static until the system is fully upgraded.
If we try upgrading the keyring again, we see a bunch of errors like this:
==> Appending keys from archlinux.gpg... ==> Locally signing trusted keys in keyring... -> Locally signing key DDB867B92AA789C165EEFA799B729B06A680C281... ==> ERROR: DDB867B92AA789C165EEFA799B729B06A680C281 could not be locally signed. -> Locally signing key 91FFE0700E80619CEB73235CA88E23E377514E00... ==> ERROR: 91FFE0700E80619CEB73235CA88E23E377514E00 could not be locally signed. -> Locally signing key AB19265E5D7D20687D303246BA1DFB64FFF979E7... ==> ERROR: AB19265E5D7D20687D303246BA1DFB64FFF979E7 could not be locally signed. -> Locally signing key 0E8B644079F599DFC1DDC3973348882F6AC6A4C2... ==> ERROR: 0E8B644079F599DFC1DDC3973348882F6AC6A4C2 could not be locally signed. -> Locally signing key D8AFDDA07A5B6EDFA7D8CCDAD6D055F927843F1C... ==> ERROR: D8AFDDA07A5B6EDFA7D8CCDAD6D055F927843F1C could not be locally signed.
It’s not entirely clear to me why this fails, but I found this manual intervention from 2015 which fixes the problem. I wonder if the Scaleway image was upgraded from an older installation and the manual intervertion steps were never run? Who knows.
So now we’ve got a working keyring, we can turn package verification back on by setting
SigLevel = Required DatabaseOptional in
/etc/pacman.conf, and upgrade the rest of the system:
Great, now we just need to reboot the server. When I did this, it took a bit longer than I expected (~ 2 minutes) to come back online and be accessible by ssh again.
There are some other quirks of this Scaleway image:
- The file permissions on directories like
/binare 775 when they should be 755. This probably doesn’t really matter as there shouldn’t be any users in the
- pacman is complaining about the package cache directory not existing, but whenever I create it disappears after pacman runs. This means rollbacks are impossible.
I’ll follow this up at some point with a blog post about how this blog is hosted.