Arch Linux on Scaleway

I run Arch Linux on all of my physical computers, but I’d always opted for the ‘safe’ choice of Ubuntu for cloud servers. When I was revamping this website, I thought I’d take the opportunity to try out running Arch in the cloud.

I’d previously used Digital Ocean droplets for hosting, but they don’t support Arch Linux for new droplets. I’d heard good things about Scaleway, who do offically support Arch, so I gave it a go. The results were, ehhh, OK.

Rolling release

First, a little background. Arch Linux is a rolling release distro. This means that each package is updated regularly, and updates are available fairly soon after they land upstream. It also means there are no versioned releases of Arch Linux, just a collection of packages. This is distinct from say, Ubuntu, which has a new release every six months and a new LTS release every two years.

By upgrading the system early and often, issues with upgrades can largely be avoided. On the other hand, upgrading one LTS version of Ubuntu to the next involves applying two years’ worth of upgrades in one go, which can be a problem. I must admit to having maintained a few Ubuntu servers which have fallen out of the LTS period and haven’t had any updates. It’s true that upgrade to Arch systems can sometimes break things, but it’s very rare that my system has broken when I was not to blame.

Anyway, this was going to be about running Arch on Scaleway…

Upgrading the Scaleway image

When first logging into the newly created server, the first thing to do is to make sure all of the installed software is up to date. Sadly, it looks like Scaleway’s Arch Linux image was created in 2018, so it’s pretty out of date. The Arch Linux news page shows a lot of manual intervention steps between now and then!

We’re going to need to sync the mirrors and upgrade a few packages one at a time. Usually, partial upgrades like this are a bad idea but in this case we have no choice so let’s sync the mirrors:

pacman -Sy

The first big problem is that the arch keyring is out of date, so any attempts to upgrade any packages fail due to signature errors. No big deal, let’s upgrade the keyring:

# pacman -S archlinux-keyring

Uh oh. The next big problem is that the default format used to compress Arch packages changed from .gz to .xz, and the installed version of libarchive doesn’t support this format. libarchive can’t be upgraded either, because it comes packaged in .xz form. Luckily, the Arch maintainers foresaw this sort of thing happening so they distribute a binary of pacman which has built-in support for .xz packages. Let’s grab it now:

# curl -o pacman-static https://pkgbuild.com/~eschwartz/repo/x86_64-extracted/pacman-static
# chmod +x pacman-static

Let’s upgrade libarchive, but first, we’ve got a bit of a catch-22 with the keyring that we need to resolve. Let’s disable signature verification for now, by setting SigLevel = Never in /etc/pacman.conf.

./pacman-static -S libarchive

In theory, we could stop using pacman-static at this point and go back to regular pacman. However, there are a few hooks that our version of pacman doesn’t support, so we’ll keep using pacman-static until the system is fully upgraded.

If we try upgrading the keyring again, we see a bunch of errors like this:

==> Appending keys from archlinux.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key DDB867B92AA789C165EEFA799B729B06A680C281...
==> ERROR: DDB867B92AA789C165EEFA799B729B06A680C281 could not be locally signed.
  -> Locally signing key 91FFE0700E80619CEB73235CA88E23E377514E00...
==> ERROR: 91FFE0700E80619CEB73235CA88E23E377514E00 could not be locally signed.
  -> Locally signing key AB19265E5D7D20687D303246BA1DFB64FFF979E7...
==> ERROR: AB19265E5D7D20687D303246BA1DFB64FFF979E7 could not be locally signed.
  -> Locally signing key 0E8B644079F599DFC1DDC3973348882F6AC6A4C2...
==> ERROR: 0E8B644079F599DFC1DDC3973348882F6AC6A4C2 could not be locally signed.
  -> Locally signing key D8AFDDA07A5B6EDFA7D8CCDAD6D055F927843F1C...
==> ERROR: D8AFDDA07A5B6EDFA7D8CCDAD6D055F927843F1C could not be locally signed.

It’s not entirely clear to me why this fails, but I found this manual intervention from 2015 which fixes the problem. I wonder if the Scaleway image was upgraded from an older installation and the manual intervertion steps were never run? Who knows.

So now we’ve got a working keyring, we can turn package verification back on by setting SigLevel = Required DatabaseOptional in /etc/pacman.conf, and upgrade the rest of the system:

./pacman-static -Syu

Great, now we just need to reboot the server. When I did this, it took a bit longer than I expected (~ 2 minutes) to come back online and be accessible by ssh again.

Next steps

There are some other quirks of this Scaleway image:

I’ll follow this up at some point with a blog post about how this blog is hosted.

IPv6

It turns out there’s another issue with IPv6 connectivity. By default, the Scaleway image expects the network interface to be called eth0, but on the Arch image it’s called ens2. This means the init scripts fail to assign an IPv6 address to the interface. There’s a workaround on this GitHub issue which I’ve wrapped up in a systemd service so that it runs automatically when the system boots.

Here is the script which lives at /root/ipv6-setup.sh:

#!/bin/bash
set -euo pipefail

IPV6_NETMASK=$(scw-metadata IPV6_NETMASK)
IPV6_GATEWAY=$(scw-metadata IPV6_GATEWAY)
IPV6_ADDRESS=$(scw-metadata IPV6_ADDRESS)
ip addr add ${IPV6_ADDRESS}/${IPV6_NETMASK} dev ens2
ip -6 r add default via ${IPV6_GATEWAY} dev ens2

and the systemd service:

[Unit]
Description = Scaleway IPv6 setup

[Service]
Type = oneshot
ExecStart = /root/ipv6-setup.sh

[Install]
WantedBy = default.target

Moving to Linode

I recently found out that Linode (referral link) also offer Arch Linux hosting, and so far it seems much better than Scaleway’s offering. The image is up-to-date so none of the above setup tasks are necessary, and it can be hosted on a much smaller machine which means it’s substantially cheaper. It basically just worked out of the box!