Tooting in a portable service

Earlier this month, the drive for my root filesystem on my home server failed. After replacing the drive, Mastodon needed a bit of lovin’ to get working again. So I thought about how I could complicate it by using containers to isolate it from the host. I wrote about it in my previous post, Tooting in a container.

A couple hundred hours later, my new drive failed as it, and the former, were apparently involved in a suicide pact.

I’ve been waiting patiently for the world’s slowest retailer of computer equipment to transport some replacement drives to a nearby outlet for me to buy & pick up. I got them the other day and set everything up again, this time with raid1 mirroring in LVM. I also rebuilt it with some parts from an old employer and the machine now boasts one gigabit ethernet. 2008, here I come!

While I was waiting for the replacement disks, I thought about how to redo my Mastodon install. I considered using LXD because it’s fun to use and fairly well documented and it makes it easy to map user and groups between a host and its unprivileged containers.

A concern was that it seemed geared towards running machines as containers rather than single services. While you can configure a container’s init process to be any program, doing so is not at all typical. Moreover, I wanted something that worked nicely with systemd in terms of logging and service management. I’m still not quite sure if that is difficult with LXD. I haven’t found an obvious solution like integration with machinectl or anything.

I stumbled upon systemd’s portable services feature/portablectl and it seems pretty appropriate. It’s not as comprehensive as something like LXD, but it gives me isolated services that I can interact with through systemctl and journalctl as if they were any other service. Which is really nice. And making multiple systemd services available from one portable service container thingy is handy for Mastodon since it has three services, mastodon-web, mastodon-streaming, mastodon-sidekiq. Getting that to work was a problem with my systemd-nspawn approach.

Previously, I had to separate out the service’s home directory (which is used for runtime, configuration, and state…) and bind mount that over the read-only root filesystem in the container. This was mainly a result of the three services detail. I don’t think this is necessary with portable services.[1] But I’ve included it in the configuration below because that’s the way that it is.

The steps for building the portable service that I’m using is nearly the same as my previous post. Except:

• The portable service rootfs is at /var/lib/portables/mastodon, instead of /var/lib/machines/mastodon.

• The service files for Mastodon, found in the sources under dist, need to be copied to a place (in the “container”) where systemd unit files are normally found. I put mine in /usr/local/lib/systemd/system.[2]

• After attaching the portable service, portablectl attach mastodon, supplement each service with the following overwrite. For me, these went to:

  • /etc/systemd/system.attached/mastodon-web.service.d/30-memes.conf

  • /etc/systemd/system.attached/mastodon-streaming.service.d/30-memes.conf

  • /etc/systemd/system.attached/mastodon-sidekiq.service.d/30-memes.conf

  You probably want to link them if you want them all the same and maybe change them.

  Below is the contents of this overwrite file with some comments.

[Service]
# This allows the service to communicate to
# postgres over a UNIX socket.
BindPaths=/var/run/postgresql
# Without this it seems like the service won't be
# able to access /home/mastodon. systemd.exec(5)
# suggests that, by setting this to tmpfs, we can
# make our service home directory available to the
# service with BindPaths.
ProtectHome=tmpfs
BindPaths=/home/mastodon
# My systemctl show-environment doesn't have
# /bin in PATH. But the portable services will
# expect it for /bin/bash and maybe other things.
Environment=PATH=/usr/local/bin:/usr/bin:/bin
# Needed for some ruby gem "blurhash" to use ffi...?
MemoryDenyWriteExecute=no

sqwishy@banana ~> sudo systemctl status -n0 mastodon-{web,sidekiq,streaming}
● mastodon-web.service - mastodon-web
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-web.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-web.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:19 PDT; 9min ago
 Main PID: 17531 (bundle)
    Tasks: 29 (limit: 4649)
   Memory: 410.3M
   CGroup: /system.slice/mastodon-web.service
           ├─17531 puma 3.12.1 (tcp://0.0.0.0:3000) [live]
           ├─17607 puma: cluster worker 0: 17531 [live]
           └─17609 puma: cluster worker 1: 17531 [live]

● mastodon-sidekiq.service - mastodon-sidekiq
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-sidekiq.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-sidekiq.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:21 PDT; 9min ago
 Main PID: 17560 (bundle)
    Tasks: 12 (limit: 4649)
   Memory: 205.2M
   CGroup: /system.slice/mastodon-sidekiq.service
           └─17560 sidekiq 5.2.7 live [0 of 5 busy]

● mastodon-streaming.service - mastodon-streaming
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-streaming.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-streaming.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:19 PDT; 9min ago
 Main PID: 17526 (node)
    Tasks: 18 (limit: 4649)
   Memory: 50.6M
   CGroup: /system.slice/mastodon-streaming.service
           ├─17526 /usr/bin/node ./streaming
           └─17551 /usr/bin/node /home/mastodon/live/streaming