Tooting in a portable service

Sat Jul 27 '19

Earlier this month, the drive for my root filesystem on my home server failed. After replacing the drive, Mastodon needed a bit of lovin’ to get working again. So I thought about how I could complicate it by using containers to isolate it from the host. I wrote about it in my previous post, Tooting in a container.

A couple hundred hours later, my new drive failed as it, and the former, were apparently involved in a suicide pact.

I’ve been waiting patiently for the world’s slowest retailer of computer equipment to transport some replacement drives to a nearby outlet for me to buy & pick up. I got them the other day and set everything up again, this time with raid1 mirroring in LVM. I also rebuilt it with some parts from an old employer and the machine now boasts one gigabit ethernet. 2008, here I come!

While I was waiting for the replacement disks, I thought about how to redo my Mastodon install. I considered using LXD because it’s fun to use and fairly well documented and it makes it easy to map user and groups between a host and its unprivileged containers.

A concern was that it seemed geared towards running machines as containers rather than single services. While you can configure a container’s init process to be any program, doing so is not at all typical. Moreover, I wanted something that worked nicely with systemd in terms of logging and service management. I’m still not quite sure if that is difficult with LXD. I haven’t found an obvious solution like integration with machinectl or anything.

I stumbled upon systemd’s portable services feature/portablectl and it seems pretty appropriate. It’s not as comprehensive as something like LXD, but it gives me isolated services that I can interact with through systemctl and journalctl as if they were any other service. Which is really nice. And making multiple systemd services available from one portable service container thingy is handy for Mastodon since it has three services, mastodon-web, mastodon-streaming, mastodon-sidekiq. Getting that to work was a problem with my systemd-nspawn approach.

Previously, I had to separate out the service’s home directory (which is used for runtime, configuration, and state…) and bind mount that over the read-only root filesystem in the container. This was mainly a result of the three services detail. I don’t think this is necessary with portable services.[1] But I’ve included it in the configuration below because that’s the way that it is.

The steps for building the portable service that I’m using is nearly the same as my previous post. Except:


# This allows the service to communicate to
# postgres over a UNIX socket.
# Without this it seems like the service won't be
# able to access /home/mastodon. systemd.exec(5)
# suggests that, by setting this to tmpfs, we can
# make our service home directory available to the
# service with BindPaths.
# My systemctl show-environment doesn't have
# /bin in PATH. But the portable services will
# expect it for /bin/bash and maybe other things.
# Needed for some ruby gem "blurhash" to use ffi...?

You may need to suit this to your needs if you want to use it. Some things, like the Environment and MemoryDenyWriteExecute options, probably ought to be part of the service file that is published from the container; since they have to do with Mastodon’s ability to function as a portable service and less to do with the host configuration. But I tried leaving Mastodon’s service files unmodified so I can overwrite them if they change upstream without having to keep track of my modifications.

sqwishy@banana ~> sudo systemctl status -n0 mastodon-{web,sidekiq,streaming}
● mastodon-web.service - mastodon-web
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-web.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-web.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:19 PDT; 9min ago
 Main PID: 17531 (bundle)
    Tasks: 29 (limit: 4649)
   Memory: 410.3M
   CGroup: /system.slice/mastodon-web.service
           ├─17531 puma 3.12.1 (tcp:// [live]
           ├─17607 puma: cluster worker 0: 17531 [live]
           └─17609 puma: cluster worker 1: 17531 [live]

● mastodon-sidekiq.service - mastodon-sidekiq
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-sidekiq.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-sidekiq.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:21 PDT; 9min ago
 Main PID: 17560 (bundle)
    Tasks: 12 (limit: 4649)
   Memory: 205.2M
   CGroup: /system.slice/mastodon-sidekiq.service
           └─17560 sidekiq 5.2.7 live [0 of 5 busy]

● mastodon-streaming.service - mastodon-streaming
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-streaming.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-streaming.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:19 PDT; 9min ago
 Main PID: 17526 (node)
    Tasks: 18 (limit: 4649)
   Memory: 50.6M
   CGroup: /system.slice/mastodon-streaming.service
           ├─17526 /usr/bin/node ./streaming
           └─17551 /usr/bin/node /home/mastodon/live/streaming

Lennart Poettering also has a walkthrough for portable services on his blag; check it out if you’re interested in looking at some proper instructions and explanation.