Tooting in a portable service

Saturday, 27 July 2019

Saturday, July 27, 2019

Running Mastodon services inside of systemd portable service.


technical, systemd, linux

Earlier this month, the drive for my root filesystem on my home server failed. After replacing the drive, Mastodon needed a bit of lovin’ to get working again. So I thought about how I could complicate it by using containers to isolate it from the host. I wrote about it in my previous post, Tooting in a container.

A couple hundred hours later, my new drive failed as it, and the former, were apparently involved in a suicide pact.

I’ve been waiting patiently for the world’s slowest retailer of computer equipment to transport some replacement drives to a nearby outlet for me to buy & pick up. I got them the other day and set everything up again, this time with raid1 mirroring in LVM. I also rebuilt it with some parts from an old employer and the machine now boasts one gigabit ethernet. 2008, here I come!

While I was waiting for the replacement disks, I thought about how to redo my Mastodon install. I considered using LXD because it’s fun to use and fairly well documented and it makes it easy to map user and groups between a host and its unprivileged containers.

I might still use LXD to replace the Docker containers running sonarr and radarr. Since those are packaged for Arch Linux, they might be easier to keep up-to-date.

A concern was that it seemed geared towards running machines as containers rather than single services. While you can configure a container’s init process to be any program, doing so is not at all typical. Moreover, I wanted something that worked nicely with systemd in terms of logging and service management. I’m still not quite sure if that is difficult with LXD. I haven’t found an obvious solution like integration with machinectl or anything.

I stumbled upon systemd’s portable services feature/portablectl and it seems pretty appropriate. It’s not as comprehensive as something like LXD, but it gives me isolated services that I can interact with through systemctl and journalctl as if they were any other service. Which is really nice. And making multiple systemd services available from one portable service container thingy is handy for Mastodon since it has three services, mastodon-web, mastodon-streaming, mastodon-sidekiq. Getting that to work was a problem with my systemd-nspawn approach.

Previously, I had to separate out the service’s home directory (which is used for runtime, configuration, and state…) and bind mount that over the read-only root filesystem in the container. This was mainly a result of the three services detail. I don’t think this is necessary with portable services.1 But I’ve included it in the configuration below because that’s the way that it is.


In an ideal world you probably would have application configuration & state made available from the host to the portable service through a bind mount or something, so that updating the service can be done simply by replacing the rootfs with one that includes the new runtime. With Mastodon, however, the runtime, configuration, and state and everything are not self-contained.

See also systemd.exec(5) and options like StateDirectory and ConfigurationDirectory (and others). They seem to be a nice mechanism for this.

The steps for building the portable service that I’m using is nearly the same as my previous post. Except:

  • The portable service rootfs is at /var/lib/portables/mastodon, instead of /var/lib/machines/mastodon.

  • The service files for Mastodon, found in the sources under dist, need to be copied to a place (in the “container”) where systemd unit files are normally found. I put mine in /usr/local/lib/systemd/system.2

  • After attaching the portable service, portablectl attach mastodon, supplement each service with the following overwrite. For me, these went to:

    • /etc/systemd/system.attached/mastodon-web.service.d/30-memes.conf

    • /etc/systemd/system.attached/mastodon-streaming.service.d/30-memes.conf

    • /etc/systemd/system.attached/mastodon-sidekiq.service.d/30-memes.conf

    You probably want to link them if you want them all the same and maybe change them.

    Below is the contents of this overwrite file with some comments.


Maybe they should be in /usr instead of /usr/local but I always feel bad whenever I think about doing that because I’m not a package maintainer and /usr is for important packages from important maintaines.


# This allows the service to communicate to
# postgres over a UNIX socket.
# Without this it seems like the service won't be
# able to access /home/mastodon. systemd.exec(5)
# suggests that, by setting this to tmpfs, we can
# make our service home directory available to the
# service with BindPaths.
# My systemctl show-environment doesn't have
# /bin in PATH. But the portable services will
# expect it for /bin/bash and maybe other things.
# Needed for some ruby gem "blurhash" to use ffi...?

You may need to suit this to your needs if you want to use it. Some things, like the Environment and MemoryDenyWriteExecute options, probably ought to be part of the service file that is published from the container; since they have to do with Mastodon’s ability to function as a portable service and less to do with the host configuration. But I tried leaving Mastodon’s service files unmodified so I can overwrite them if they change upstream without having to keep track of my modifications.

sqwishy@banana ~> sudo systemctl status -n0 mastodon-{web,sidekiq,streaming}
● mastodon-web.service - mastodon-web
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-web.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-web.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:19 PDT; 9min ago
 Main PID: 17531 (bundle)
    Tasks: 29 (limit: 4649)
   Memory: 410.3M
   CGroup: /system.slice/mastodon-web.service
           ├─17531 puma 3.12.1 (tcp:// [live]
           ├─17607 puma: cluster worker 0: 17531 [live]
           └─17609 puma: cluster worker 1: 17531 [live]

● mastodon-sidekiq.service - mastodon-sidekiq
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-sidekiq.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-sidekiq.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:21 PDT; 9min ago
 Main PID: 17560 (bundle)
    Tasks: 12 (limit: 4649)
   Memory: 205.2M
   CGroup: /system.slice/mastodon-sidekiq.service
           └─17560 sidekiq 5.2.7 live [0 of 5 busy]

● mastodon-streaming.service - mastodon-streaming
   Loaded: loaded (/var/lib/portables/mastodon/usr/local/lib/systemd/system/mastodon-streaming.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system.attached/mastodon-streaming.service.d
           └─10-profile.conf, 20-portable.conf, 30-memes.conf
   Active: active (running) since Sat 2019-07-27 16:51:19 PDT; 9min ago
 Main PID: 17526 (node)
    Tasks: 18 (limit: 4649)
   Memory: 50.6M
   CGroup: /system.slice/mastodon-streaming.service
           ├─17526 /usr/bin/node ./streaming
           └─17551 /usr/bin/node /home/mastodon/live/streaming

Lennart Poettering also has a walkthrough for portable services on his blag; check it out if you’re interested in looking at some proper instructions and explanation.