VPS migration & selinux resentment

Earlier this year I went to renew my VPS at OVH. As well as a domain with Gandi.

The price to renew the domain with Gandi went up from $51.15 to $72.59 while it was in my cart. I asked them what that was about and they replied, paraphrasing, “ya we raised the price 🖕”. So instead I transferred my domain to porkbun.com for about $25.

At OVH it was going to cost $194 to renew my VPS for one year. I’m not sure why, but they wouldn’t offer a discount for paying for an entire year upfront as they typically do. For $82, I switched to their cheapest AMD EPYC VPS. Same specs except less memory, which I didn’t need that much of anyway.

The service on that machine collects a lot of images and serves them on a website. Since, I didn’t get around to making a good way of removing old files to free up space for new ones, migrating it to a new machine was an opportunity for housekeeping.

This was as straightforward as you might expect:

• I started the service on the new machine to begin collecting data.

• Exported data I wanted to retain from the old machine up until the timestamp of the first event on the new machine. (Timestamps were in messages from an external service that both connected to, rather than wall clock which could differ between machines.)

• Imported that historical data into the SQLite database on the new machine and copied corresponding images on disk with tar and ssh.

After testing the new site in my browser, I updated the DNS records with porkbun to the new host’s address.

Even though the DNS record had a ten minute expiry, the old server still had connections when it was shut down about a day later. Those clients reconnected to the new host pretty quickly on their own.

It all went pretty smoothly and, in the end, I got that nice chart out of it.

Fedora has a running prank of installing selinux when you install Fedora. I disabled selinux on my new VPS using setenforce.

About a month after setting up the machine, I tried rebooting it to troubleshoot a networking issue. After a few minutes, the webserver did not come up and attempts to ssh into the machine timed out. Yet, the OVH dashboard reported that the machine was running.

I ship application and system logs and metrics using vector which are then stored in loki. By some miracle of science, vector was shipping logs during this time that the machine was not accepting ssh connections.

[audit] AVC avc:  denied  { read } for  pid=818 comm="sshd" name="sshd_config" dev="sda5" ino=974214 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0
[sshd] /etc/ssh/sshd_config: Permission denied

Evidently, selinux was keeping my system secure by preventing sshd from reading its own configuration file in order to start up. At that moment I realized I had only temporarily disabled selinux with setenforce and forgotten to disable it across reboots.

I used a recovery boot thingy that OVH provides to boot my VPS. OVH emails me SSH login credentials. I log in and disable selinux persistently. Now the machine boots normally.

What really got to me, is that I put a fair bit of my own time into reducing downtime in my service. I went so far as to use sd_pid_notify_with_fds and sd_listen_fds to prevent missing messages broadcast by an external event source between service restarts. A number of times, the service upgraded without closing the socket it uses to read incoming events. Events would be buffered and read when the service comes back up; instead of being missed entirely while the connection was down.

It worked. And even though this was a side-project of no real significance to anyone, I felt happy that I had no unplanned downtime in just over two years of running it.

By the time I finished fumbling my way through the repair and the system came back online, it had been down for almost an hour. I understand that I forgot to disable selinux, and that selinux would prevent my service from starting because I didn’t write policies for it. That’s my fault, so that’s fair I guess. And nginx wouldn’t start either, as far as I know, because Fedora doesn’t ship a policy that lets it bind to port 80, whatever. But I don’t know why sshd would not be allowed to read its own configuration file at /etc/ssh/sshd_config. This makes no sense.

My first guess was, if you run with selinux disabled temporarily and you upgrade your system, then packages and files or services (like sshd) won’t be installed with selinux contexts. So when you reboot, and selinux is enabled again, it misbehaves because of missing contexts.

On the other hand, setenforce 0 doesn’t actually disable selinux, according to the man page, it just sets it to permissive mode. And permissive mode is practically the same as selinux being enabled (in enforcing mode) in the sense that it runs and evaluates denials based on policies. The key difference being it doesn’t act on those denials and just reports them. As far as I know, you are meant to be able to go from enforcing mode to permissive mode and back without issue – for the purpose of troubleshooting or something. So if being in permissive mode messes up selinux contexts, that wouldn’t make sense.

Alternatively, it’s possible I copied /etc/ssh/sshd_config over from the old machine with tar and, by default, it doesn’t copy extended attributes or selinux context stuff. If I had done this, I would expect to see it in my shell history next to where I did this for a bunch of other files. But I don’t see that. And the file creation timestamp doesn’t suggest it was copied from the old machine either.

So I have no idea how this happened. I tried to ask about it on #fedora on irc.libera.chat. To be upfront, I was pretty snarky after wasting an hour of my Sunday afternoon on this piece of shit software. I was advised; to enable sshd.service, don’t ssh in as root, and that I screwed it up because it “just works”.

I clarified I wasn’t asking whether or not I “screwed it up” but how – so I can not do that in the future. I was pretty sarcastic and, predictably, the conversation in this internet chat room was pointless as nobody learned anything and everyone was rude to everyone else.

In the end, I am surprised that sshd was not allowed to read its own configuration file. And this doesn’t inspire confidence that selinux won’t produce further surprises. How is an selinux enjoyer meant to minimize the surprises that might prevent their machine from booting?

Presumably you can set the system to permissive, restart a service, and check the audit logs for denials that would have prevented it from starting in enforcing mode. Then do that for NetworkManager, systemd-networkd, systemd-resolved, or whatever other services you can think of that are important that selinux could possibly interfere with?

But if your system is undergoing change as part of a reboot, as with dnf system-upgrade, will changes in that upgrade invite the wrath of the machine spirit? I have no idea. Do you? (If so, please elaborate in the comments section below – the engagement helps promote my content on this advertising platform!)

If Fedora shipped selinux permissive by default and allowed me to opt-in specific services to run under enforcing mode, that would be interesting to me. At least that way, I could voluntarily waste my time with it on my own terms instead of on a Sunday afternoon when I’d rather be trying to unfuck whatever ufw did to my nftables.

As far as I can tell, the security selinux provides is job security for system administrators using it to create fragile complex systems that require specialized knowledge to maintain.

In conclusion, selinux is a great addition to any Linux based operating system. My VPS was never safer or more secure than on the day that none of its services were running.