SCM_RIGHTS

Tuesday, 21 May 2019

Date
May 21, 2019
summary

Duplicating file descriptors between processes across a UNIX socket.

tags

linux, technical

SCM_RIGHTS is a feature that allow file descriptors to be duplicated across UNIX domain sockets.1 This allows you to pass access to resources to other processes.

1

Cloudflare’s blag has a nice post explaining it a bit better and how they used it to support TLS 1.3.

I wanted to try using this to do some sort of process reloading without dropping connections and to get experience with Rust & mio, a library for selecting on file descriptors (using epoll in Linux).

The source code is available at github.com/sqwishy/scmrights-thingy.

Flow

My program simultaneously listens on a UNIX socket and connects over TCP to a “chat” server (nc -l --chat). Data from the TCP stream is simply copied to stdout. This is the connection we want to hand off and keep open when restarting.

The UNIX socket is used to convey some application-specific control messages (since we might want to extend this protocol to convey other sorts of information in the future) and to pass the file descriptors. The exchange is roughly:

  1. Program A accepts a UNIX connection from program B, which is just starting up.

  2. B sends a message requesting the file descriptors. A reads the message.

  3. A calls sendmsg to send an SCM_RIGHTS UNIX domain socket control message. B reads the message with recvmsg.

  4. B sends a message to A telling it to quit. A reads the message and quits.

  5. After A quits, B’s UNIX connection will be closed so B figures that the exchange went well. It starts up normally but uses the file descriptor acquired to initialize the chat client.

Since we don’t want this little dance to be happening more than once at any given moment, only one connection is accepted from the UNIX socket at a time.

Implementation

I made an example and it put in asciinema.

A thumbnail and link to a terminal session demonstrating a connection being handed off between processes.
In this example we start a chat server, two clients, and show the scmrights-thingy program restarting without the connection dropping. Finally, a sequence of numbers are sent quickly and the exchange happens without information being lost–mainly because it’s not buffered or anything by example program.

Why?

My scenario is pretty naive. In more realistic applications, I’d expect more information than just a handle to a system resource would need to be shared; like buffers or state about the protocol. But these could be serialized and sent across the UNIX socket as a normal packet.

Bonus meme: systemd can do this for you using sd_pid_notify_with_fds. See the documentation on sd_notify, which also suggests sharing application state using memfd_create

So you might use this as an approach to “hot code reloading”. Maybe you want to run a new process to run a program under a new configuration, upgrade it to run a new version, switch between debug & release binaries, or to show off at a dinner party.

Scribbled in crayon on Tuesday, 21 May 2019

Tagged with linux and technical.

Categorized as “