Using nginx to reverse proxy the internet

Wed Sep 22 '21

nginx is a reverse proxy; often used to proxy incoming connections or requests at edge of your network on their way from the internet to some service in your internal network. One reason to do this is to terminate TLS at your edge with nginx instead of at each service in your network.

However, we can also use the ngx_http_proxy_module to initiate TLS at nginx on its way from our internal network to the internet.

Why?

Because we want to open a WebSocket and keep it open between program restarts.

Imagine we have a service that initiates a long-lived connection and it is not great when the socket is closed for some reasons:

The socket allows our service to listen to events broadcast by the peer and events are missed when the connection is closed until it is reestablished.
Establishing a new connection requires starting TLS session that takes several hundred milliseconds to complete. (Events are missed during this.)

Our proper goal is to not lose events. That is, never have no connections open.

We could approach this in other ways, like with redundancy or by never restarting, but that’s not what I did. So, in this scenario, we enjoy restarting things to update them and we don’t want redundant machines or networks because computers are bad.

How?

In a previous post about SCM_RIGHTS I illustrated duplicating a file descriptor to another process so that a connection can outlive the process that created it.

Similarly, systemd’s sd_pid_notify_with_fds & sd_listen_fds function can be used by a systemd service (including a program started with systemd-run) to keep a file descriptor open across a service restart.

When our service establishes this long-lived connection, we share the socket with systemd so that we can get it back later after we restart.

For this to work reasonably, the program needs enough information about the open socket to use it correctly when starting back up.

As it happens, after a clean exit, there is no state in our application worth retaining having to do with either the service or the WebSocket protocol itself. The only state that complicates things is the TLS connection that our WebSocket runs over. To solve that, we avoid initiating the TLS connection in our process and, instead, delegate that to nginx.

proxy_pass https://…

Normally, our program opens a TLS connection to example.com. Instead, we want a cleartext connection to nginx and a TLS connection from nginx to example.com through which the request to open a WebSocket is proxied.

I run nginx on the same machine as my service, so I can connect to it at the hostname localhost. But we’ll also include the proxied hostname as a subdomain so that nginx knows where to proxy our request to – so example.com.localhost for instance.

The nginx configuration looks like:

server {
    listen       127.0.0.1:80;
    listen           [::1]:80;
    server_name  ~^(?<meme>.+)\.localhost$;
    resolver     127.0.0.53;

    location / {
        proxy_pass            https://$meme;
        proxy_ssl_server_name on;
        proxy_ssl_protocols   TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    }
}

That’s mostly it for normal HTTP requests – it should let you fetch http://www.wikipedia.org.localhost – but it won’t upgrade WebSockets yet. And there a few things to take note of:

You may need or want to use something entirely different for your resolver.
proxy_ssl_server_name is important for connecting to servers that require the server name to be included with TLS (SNI). I think this is most CDNs & DDoS mitigation like Cloudflare (a lot of the web…)
proxy_ssl_protocols is, at the time of writing, the default list of supported protocols with the addition of TLSv1.3. Otherwise, nginx cannot connect to servers that require TLSv1.3.

Finally, the nginx documentation on WebSocket proxying shows how we can extend the configuration above to support for upgrading connections to WebSockets.

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen       127.0.0.1:80;
    listen           [::1]:80;
    server_name  ~^(?<meme>.+)\.localhost$;
    resolver     127.0.0.53;

    location / {
        proxy_pass            https://$meme;
        proxy_set_header      Connection $connection_upgrade;
        proxy_set_header      Upgrade    $http_upgrade;
        proxy_ssl_server_name on;
        proxy_ssl_protocols   TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    }
}

Be aware of these caveats:

This isn’t a full web proxy that will rewrite hyperlinks and resource URLs in HTML documents.
Restarting nginx will bring down the connection.
It is possible to write IPv4 addresses into these domain names. Like 127.0.0.1.localhost.
The X-Accel- headers can be included in responses by upstream servers to get nginx to behave in interesting ways.

For example, X-Accel-Redirect performs an internal redirect prompting nginx to process a new URI and even match locations marked internal in nginx configurations. As far as I can tell, redirects in this way are evaluated within the same server block; so the above configuration is relatively benign.

Nevertheless, it’s probably a good idea to use proxy_ignore_headers to disable this behaviour.
```
proxy_ignore_headers "X-Accel-Redirect"
                     "X-Accel-Expires"
                     "X-Accel-Limit-Rate"
                     "X-Accel-Buffering"
                     "X-Accel-Charset";
```

So that’s it.

Now you know how to configure nginx, a reverse proxy typically used to terminate TLS for incoming connections from the internet to your internal network, as a reverse reverse proxy to de-terminate TLS for connections from our internal network to the internet.

And by offloading TLS to nginx and sharing the socket between processes with systemd or SCM_RIGHTS, a program can exit, restart, and resume without dropping the WebSocket.

Remember, if you want to see more blog posts like this one – that are made by copying documentation from places and talking about it – be sure to subscribe to my follows on OnlyFans at http://gofundme.subscribestar.librapay.com.localhost/patreon