Server 102: Access Everywhere

This article follows on from our project on taking better care of our digital selves. As such, I will basically assume you've followed the previous steps to get here. If not, do that.

So at this point, we have a server, ready to receive connections from other computers and provide access to our external harddrive with all our files. We also have our server backing up to another external harddrive regularly and automatically. Finally, we have an offsite backup in the form of cloud storage that we want to integrate.

One of the main issues I encountered setting my server up is that it was really difficult to figure out what the IP address was. There could be all sorts of reasons why you have trouble with the IP address of your server. Your ISP might change the IP address every so often. There might be various firewalls in place. But the IP address is the easiest way to get access to another computer, so we'd really like to have it available so we can get access to our files. There are two excellent projects that solve this particular problem. Either will get us access to our files, and I use both to great success. But you don't have to do both.

¶Syncthing

The first is the easiest. Syncthing is a continuous file syncronisation program, that will continuously sync whatever folders you like across however many devices you like. You can get it on most Linux distros, Mac, Windows, iOS, and Android. I am, for example, syncing folders across my server, my computer, and my phone with Syncthing. I will simply let you go off and follow the documentation, but I'll make a couple of notes:

1. I found it easiest to go into the harddrive attached to my server, and split my files into a number of sensible directories then have Syncthing sync each folder individually. This, as opposed to just syncing one folder with everything inside it. This means that I can just sync my music, books, and research library to my phone for example, and not every single file I own. My computer and my server in contrast share almost all my files.
2. If you do this and your external harddrives are formatted as ntfs as the last article would have had you do, then in the Syncthing dashboard on your server, you need to tell Syncthing to ignore file permissions when you add a folder for syncing. This is because the ntfs3g program makes all the files in an ntfs partition belong to the super-user no matter what you do (even if you change the ownership of those files). If you don't do this, you're going to have this problem where changes you make on another device can't be synced to your server because Syncthing doesn't have the correct permissions. There is apparently a way to sort this out with ntfs3g mount options, but I never bothered to. I will write about why that is at some point.
3. Syncthing uses addresses, or keys, that it makes up. This means we circumvent the IP address problem entirely. Syncthing just does it's (sync)thing regardless of what the networks are doing around it. This is very helpful, particularly for your phone.
4. If you have a headless setup (like a headless Pi) or you have an operating system that has no graphical user interface, you will not benefit from the previous point. You will need to get access to the IP address of the server so that you can use the Syncthing dashboard and start syncing files. There is probably more than one solution to this problem, one of which I discuss next.
5. If you don't understand how to access the dashboard of Syncthing once it's installed and the docs aren't clear enough, scroll down to the heading in this article Syncthing dashboard from anywhere, and read the part where I explain about ports and IP addresses.

So, with Syncthing, we now have the ability to get our files from our external harddrive onto whatever device we want. We can also add devices into this little network, so if we want to share folders with other devices or friends and family, we can do that too.

However, Syncthing only allows us to sync files. We can go a little further, and create a subnetwork that will drastically improve our ability to build upon our digital architecture.

¶Enter the relay node

The idea here came from two excellent articles by Jordan Crawford here and here (archived versions here and here). The first article sets up the problem, and the second solves it. The broad thrust is to use a virtual private network (VPN) to connect your server to your computer via a third device which acts as a relay.

A VPN is an encrypted tunnel between two or more devices. Most people these days use VPNs to tunnel to some business' servers like protonvpn or privateinternetaccess, anonymising their piracy or porn browsing habits, or tricking Netflix into thinking they're in another country. But we can use one of these networks to create a direct connection between our server and other devices. More importantly, we can use it to create a static IP address on the VPN that we can use instead of whatever our ISP hands out.

So, as usual, I will hand you off to that project (archived version) and make a couple of notes:

1. What you'll end up with is a subnetwork that exists independently of whatever other networks are happening on your devices. So, for example, you might have your server connected to your home internet network and your computer connected to your work internet, but you can still use the IP address in the VPN subnetwork to access the devices with, for example, SSH. If you're paranoid, you might think about using different IP addresses than the ones he suggests you set up, but that would require a little research as to what IP addresses are best to use.
2. Jordan describes only two nodes—a 'home' node and a relay node he calls a 'cloud' node (more on this in the next point). We want at least three—one for our server, one for the relay, and one for our computer. We may even want more than three. We can have as many as we like, we just need to set them up like we set up the 'home' node Jordan describes. You can even get this working on an Android phone. There's a few tinc apps, and it's a little fiddly because you can't just copy the files around like you can on a computer, but it can be done.
3. The other major contribution to this configuration is this relay node. So we have our computer node, our server node, and we're now adding another node which will act as a relay. This is going to cost money, but it's not particularly expensive—the equivalent to a beer a month. This relay node means that we will now be able to get to our network, and thus any of our devices, from anywhere. Not just our computer. We could go to the local internet cafe ¹ or use a friend's computer or phone, and still get to our files.

This relay node is not well described by Jordan, so I will describe it here. You will be renting a computer from a company. Vultr and Digital Ocean are two popular and cheap companies that rent these out. I personally use Vultr, so I'll use them to illustrate. For this, you only need the cheapest one. On Vultr, this is a $5USD per month 'cloud compute' instance. Sign up, and click through all the options—it will ask where you want it (some country close to you is probably best for speed, though it will only mean a difference of a couple seconds at most), how much memory you want (512MB is fine), how many CPU cores you want (1 is fine), how much storage you want (10 GB is the minimum and will be more than enough) and how much bandwidth (I've never even come close to a quarter of their lowest, 1TB). If you're worried about these, go read up on them, because I won't explain them here. You'll also choose an operating system. Just choose the same one as your current home server, for ease of use. Once you've done that, you now have another server, somewhere else. Vultr provides plenty of information explaining how to access it, but you'll do so via SSH which requires the IP address of the computer and the password—both of which Vultr should give you. Same is true of whatever service you choose if you don't use Vultr. At this point, you can use it just like you use your current home server, except that you have no desktop environment—you can only use the terminal. If you still haven't got the hang of this, go do a tutorial on the Linux command line.

I'll describe three usecases here to illustrate why this relay node is so useful. The first will be showing how we can use the VPN generated IP addresses to access the Syncthing dashboard on our server, helpful in particular if we are using a headless machine like a Pi, or a operating system with no graphical user interface on our server. The second is to set up our own cloud service, so that we can have all of those wonderful features—file sharing, access from anywhere, a nice graphical user interface for our files—without resorting to Microsoft or Google, or whoever. This will allow us to secure our offsite backup so no one is reading your files—an article I will write shortly. The third is to set up another server, doing even more backups, at a friend's house. This is a neat solution to our offsite backup worries, and having this relay node means that no matter what your friend's IP address is, so long as the server is connected to the internet, you can SSH into it and manage it from wherever you are. As an example, I can manage a node in Australia from where I am in the UK. It is in fact, what you'll do when you set up the relay node which is a computer off on some company's server farm somewhere.

¶Syncthing dashboard from anywhere

This is pretty straightforward, but we need to learn a little bit of networking stuff first. Syncthing will make its dashboard available on port 8384 by default. The ports of an IP address are conceptually similar to ports in a harbour. If a boat is docked at a port in the harbour, you can only visit that boat at that harbour and that port. In a similar kind of way, the port of an IP address allows a program or service to 'dock' to the device at the IP address, and thus pass information back and forth. Most people call this a 'listen address', so we would say that Syncthing is 'listening' at port 8384. So Syncthing 'docks' it's dashboard at port 8384 by default—it's 'listening' on that port—and so you can go and visit that dashboard at that IP address and port number.

Now, your device can have a bunch of IP addresses. For example, you might have an IP address from your internet service provider. You also now have another IP address from your VPN. You also have an IP address that lets your computer refer to itself. Every device has this—it's called the localhost address, and it's always 127.0.0.1. This address can't be accessed by other devices—indeed, it can't because if you tried, the device attempting it would just try to talk to itself. I guess it's like the computer equivalent of the word 'me'. Most programs will listen at a port at the localhost address by default. Syncthing does this. At the time of writing, it docks its dashboard at 127.0.0.1:8384 by default. This is why, when you follow the Syncthing docs, you access the dashboard by typing 127.0.0.1:8384 in the address bar. Alternatively, you can type localhost:8384 because most browsers know that localhost should be resolved to 127.0.0.1.

We actually want to access it from any IP address attached to the device, or rather we want Syncthing to dock the dashboard at port 8384 on every IP address for that device. We want this because we want to use the IP address given to us by the VPN, not the localhost address. We can do that easily, we just search for something like 'changing the listen address for syncthing gui' (GUI being the graphical user interface, or dashboard, for Syncthing). At the time of writing, you could find that in the Syncthing docs here. Incidentally, we can change the port Syncthing docks the dashboard to in a similar way, if we want, we should just be careful here because certain port numbers will be used already, so do a bit of research on what ports are usually free before you do that. Another thing you might think about is that if Syncthing is docking its dashboard at port 8384, that means nothing else can use that port. Syncthing can't even use that port to transmit file information to the other devices in its efforts to sync files. So, that means it needs to also be using another port to do this, and indeed it does. At the time of writing, it's port 22000.

Assuming you followed Jordan's article, you now have three nodes on your network. We'll use the IP addresses he describes to illustrate—so let's say our relay node is 10.0.0.1, our server is 10.0.0.2 and our computer is 10.0.0.3. We want to access the server Syncthing dashboard, so we can now access that from our computer: type into our browser 10.0.0.2:8384 and you will be presented with the Syncthing dashboard for your server.

¶Your own cloud

This is probably my favourite thing I've done with my own server, even though I almost never use it anymore. Nextcloud is an open source cloud platform. Feature-wise, it's almost indistinguishable from Google Drive or Onedrive. The difference, of course, is that you will be hosting it yourself on your server node. Nextcloud has excellent documentation. But I actually recommend installing Nextcloud with Snapcraft. Snapcraft bundles up programs in little self-contained packages called 'snaps'. For something kind of complex like Nextcloud, it makes installation (and uninstallation) a great deal more straightforward and it's worth reading a little about. You can find snaps for most programs, and you might find it helpful for other things you want to play with on your server. You can find the Nextcloud Snap here, also with excellent documentation. You want to read through this, because installing with Snapcraft will mean that some configuration stuff will be different from Nextcloud installed via ordinary means. Everything else will be the same as per the official Nextcloud documentation.

Of particular interest to us is the fact that we need to enable removable media for the Nextcloud Snap. We have all our files on our external harddrive, and we want to be able to get into that with Nextcloud. So follow the instructions to connect removable media for the Nextcloud Snap. You're not done there though. Once you've followed the instructions to set Nextcloud up, and you're logged into the dashboard, you need to go on and install the additional External Storages app from within the Nextcloud dashboard and configure that. The documentation related to the External Storages app will be the same as the official Nextcloud docs, as opposed to the Snap docs. We want to do it this way because it means that our Nextcloud installation is nice and superfluous. If it breaks (which has happened to me a couple of times) you can just uninstall it (which is extremely easy with a Snap package) and reinstall it, then go through the process to connect your harddrive again. The files on your external harddrive won't care about or be affected by this at all.

The second thing you might want to note is that you can change the port. You probably want to do this. It uses port 80 and 443 by default and that will become problematic if you ever want to set up a webserver on your server node. You can read more about webservers at the bottom of this article.

¶Set up a server at another location

There's actually not much to this. Get another old computer, or a Raspberry Pi, and another couple of harddrives, and set up another server like you did in the last article at another location like your parents' or a friend's house. Give it a Tinc profile, and you can do anything there that you do on the first one you set up. The biggest benefit is that you no longer need the cloud for an offsite backup.

¶Prettying the IP addresses

If you read Jordan's first article (archived version here) you'll notice he spoke about this. Essentially, you can set up a special kind of server—a web server—that will allow us to use domain names instead of IP addresses. Domain names, like www.example.com, are actually just aliases for IP addresses. When you type one into your browser, you send a request to a domain name system (DNS) resolver, who then has to figure out what IP address that domain name belongs to. Then your request gets forwarded on to that IP address.

So you can go off and buy a domain name, point that domain name at the IP address of the relay node. On the relay node, you can install a webserver like Nginx or lighttpd which will redirect requests for that domain name to the subnetwork IP address of your VPN network. In addition, one domain name can support a number of second-level domain names. So, as an example, btrmt.org gets directed to the IP address of a webserver that serves this website. www.btrmt.org also gets sent here, but in theory I could send that somewhere else. syncthing.btrmt.org goes nowhere but I could redirect that to my version of 10.0.0.2:8384. nextcloud.btrmt.org also goes nowhere but I could similarly redirect that to my version of 10.0.0.2:<nextcloud's port>. I'm not going to write this up, but I'll send you off with some pointers for searching:

• You need to buy a domain name from a domain registrar, then change the DNS records to point to the IP address of your relay node so the DNS resolver knows where to forward requests. A domain name registrar doesn't have to manage your DNS records, and might charge a fee for this. At the time of writing, you're getting ripped off if that's the case. Most registrars will do this for free, it's only the big name monsters that try to rip you off like crazydomains and godaddy. Go have a hunt around for registrars that people like.
• You need to install a webserver on your relay node. Nginx is extremely popular and easy to use, but there are a few. I also like lightppd. Go hunt. Webservers always use ports 80 and 443 (although you can change this if you're a loon). Requests from a browser to a domain name or an IP address will come in at your.relay.ip.address:80 by default, unless you explicitly type in a port number. A webserver listening at this address can then decide what to do with that request. You set up records to help the webserver decide. By default, requests will come in as HTTP (i.e. not encrypted) and, as I said, come in at port 80. You can, and really should, make those requests use HTTPS where possible, which uses port 443. You'll need a different service for this that provides HTTPS certificates, and then modify your webserver records to redirect all requests to HTTPS. Once upon a time, this cost money, but certificates can now be obtained easily for free. I recommend certbot, which will let you do this very easily and if you're using Nginx or Apache, will alter your webserver records automatically to make this happen for you. You can, and should, read more about all that on their website because I'm running out of steam here.
• You need to then set up a reverse proxy to tell your webserver where to send the request. This is a type of record on the webserver that receives a request for a domain and then forwards it on somewhere else. In this case, you're forwarding on to a different device—your home server. But you can use reverse proxies to forward it to another port on the same device. So, if for some strange reason you had Syncthing running on your relay node, you could reverse proxy a request to port 8384. Make sure you read about how this works, and also make sure you set it up correctly for each service you're reverse proxying. For example, Nginx—I believe—limits the size of some data packets by default. For Nextcloud, you don't necessarily want this because you might want to upload a big file. So do some reading.

¶Outro

And now, we're done. We have laid the foundations of our digital architecture. Your digital life is completely modular; far, far safer than it was before; and you can build on top of this in a number of very pleasing ways. You can access your files from anywhere, from any device. And if you did everything in this article, you have a pretty good foundational understanding of server, networks, and the internet more broadly. Well done.

On to the next thing.