Self-hosting

Server rack interconnects work in progress

When you host a web-based software on your own (self), it is called self-hosting. It could be a simple HTML website, a dynamic PHP blog, or a distributed social network.

Typically "self" refers to configuration of the software (operating system, packages, database, and so on) alone and the physical server is operated by a cloud provider like AWS, DigitalOcean, or Hetzner. But some people self-host on computers they physically control, in their home or office.

⚠ Note: The examples used throughout this article are not endorsements/suggestions. They're for illustrative purposes with the hope that a novice reader might have heard about that service provider/software and can consequently gain greater understanding.

Elements of self-hosting

Networking

When you self-host a service, you have to ensure the users/clients are able to reach the service. This requires networking to be setup correctly.

You would typically need a domain name. A domain name (like fsci.in) is purchased via a registrar (like NameCheap, Gandi.net) for (mostly) about ₹1000-1500 per year. The registrar puts your (registrant) details in the registry (which is typically operated by countries or very big companies). And once this is done, you become the proud owner of the domain name. The registration will have to be annually renewed (unless you registered for multiple years). All of this process is overseen by ICANN.

When you register a domain (say, fsci.in), you are automatically in charge of all subdomains (wiki.fsci.in, videos.fsci.in, meet.fsci.in, etc) and these do not require further registration with the registry. How do you configure these then? That's where DNS comes in. DNS allows you to specify which server your domains and subdomains should be connected to.

Note: You might already have seen the term DNS in your router or network settings. It's the same concept. The public/free DNS service you use on your computer is what your browser, etc will use to find out IP addresses of websites you want to visit. But even that DNS service needs to know how to answer a request for your domain. And therefore you need an "authoritative" DNS server which is configured with the right answer. This is often provided by the registrar themselves, but you can also have a service provider different from your registrar be your authoritative DNS server. DNS servers talk to each other. Therefore, even if you have a different DNS server on your router/network, it will eventually ask your authoritative DNS server for the answers.

You can, for example, have fsci.in pointed to 35.185.44.232 with a DNS 'A' record and have wiki.fsci.in pointed to 135.181.250.25 with another DNS 'A' record. That means, when someone types in fsci.in on their browser, the browser will connect to the IP address 35.185.44.232 which will connect it to a server on the internet with that particular IP address. And similarly when someone types in wiki.fsci.in the browser will connect to a different server, the one with the IP address 135.181.250.25.

Note: DNS has many types of records. A records are not the only type.

Server

The word server has two meanings. It can denote the computer that's running your services (for example VPS, i.e., Virtual Private Server), it can also mean the software that's running your services within your server (example: Caddy, Nginx, Apache).

When you are self-hosting using cloud servers, you would have to purchase a server by paying the service provider (starting from about ₹300-500 per month). The provider (AWS, DigitalOcean, Hetzner, etc) would then run a (virtual) computer in their data center for you. They will ensure that this computer never shuts down, is always kept cool, and is always connected to the internet. They will also provide an IP address for this server. (This is the IP address that you put in your DNS records).

When your service provider creates the server, they will also install an operating system of your choice (like Debian, Ubuntu, CentOS) on that server. You will be expected to log-in using SSH and configure the server to your liking — installing packages, updating the operating system, editing configuration files, etc.

Once a user's network request reaches your server, the server software you install (like Caddy, Nginx, Apache) can respond to it based on how it is configured. For a simple HTML site, you typically do not need any additional software and your server software can directly respond with the HTML file from the configured folder. For more complicated services, your server will typically talk to a different software to get the response which eventually gets sent to the user.

Software

Examples of web-based software include WordPress (for blogging), MediaWiki (for wiki), Loomio (for decision making), Prosody (for XMPP instant messaging), NextCloud (for file sharing) and any React/NodeJS/Vue/NextJS/Django/Java/Rails/Flask/etc web projects that you build. Each of these differ in how they are installed and run and you would have to rely on the documentation specific to the software to figure out how to run the service.

Many software projects use Docker as a uniform way to deploy their software on any operating system. Therefore, many software documentation will ask you to install/run their software using docker. In such cases, there is also often a Docker Compose configuration they recommend which helps with deploying the software and other components it depends on (like the database).

Database

Most software store important data in a different software – the database (eg: mysql/mariadb, postgresql, sqlite). Some software support using any database software, some require a specific database software.

Difficulties in self-hosting

Although conceptually straightforward, there are some real life scenarios which might make self-hosting a difficult choice for some people.

Reliability and uptime

Your self-hosted setup can stop working for various reasons. Your site could become popular and get too many requests, thus overloading your server. You could run out of storage space on your disk. You might exceed the bandwidth provided by your internet service provider. Your cloud provider might restart your server without warning, or shut you out because you missed paying.

You might have to account for these when deploying a super-important service.

Data loss and backups

There are ways in which you might lose some or all of the data stored by your services. The server could crash. You might accidentally delete some important files or database. You might lose access to the server backend because you lost your SSH credentials and password. The provider might kick you out without warning. Issues during software upgrade could cause data corruption.

Frequently backing up important data is advised, but also difficult to practice. Some cloud providers have backup services that can do this at an extra cost. There is no guarantee that you'll be able to restore data from backup too.

Security

There are several bad people on the internet who will try to gain access to your server for various reasons. They might hack into your accounts at your service providers (by phishing, stealing passwords, etc), into your operating system (by brute forcing, etc), or into your software (by exploiting vulnerabilities, etc). Once they gain access to your system they might steal your data and misuse it, hold it ransom, or destroy it altogether. They might run harmful software on your server. They might sometimes even be silently collecting information without revealing their presence.

When a software maker becomes aware of a security bug (vulnerability) in their software, they fix the bug (patch it) and release new version of their software without the vulnerability. Therefore, keeping your software (operating system and packages) up-to-date with new versions of software is crucial. (But newer versions of software could introduce newer bugs and other incompatibilities).

Even if the core packages of your software is regularly updated, plugins/extensions can also make the software vulnerable. For example, WordPress has several thousands of plugins (developed by numerous people) and some of these might have vulnerabilities that are not fixed.

Cost

The more powerful and complicated your self-hosting setup is, the more costly it becomes to run it. You would need to purchase servers with more RAM/storage/CPU and that would be more expensive. You would need larger backups and that'll need to be paid for. And so on.

While it is possible to find relatively cheap hosting providers, some of them are also more risky. For example, there are complaints about Hetzner kicking users out and deleting their data for missing payments. Some providers (like AWS) offer a free tier to begin with, but at the end of the free tier you would have to pay relatively more (and by then you might have done too much work that is difficult to move to a different provider).

As costs depend on the exact services you're running, it is difficult to suggest a reasonable cost. But if you are running a simple website or service, it should cost you about ₹500-1500 per month (and not more).

Tips

Prefer static websites when possible

A static website refers to a bunch of HTML pages (and the images/styles/javascript required for these) that can be directly served by any web server software. These can be generated by hand (by typing out HTML), by static site generators (like hugo, jekyll), or by using a static export feature of dynamic websites (eg: Simply Static plugin of WordPress, or Static Export feature of NextJS).

Once generated, these HTML files can be served to any number of users without any change, and without running any other software. Therefore they're simple to host and serve. You can simply use your own web server and point it to the folder containing HTML files to serve the website.

There are several free hosting providers who host simple HTML static websites too! These include Codeberg Pages, Grebedoc, Cloudflare Pages, Netlify, Gitlab Pages, Github Pages, etc.

Although it has limitations in terms of personalization and dynamic features, several websites (including FSCI's website) run as static websites. These are easier to be made secure, can be easily hosted, can be optimized to load very fast, and has several other advantages.

Read documentation

Some people don't read. And then they make mistakes that they could have avoided if they had read the documentation. One could say that this is a design issue and that the person who did the documentation is to be blamed. But if you want to host services on your own, a mature approach would be to hold yourself accountable in reading documentation. Read documentation. Even if it is long. Even if it will take a long time. Even if you don't understand most of it. (Try to understand more about topics you don't understand). It will help.

If you can read the code, you will be able to solve more problems

While good documentation is enough to deploy software, documentation is not always complete or correct. If you are able to read the source code you will be able to get an exact answer to various problems you come across. You might be able to spot bugs, report them, and even fix them! Since free software is a collective enterprise, your contribution in this way is welcome and encouraged! (Be mindful about the personal preferences of maintainers, though)

LLMs are Stochastic Parrots

While LLMs are good at giving you guidance on things about which there is plenty of literature on the internet, many sysadmin tasks are very contextual to your setup (the state of your server and the peculiarities of the software you're dealing with). If you do not know what you're doing, using LLM can be dangerous as they can give generic answers which might not be the best for you. If you do know what you're doing, you probably will be using LLM like a search engine.

See Also

• Sysadmin roadmap

External Links

• How I host websites — blog post by Keith Cirkel, engineer at Mozilla