Note: Currently new registrations are closed, if you want an account Contact us

Difference between revisions of "Poddery - Diaspora, Matrix and XMPP"

From FSCI Wiki
(Updated according to the current setup in Hetzner server)
Line 1: Line 1:
We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.
We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.


= Environment =
= Environment =
== Hosting ==
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:


* 4 Dedicated x86 64bit Cores
* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 8GB Memory
* 4TB HDD
* 50GB SSD Disk
* 32GB DDR3 RAM
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month
 
 
Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:
 
* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month


== Operating System ==
== Operating System ==
 
* Debian Buster
We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.
 
=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
  ...
  PermitRootLogin no
  ...
  PasswordAuthentication no
  ...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
  sudo ufw default deny incoming
  sudo ufw default allow outgoing
  sudo ufw allow ssh
  sudo ufw enable
Currently ufw is disabled as it is crashing the server.
 
* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
  ...
  LogLevel VERBOSE
  ...
 
  sudo systemctl restart ssh
  sudo systemctl enable fail2ban
  sudo systemctl start fail2ban
 
Check '''/var/log/fail2ban.log''' for logs
 
Unban an IP:
  sudo fail2ban-client set sshd unbanip <banned_ip>
 
Here sshd is the defaut jail name, change it if you are using a different jail.
 
=== System health check ===
 
* Create partitions root, boot and swap.
* Setup RAID 1:
  mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
 
* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
  # Make sure '''lvm2''' and '''udev''' packages are installed
  sudo apt-get install lvm2 udev
 
  # Replace X with valid number according to '''lsblk'''
  sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
  sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
  sudo lvcreate -n log /dev/data -L <size_of_disk> # currently 50G
  sudo lvcreate -n db /dev/data -L <size_of_disk> #currently 500G
  sudo lvcreate -n diaspora /dev/data -l 100%FREE
* /dev/mapper/diaspora is an encrypted device
  # Make sure '''cryptsetup''' package is installed
  sudo apt-get install cryptsetup
 
  # Give disk encryption password as specified in the [[#Server_Access|access repo]]
  sudo cryptsetup luksFormat /dev/data/diaspora
  sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
  sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
  sudo mkdir /var/lib/diaspora
  sudo mount /dev/mapper/diaspora /var/lib/diaspora


== User Visible Services ==
== User Visible Services ==
=== Diaspora ===
=== Diaspora ===
 
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* For live statistics see https://poddery.com/statistics
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]


=== Chat/XMPP ===
=== Chat/XMPP ===
 
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.  
* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster]
  # Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.
  sudo mysql -u root -p # Enter password from the access repo
 
  CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
  GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
  FLUSH PRIVILEGES;
 
  sudo chown -R root:ssl-cert /etc/letsencrypt
  sudo chmod g+r -R /etc/letsencrypt
  sudo chmod g+x /etc/letsencrypt/{archive,live}
 
  sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.
 
==== Set Nginx Conf for BOSH URLS ====
 
* Add this configuration in nginx configuration file to enable the BOSH url to make JSXC Working.
 
'''Nginx'''
 
 
upstream chat_cluster {
  server localhost:5280;
}
 
location /http-bind {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-Proto https;
  proxy_redirect off;
  proxy_connect_timeout 5;
  proxy_buffering      off;
  proxy_read_timeout    70;
  keepalive_timeout    70;
  send_timeout          70;
  client_max_body_size 4M;
  client_body_buffer_size 128K;
  proxy_pass http://chat_cluster;
}
 
 
Plz look [https://wiki.diasporafoundation.org/Integration/Chat#Nginx here] for more details. And apache settings [https://github.com/jsxc/jsxc/wiki/Prepare-apache here] :)


=== Chat/Matrix ===
=== Chat/Matrix ===
 
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* We use Synapse server for setting up the Matrix server.
* Synapse is currently installed directly from their [https://github.com/matrix-org/synapse/#synapse-installation official GitHub repo]
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database
 
==== Workers ====
 
For scalability, we are running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect `synapse.app.appservice` is running on poddery.com
 
A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers. (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something.)
 
The worker config can be found at <code>/etc/matrix-synapse/workers</code>
 
Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
 
These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
 
  enable_media_repo: False
  send_federation: False
  start_pushers: False
  update_user_directory: false
 
These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
  matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service
 
==== Upgrade ====
 
First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra need to be done. Then, just run <code>/root/upgrade-synapse</code>


=== Homepage ===
=== Homepage ===
 
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].  
Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.
 
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
  # Make sure '''git''' and '''acl''' packages are installed
  sudo apt-get install git acl
 
  # Grant rwx permissions for the ssh user to /usr/share/diaspora/public
  sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public
 
  # Clone poddery.com repo
  cd /usr/share/diaspora/public
  git clone https://git.fosscommunity.in/community/poddery.com.git
  cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
  cd .. && rmdir poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
  cd /usr/share/diaspora/public/save
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery
  git submodule init
  git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.
 
=== Riot-web Updation ===
  https://chat.poddery.com/#/welcome
  Backup current riot-web folder from riot to riot-backup
  wget https://github.com/vector-im/riot-web/releases/download/v1.0.1/riot-v1.0.1.tar.gz
  tar -xvf riot-v1.01.tar.gz
  cp -r riot-v1.0.1/* /var/www/riot/
  rm -rf ./riot-v1.0.1*
  Transfer the old config.json,home.html,home-status.html from riot-backup to /var/www/riot/
  systemctl restart nginx


== Backend Services ==
== Backend Services ==
=== nginx ===
=== Web Server / Reverse Proxy ===
 
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.
Front-end for Diaspora and Matrix.
 
=== PostgreSQL ===
 
Backend for Matrix.
 
=== MySQL ===
 
Backend for Diaspora.
 
'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).
 
=== exim ===


For sending emails.
=== Database ===
sudo dpkg-reconfigure exim4-config
* PostgreSQL for Matrix
* MySQL for Diaspora


=== sslh ===
''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).


Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).
=== Email ===
 
* Exim
NOTE: This service has been disabled since the community decided that XMPP service no longer needs to be served via port 443, see this [https://www.loomio.org/d/xSiI8FGT/xmpp-service-on-port-443-and-sslh-complexity loomio post] for more details.


=== SSL/TLS certificates ===
=== SSL/TLS certificates ===
 
* Letsencrypt
# letsencrypt certonly --webroot -w /usr/share/diaspora/public  -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
 
# cp  -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*
 
Make sure the certificates used by prosody are symbolic links to letsencrypt default location.
 
# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem
 
# crontab -e
30 2 * * 1 letsencrypt renew  >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload
 
=== Handling critical data ===
  sudo /etc/init.d/mysql stop
  sudo mv /var/lib/mysql /var/lib/diaspora
  sudo ln /var/lib/diaspora/mysql /var/lib/mysql
  sudo mkdir /var/lib/diaspora/uploads
  sudo chown -R diaspora: /var/lib/diaspora/uploads
  sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads
 
=== Services health check ===
 
Sample output - Look for "Active: active (running)"
 
  systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
  nginx.service - A high performance web server and a reverse proxy server
  Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
  Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
  Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
  Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
  Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Main PID: 5072 (nginx)
  CGroup: /system.slice/nginx.service
          ├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
          ├─13149 nginx: worker process
          ├─13150 nginx: worker process
          ├─13151 nginx: worker process
          └─13153 nginx: worker process
 
  systemctl status diaspora # Diaspora service
  diaspora.service - LSB: Diaspora application server
  Loaded: loaded (/etc/init.d/diaspora)
  Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
  Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
  Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
  CGroup: /system.slice/diaspora.service
          ├─  850 unicorn worker[0] -c config/unicorn.rb -D
          ├─ 5174 sudo -u diaspora -E -H ./script/server
          ├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
          ├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
          ├─ 5222 unicorn master -c config/unicorn.rb -D
          └─31717 unicorn worker[1] -c config/unicorn.rb -D 
  systemctl status matrix-synapse.service # Synapse Matrix Server
  matrix-synapse.service - Synapse Matrix homeserver
  Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
  Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
  Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
  Main PID: 15808 (python2.7)
  CGroup: /system.slice/matrix-synapse.service
          └─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
 
  systemctl status prosody # Prosody XMPP Server
  prosody.service - LSB: Prosody XMPP Server
  Loaded: loaded (/etc/init.d/prosody)
  Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
  Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
  Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
  Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
  CGroup: /system.slice/prosody.service
          └─6231 /usr/bin/lua5.1 /usr/bin/prosody
 
  systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
  sslh.service - SSL/SSH multiplexer
  Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
  Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
    Docs: man:sslh(8)
  Main PID: 5444 (sslh)
  CGroup: /system.slice/sslh.service
          ├─  713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─  830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          ├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
          └─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg


= Coordination =
= Coordination =
 
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks
* [https://git.fosscommunity.in/community/poddery.com/issues issue tracker] - we use this to track progress of tasks


=== Contact ===
=== Contact ===
Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)


The following people have their GPG keys in the password file.
The following people have their GPG keys in the password file:
Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE), Akhil Varkey (ID: 0x32FF6C6F5B7AE248)


Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)
We recommend you to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].
 
We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].


=== Server Access ===
=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access


Maintained in a private git repo at -> https://git.fosscommunity.in/community/access
= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (''sda'' and ''sdb'').
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (''md0'' - 16GB), boot (''md1'' - 512MB) and root (''md2'' - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure lvm2, udev and cryptsetup packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_access|access repo]]
cryptsetup luksOpen /dev/mdX poddery
# LVM Setup
# Create physical volume name '''poddery'''
pvcreate /dev/mapper/poddery
# Create volume group named '''data'''
vgcreate data /dev/mapper/poddery
# Create logical volumes named log, db and static
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE
# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery


= Setting up Backup =
# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery


Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.
== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...


Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts
* Firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw enable


  # apt-get install lvm2 cryptsetup
* fail2ban configured against brute force attacks
  # Check for the following line /etc/ssh/sshd_config:
...
LogLevel VERBOSE
...
# Restart SSH and enable fail2ban
sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban
# To unban an IP, first check ''/var/log/fail2ban.log'' to get the banned IP and then run the following
# Here sshd is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>


Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume
== Diaspora ==
* Diaspora installation and configuration:
apt install diaspora-isntaller
# Move MySQL data to encrypted partition
systemctl stop mysql
# Make sure /dev/data/db is mounted to /var/lib/db
mv /var/lib/mysql /var/lib/db
ln -s /var/lib/db/mysql /var/lib/mysql


# pvcreate /dev/nbd1
* Modify configuration files at ''/etc/diaspora'' and ''/etc/diaspora.conf'' as needed (backup of the current configuration files are available in the [[#Server_access|access repo]]).
# vgcreate data /dev/nbd1
* Homepage configuration:
# lvcreate -n diaspora -L 46.5G /dev/data
  # Make sure git and acl packages are installed
 
  sudo apt-get install git acl
# cryptsetup luksFormat /dev/data/diaspora
   
# cryptsetup luksOpen /dev/data/diaspora diaspora
  # Grant rwx permissions for the ssh user to /usr/share/diaspora/public
 
  sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public
and update /etc/crypttab
# <target name> <source device>        <key file>      <options>
# Clone poddery.com repo
diaspora /dev/data/diaspora none luks
cd /usr/share/diaspora/public
 
  git clone https://git.fosscommunity.in/community/poddery.com.git
 
  cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
# mkfs.ext4 /dev/mapper/diaspora
cd .. && rmdir poddery.com
  # mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2
 
# mount -a
  # apt-get install mysql-server
 
Move MySQL data directory to encrypted volume
  # /etc/init.d/mysql stop
  # mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql
 
Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication
 
Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)
 
Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication
 
# adduser sshtunnel --disabled-login
# su sshtunnel
 
Generate SSH key pair and copy public key to target system
  $ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N
 
Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1
 
Uploads are rsynced every hour
 
# crontab -e
# m h  dom mon dow  command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log
 
 
'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com
 
= Add more disk space =
 
# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)
 
= Maintenance history =
This section holds maintenance/issue history for future tracking.
 
'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.
 
1. Prosody error - Failed to load private key
 
  certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
  tls error  Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.
 
This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.
 
Note that Poddery uses Letsencrypt for ssl.
 
Fix:
 
* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.


'''If replication fails, you can restart it following the instructions here'''
* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update


https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594
= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.


[[Category:Services]]
[[Category:Services]]

Revision as of 03:31, 1 May 2019

We run decentralized and federated Diaspora social netowrk, XMPP and Matrix instant messaging services at poddery.com. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. chat.poddery.com provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

Environment

Hosting

Poddery is hosted at Hetzner with the following specs:

  • Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
  • 4TB HDD
  • 32GB DDR3 RAM

Operating System

  • Debian Buster

User Visible Services

Diaspora

Chat/XMPP

  • Prosody is used as the XMPP server which is modern and lightweight.
  • Currently installed version is 0.11.2 which is available in Debian Buster
  • All XEPs are enabled which the Conversations app support.

Chat/Matrix

Homepage

Homepage and other static pages are maintained in FSCI GitLab instance.

Backend Services

Web Server / Reverse Proxy

  • Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

Database

  • PostgreSQL for Matrix
  • MySQL for Diaspora

TODO: Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

Email

  • Exim

SSL/TLS certificates

  • Letsencrypt

Coordination

Contact

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file: Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE), Akhil Varkey (ID: 0x32FF6C6F5B7AE248)

We recommend you to setup Vim GPG Plugin for transparent editing. If you are new to GPG, then follow this guide.

Server Access

Maintained in a private git repo at https://git.fosscommunity.in/community/access

Configuration and Maintenance

Disk Partitioning

  • RAID 1 setup on 2x2TB HDDs (sda and sdb).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
  • Separate partitions for swap (md0 - 16GB), boot (md1 - 512MB) and root (md2 - 50GB).
  • LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure lvm2, udev and cryptsetup packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the access repo
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume name poddery
pvcreate /dev/mapper/poddery
# Create volume group named data
vgcreate data /dev/mapper/poddery
# Create logical volumes named log, db and static
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE 

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

Hardening checklist

  • SSH password based login disabled (allow only key based logins)
  • SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...
  • Firewall enabled with only the ports that needs to be opened (ufw tutorial)
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw enable
  • fail2ban configured against brute force attacks
# Check for the following line /etc/ssh/sshd_config:
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

# To unban an IP, first check /var/log/fail2ban.log to get the banned IP and then run the following
# Here sshd is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

Diaspora

  • Diaspora installation and configuration:
apt install diaspora-isntaller

# Move MySQL data to encrypted partition
systemctl stop mysql
# Make sure /dev/data/db is mounted to /var/lib/db
mv /var/lib/mysql /var/lib/db
ln -s /var/lib/db/mysql /var/lib/mysql
  • Modify configuration files at /etc/diaspora and /etc/diaspora.conf as needed (backup of the current configuration files are available in the access repo).
  • Homepage configuration:
# Make sure git and acl packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com
  • Save Poddery repo is maintained as a sub module in poddery.com repo. See this tutorial for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

History

  • See here for the archive of Poddery wiki page before the migration to Hetzner.
Retrieved from "https://wiki.fsci.in/index.php?title=Poddery_-_Diaspora,_Matrix_and_XMPP&oldid=10792"