Poddery/Archive

We run Diaspora, XMPP and Matrix services at poddery.com. Diaspora username and password can be used to access XMPP and Matrix services. chat.poddery.com provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =

Hosting
We are on a C2S instance of scaleway.com bare metal cloud server.


 * 4 Dedicated x86 64bit Cores
 * 8GB Memory
 * 50GB SSD Disk
 * 1 Flexible Public IPv4
 * 300Mbit/s Unmetered bandwidth
 * 2.5Gbit/s Internal bandwidth
 * €11.99 Per Month

Due to performance issues we are migrating to a new server (C2M instance of scaleway.com) with the following specs:


 * 8 Dedicated x86 64bit Cores
 * 16GB Memory
 * 50GB SSD Disk
 * 1 Flexible Public IPv4
 * 500Mbit/s Unmeterd bandwidth
 * 5Gbit/s Internal bandwidth
 * €17.99 Per Month
 * Extra 150GB SSD
 * Total €20.99 Per Month

Operating System
We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

Hardening checklist
/etc/ssh/sshd_config: ... PermitRootLogin no  ... PasswordAuthentication no ... sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow ssh sudo ufw enable Currently ufw is disabled as it is crashing the server.
 * SSH password login disabled (allow only key based logins)
 * root SSH login disabled (use a normal user with sudo)
 * Firewall enabled with only the ports we need opened (ufw tutorial)

/etc/ssh/sshd_config: ... LogLevel VERBOSE ...
 * fail2ban configured against brute force attacks

sudo systemctl restart ssh sudo systemctl enable fail2ban sudo systemctl start fail2ban

Check /var/log/fail2ban.log for logs

Unban an IP: sudo fail2ban-client set sshd unbanip 

Here sshd is the defaut jail name, change it if you are using a different jail.

System health check
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
 * Create partitions root, boot and swap.
 * Setup RAID 1:

# Make sure lvm2 and udev packages are installed sudo apt-get install lvm2 udev # Replace X with valid number according to lsblk sudo pvcreate /dev/nbdX sudo vgcreate data /dev/nbdX sudo lvcreate -n log /dev/data -L  # currently 50G sudo lvcreate -n db /dev/data -L  #currently 500G sudo lvcreate -n diaspora /dev/data -l 100%FREE # Make sure cryptsetup package is installed sudo apt-get install cryptsetup
 * There should be a data disk attached (added from cloud.scaleway.com)
 * The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
 * /dev/data is an lvm volume group created from /dev/nbdX
 * /dev/data/diaspora is an lvm logical volume
 * /dev/mapper/diaspora is an encrypted device

# Give disk encryption password as specified in the access repo sudo cryptsetup luksFormat /dev/data/diaspora sudo cryptsetup luksOpen /dev/data/diaspora diaspora sudo mkfs.ext4 /dev/mapper/diaspora sudo mkdir /var/lib/diaspora sudo mount /dev/mapper/diaspora /var/lib/diaspora
 * /dev/mapper/diaspora is an ext4 file system
 * /var/lib/diaspora should be mounted. All critical data should be on /var/lib/diaspora.

Diaspora

 * We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
 * See /usr/share/doc/diaspora-common/README for package specific configuration.
 * live statistics

Chat/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP sudo mysql -u root -p # Enter password from the access repo CREATE USER 'prosody'@'localhost' IDENTIFIED BY ''; GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost'; FLUSH PRIVILEGES; sudo chown -R root:ssl-cert /etc/letsencrypt sudo chmod g+r -R /etc/letsencrypt sudo chmod g+x /etc/letsencrypt/{archive,live} sudo systemctl restart prosody
 * We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
 * We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See XMPP over HTTPS section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

Set Nginx Conf for BOSH URLS

 * Add this configuration in nginx configuration file to enable the BOSH url to make JSXC Working.

Nginx

upstream chat_cluster { server localhost:5280; }

location /http-bind { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-Forwarded-Proto https; proxy_redirect off; proxy_connect_timeout 5; proxy_buffering      off; proxy_read_timeout   70; keepalive_timeout    70; send_timeout         70; client_max_body_size 4M; client_body_buffer_size 128K; proxy_pass http://chat_cluster; }

Plz look here for more details. And apache settings here :)

Chat/Matrix

 * We use Synapse server for setting up the Matrix server.
 * We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
 * We use nginx reverse proxy to send requests that has /_matrix/* in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
 * We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database

Workers
For scalability, we are running workers. Currently all workers specified in that page, expect `synapse.app.appservice` is running on poddery.com

A new service is installed for the workers. (Save the  file somewhere like   or something.)

The worker config can be found at

Synapse needs to be put under a reverse proxy see. A lot of  urls needs to be overridden too see

These lines must be added to  as we are running ,  ,  ,   workers respectively:

enable_media_repo: False send_federation: False start_pushers: False update_user_directory: false

These services must be enabled, and added to  and   sections of the original  : matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

Upgrade
First check synapse/UPGRADE.rst to see if anything extra need to be done. Then, just run

Homepage
Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

# Make sure git and acl packages are installed sudo apt-get install git acl # Grant rwx permissions for the ssh user to /usr/share/diaspora/public sudo setfacl -m "u::rwx" /usr/share/diaspora/public # Clone poddery.com repo cd /usr/share/diaspora/public git clone https://git.fosscommunity.in/community/poddery.com.git cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted cd .. && rmdir poddery.com
 * poddery.com -> https://git.fosscommunity.in/community/poddery.com

cd /usr/share/diaspora/public/save git submodule init git submodule update save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.
 * save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com

Riot-web Updation
https://chat.poddery.com/#/welcome Backup current riot-web folder from riot to riot-backup wget https://github.com/vector-im/riot-web/releases/download/v1.0.1/riot-v1.0.1.tar.gz tar -xvf riot-v1.01.tar.gz   cp -r riot-v1.0.1/* /var/www/riot/ rm -rf ./riot-v1.0.1* Transfer the old config.json,home.html,home-status.html from riot-backup to /var/www/riot/ systemctl restart nginx

nginx
Front-end for Diaspora and Matrix.

PostgreSQL
Backend for Matrix.

MySQL
Backend for Diaspora.

TODO: Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

exim
For sending emails. sudo dpkg-reconfigure exim4-config

sslh
Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

NOTE: This service has been disabled since the community decided that XMPP service no longer needs to be served via port 443, see this loomio post for more details.

SSL/TLS certificates

 * 1) letsencrypt certonly --webroot -w /usr/share/diaspora/public  -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com


 * 1) cp  -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
 * 2) cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
 * 3) chown -R root:ssl-cert /etc/letsencrypt
 * 4) chmod g+r -R /etc/letsencrypt
 * 5) chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

total 0 lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem
 * 1) ls -l /etc/prosody/certs/

30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log 32 2 * * 1 /etc/init.d/nginx reload 34 2 * * 1 /etc/init.d/prosody reload
 * 1) crontab -e

Handling critical data
sudo /etc/init.d/mysql stop sudo mv /var/lib/mysql /var/lib/diaspora sudo ln /var/lib/diaspora/mysql /var/lib/mysql sudo mkdir /var/lib/diaspora/uploads sudo chown -R diaspora: /var/lib/diaspora/uploads sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

Services health check
Sample output - Look for "Active: active (running)" systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled) Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS) Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS) Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 5072 (nginx) CGroup: /system.slice/nginx.service ├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─13149 nginx: worker process ├─13150 nginx: worker process ├─13151 nginx: worker process └─13153 nginx: worker process

systemctl status diaspora # Diaspora service diaspora.service - LSB: Diaspora application server Loaded: loaded (/etc/init.d/diaspora) Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS) Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS) CGroup: /system.slice/diaspora.service ├─ 850 unicorn worker[0] -c config/unicorn.rb -D ├─ 5174 sudo -u diaspora -E -H ./script/server ├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora) ├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy] ├─ 5222 unicorn master -c config/unicorn.rb -D └─31717 unicorn worker[1] -c config/unicorn.rb -D systemctl status matrix-synapse.service # Synapse Matrix Server matrix-synapse.service - Synapse Matrix homeserver Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled) Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS) Main PID: 15808 (python2.7) CGroup: /system.slice/matrix-synapse.service └─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server prosody.service - LSB: Prosody XMPP Server Loaded: loaded (/etc/init.d/prosody) Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS) Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS) Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS) CGroup: /system.slice/prosody.service └─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls) sslh.service - SSL/SSH multiplexer Loaded: loaded (/lib/systemd/system/sslh.service; enabled) Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago Docs: man:sslh(8) Main PID: 5444 (sslh) CGroup: /system.slice/sslh.service ├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg ├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg └─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =


 * loomio group - we use this for decision making.
 * Hangout with us in our Matrix room #poddery:poddery.com
 * issue tracker - we use this to track progress of tasks

Contact
Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup Vim GPG Plugin for transparent editing. If you are new to GPG, then follow this guide.

Server Access
Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts


 * 1) apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume


 * 1) pvcreate /dev/nbd1
 * 2) vgcreate data /dev/nbd1
 * 3) lvcreate -n diaspora -L 46.5G /dev/data


 * 1) cryptsetup luksFormat /dev/data/diaspora
 * 2) cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab diaspora /dev/data/diaspora none luks

and update /etc/fstab /dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2
 * 1) mkfs.ext4 /dev/mapper/diaspora
 * 2) mkdir /var/lib/diaspora
 * 1) UNCONFIGURED FSTAB FOR BASE SYSTEM


 * 1) mount -a
 * 2) apt-get install mysql-server

Move MySQL data directory to encrypted volume
 * 1) /etc/init.d/mysql stop
 * 2) mv /var/lib/mysql /var/lib/diaspora/
 * 3) ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication


 * 1) adduser sshtunnel --disabled-login
 * 2) su sshtunnel

Generate SSH key pair and copy public key to target system $ ssh-keygen -t rsa $ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
 * 1) mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log
 * 1) crontab -e
 * 2) m h  dom mon dow   command

Note: Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =


 * 1) Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
 * 2) Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
 * 3) Attach the newly created volume to server from Server page
 * 4) Power on the server
 * 5) Create physical volume (pvcreate /dev/nbdN)
 * 6) Expand volume group (vgextend data /dev/nbdN)
 * 7) Expand logical volume (lvresize --size=186G data/diaspora)
 * 8) Expand encrypted partition (cryptsetup resize diaspora)
 * 9) Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history = This section holds maintenance/issue history for future tracking.

When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com) tls error  Unable to initialize TLS: error loading private key (system lib) certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:


 * Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
 * /etc/letsencrypt/ is the ssl directory.
 * Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
 * Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

If replication fails, you can restart it following the instructions here

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594