FSCI Wiki - User contributions [en]

Codema.in

2021-09-06T15:20:56Z

AkhilVarkey:

[https://freesoftwareindia.org/ Loomio] instance at https://codema.in. Loomio is a '''co'''llaborative '''de'''cision-'''ma'''king platform (hence we're using the name '''codema''') where users can initiate discussions and put up proposals.

== Hosting ==
We are now on a [https://www.scaleway.com/virtual-cloud-servers DEV1-S instance] virtual cloud server with the following specs:
Moved to this server, by Dec 2019, to consolidate all groups related to FSCI and Indian Pirates from loomio.org and codema.fsci.org.in to a single loomio instance.

* 2 x86-64 bit Cores
* 2 GB Memory
* 20GB SSD Disk
* 100 Mbit/s Bandwidth
* 1 Reserved IP (v4)
* €5.58/Month

== Coordination ==

* Hangout with us in our Matrix room [https://matrix.to/#/#codema:poddery.com #codema:poddery.com]

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

== History ==

Before moving the service to current server, we were on a [https://www.scaleway.com/virtual-cloud-servers DEV1-S instance] virtual cloud server with the following specs:

* 2 x86-64 bit Cores
* 2 GB Memory
* 50GB SSD Disk
* 200 Mbit/s Bandwidth
* 1 Reserved IP (v4)
* €3.99/Month

We were initially on a [https://www.scaleway.com/virtual-cloud-servers VC1-S instance] virtual cloud server with the following specs until 31/01/2019.

* 2 x86-64 bit Cores
* 2GB Memory
* 50GB SSD Disk
* 1 Reserved Public IPv4
* 200Mbit/s Unmetered bandwidth
* €3.99/Month

On 01/02/2019 our server was taken down by [https://scaleway.com Scaleway] quoting payment issues. Payment was failing even after updating the credit card details and following that our server got deleted without proper notifications from Scaleway's side. Screenshot of the email from Scaleway attached below. Fortunately we were provided with a snapshot of the server from which we were able to recover codema to a new server. The service was moved to

=== Loomio Deployment Reference ===

Initially loomio was deployed and updated on codema.fsci.org.in using this official guide - https://github.com/loomio/loomio-deploy.
Currently loomio is deployed and updated on codema.in using the same official guide but differs on the fact that compose file uses [https://hub.docker.com/r/dannycarrera/nginx-proxy nginx-proxy] and additional parameter of VIRTUAL_HOST_ALIAS in the env file. The VIRTUAL_HOST_ALIAS is for redirecting www.codema.in queries to codema.in. We use the smtp server from nemo.libreinfra.org for sending outbound mails.

=== Codema data migration process ===

For migration of loomio groups from both loomio.org and codema.fsci.org.in, the group data was exported using data export options provided by loomio. Changed the url links in the json data to point towards https://codema.in. The resulting json files was imported into the new codema instance after setting up the docker service as described [https://help.loomio.org/en/user_manual/groups/data_export/#import here].

=== Codema recovery process ===

''Here's a brief description of how codema was recovered after the server take down on 01/01/2019:''

Under 'Snapshots' tab in the Scaleway dashboard we were provided with the snapshot (backup) of our codema server. A system image was created from this snapshot and it was used to create a new server with similar specifications. We lost our public IP along with the old server, so a new IP was assigned to the server and then updated the DNS A record of codema.fsci.org.in to point to this new IP. Once the server was up loomio was restarted using the following commands from the loomio installation directory:

docker-compose down
docker-compose up -d

The logs were checked for errors using the following command:

docker-compose logs -f

Loomio wasn't getting started saying the port 25 was already in use. So the application using that port (which was exim4 in this case) was killed and loomio was restarted again.

An archive of the old loomio page is available at [[Loomio/Archive_1|Loomio Archive]].

[[Category: Services]]

Codema.in

2021-09-06T15:07:20Z

AkhilVarkey: Added details about current running instance and data migration process.

[https://freesoftwareindia.org/ Loomio] instance at https://codema.in. Loomio is a '''co'''llaborative '''de'''cision-'''ma'''king platform (hence we're using the name '''codema''') where users can initiate discussions and put up proposals.

== Hosting ==
We are now on a [https://www.scaleway.com/virtual-cloud-servers DEV1-S instance] virtual cloud server with the following specs:
Moved to this server to consolidate all groups related to FSCI and Indian Pirates from loomio.org and codema.fsci.org.in to a single loomio instance.

* 2 x86-64 bit Cores
* 2 GB Memory
* 20GB SSD Disk
* 100 Mbit/s Bandwidth
* 1 Reserved IP (v4)
* €5.58/Month

== Coordination ==

* Hangout with us in our Matrix room [https://matrix.to/#/#codema:poddery.com #codema:poddery.com]

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

== History ==

Before moving the service to current server, we were on a [https://www.scaleway.com/virtual-cloud-servers DEV1-S instance] virtual cloud server with the following specs:

* 2 x86-64 bit Cores
* 2 GB Memory
* 50GB SSD Disk
* 200 Mbit/s Bandwidth
* 1 Reserved IP (v4)
* €3.99/Month

We were initially on a [https://www.scaleway.com/virtual-cloud-servers VC1-S instance] virtual cloud server with the following specs until 31/01/2019.

* 2 x86-64 bit Cores
* 2GB Memory
* 50GB SSD Disk
* 1 Reserved Public IPv4
* 200Mbit/s Unmetered bandwidth
* €3.99/Month

On 01/02/2019 our server was taken down by [https://scaleway.com Scaleway] quoting payment issues. Payment was failing even after updating the credit card details and following that our server got deleted without proper notifications from Scaleway's side. Screenshot of the email from Scaleway attached below. Fortunately we were provided with a snapshot of the server from which we were able to recover codema to a new server. The service was moved to

=== Loomio Deployment Reference ===

Initially loomio was deployed and updated on codema.fsci.org.in using this official guide - https://github.com/loomio/loomio-deploy.
Currently loomio is deployed and updated on codema.in using the same official guide but differs on the fact that compose file uses [https://hub.docker.com/r/dannycarrera/nginx-proxy nginx-proxy] and additional parameter of VIRTUAL_HOST_ALIAS in the env file. The VIRTUAL_HOST_ALIAS is for redirecting www.codema.in queries to codema.in. We use the smtp server from nemo.libreinfra.org for sending outbound mails.

=== Codema data migration process ===

For migration of loomio groups from both loomio.org and codema.fsci.org.in, the group data was exported using data export options provided by loomio. Changed the url links in the json data to point towards https://codema.in. The resulting json files was imported into the new codema instance after setting up the docker service as described [https://help.loomio.org/en/user_manual/groups/data_export/#import here].

=== Codema recovery process ===

''Here's a brief description of how codema was recovered after the server take down on 01/01/2019:''

Under 'Snapshots' tab in the Scaleway dashboard we were provided with the snapshot (backup) of our codema server. A system image was created from this snapshot and it was used to create a new server with similar specifications. We lost our public IP along with the old server, so a new IP was assigned to the server and then updated the DNS A record of codema.fsci.org.in to point to this new IP. Once the server was up loomio was restarted using the following commands from the loomio installation directory:

docker-compose down
docker-compose up -d

The logs were checked for errors using the following command:

docker-compose logs -f

Loomio wasn't getting started saying the port 25 was already in use. So the application using that port (which was exim4 in this case) was killed and loomio was restarted again.

[[Category: Services]]

Codema.in

2021-09-06T14:30:36Z

AkhilVarkey: Documentation for Loomio Setup and data migration.

System Administrators Checklist

2020-12-15T17:14:13Z

AkhilVarkey: /* Free Software Camp Tasks */

= Pre-Requisites (you need to learn yourself) =
# How to install GNU/Linux
# Familiarity with Command Line
# disk partitioning with logical volume manager
# authenticating with ssh keys

= Server basics (we will teach you) =
# switching users (sudo, su)
# remote access (scp, rsync, custom ssh port, mosh),
# software raid
# encrypted partitions/luks (using virtual machines)
# firewall with ufw
# postgresql replication (backup)
# scheduled backups (rsync and cron)
# lxc container (setup services on your local machine)
# sharing passwords with gpg encrypted files
# nginx basics (setup web server, add custom index page)
# screen/tmux/nohup
# symbolic links (ln -s)
# locales
# environment variables
# local network configuration (/etc/hosts, ip, ss).
# Starting and stopping services (systemctl)
# Log file handling (tail -f, truncate, logrotate)

== Switching users ==

sudo or su commands can be used to run commands as different users. `sudo -u <username>` for running as different user. `su - postgres` can give you a shell as postgres user.

== Remote access to machines ==

#. ssh - remote shell (with ssh server on custom ports)
#. scp/sftp/rsync - copy files
#. mosh - for bad connections

== Symbolic links ==

Symbolic links can be used to store data in data partition without changing configuration files. For example /var/lib/postgresql can be a symbolic link to /data/postgresql where /data is a dedicated partition for storing data.

== Setup correct Locales ==

`dpkg-reconfigure locales`

= Free Software Camp Tasks =
* Setup feed2toot for fsci blog, diasp.in updates - https://git.fosscommunity.in/fsfi/camp/-/issues/36#notes

= Free Software Camp Resources =
* [[Hosting_Providers_with_free_tiers_or_credits]]

Poddery - Diaspora, Matrix and XMPP

2020-06-25T13:19:40Z

AkhilVarkey: Details about compression of matrix-synapse, some re ordering and typo fixes

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Relaod tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2020-06-25T08:40:45Z

AkhilVarkey:

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Relaod tinc vpn servie at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

===Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Postgresql (for synapse) Standby configuration: Step 2===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500
max_worker_processes = 16

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2020-06-25T08:39:52Z

AkhilVarkey:

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Relaod tinc vpn servie at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

===Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Postgresql (for synapse) Standby configuration: Step 2===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500
max_worker_processes = 16

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Loomio/Archive 1

2020-05-04T17:02:45Z

AkhilVarkey: AkhilVarkey moved page Loomio/archive 1 to Loomio/Archive 1 without leaving a redirect: Making the naming compatible with Poddery Archive and keeping standard procedure for moving pages

[https://freesoftwareindia.org/ Loomio] instance at https://codema.fsci.org.in. Loomio is a '''co'''llaborative '''de'''cision-'''ma'''king platform (hence we're using the name '''codema''') where users can initiate discussions and put up proposals.

== Hosting ==
We were now on a [https://www.scaleway.com/virtual-cloud-servers START1-S instance] virtual cloud server with the following specs:

* 2 x86-64 bit Cores
* 2 GB Memory
* 50GB SSD Disk
* 200 Mbit/s Bandwidth
* 1 Reserved IP (v4)
* €3.99/Month

== Coordination ==

* Hangout with us in our Matrix room [https://matrix.to/#/#codema:poddery.com #codema:poddery.com]

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

== History ==
We were on a [https://www.scaleway.com/virtual-cloud-servers VC1S instance] virtual cloud server with the following specs until 31/01/2019.

* 2 x86-64 bit Cores
* 2GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 200Mbit/s Unmetered bandwidth
* €2.99/Month

On 01/02/2019 our server was taken down by [https://scaleway.com Scaleway] quoting payment issues. Payment was failing even after updating the credit card details and following that our server got deleted without proper notifications from Scaleway's side. Screenshot of the email from Scaleway attached below. Fortunately we were provided with a snapshot of the server from which we were able to recover codema to a new server.

=== Loomio Deployment Reference ===

Loomio was deployed and updated on codema.fsci.org.in using this official guide - https://github.com/loomio/loomio-deploy

=== Codema recovery process ===

''Here's a brief description of how codema was recovered after the server take down on 01/01/2019:''

Under 'Snapshots' tab in the Scaleway dashboard we were provided with the snapshot (backup) of our codema server. A system image was created from this snapshot and it was used to create a new server with similar specifications. We lost our public IP along with the old server, so a new IP was assigned to the server and then updated the DNS A record of codema.fsci.org.in to point to this new IP. Once the server was up loomio was restarted using the following commands from the loomio installation directory:

docker-compose down
docker-compose up -d

The logs were checked for errors using the following command:

docker-compose logs -f

Loomio wasn't getting started saying the port 25 was already in use. So the application using that port (which was exim4 in this case) was killed and loomio was restarted again.

[[Category: Services]]

Loomio/Archive 1

2020-05-04T16:59:44Z

AkhilVarkey: AkhilVarkey moved page Loomio/archive/1 to Loomio/archive 1 without leaving a redirect: Archiving old loomio information

Loomio/Archive 1

2020-05-04T16:58:33Z

AkhilVarkey: AkhilVarkey moved page Loomio to Loomio/archive/1 without leaving a redirect

Poddery - Diaspora, Matrix and XMPP

2020-04-22T09:15:05Z

AkhilVarkey:

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-05-02T09:54:50Z

AkhilVarkey: /* TLS */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run <code>/var/www/get-riot</code> and reload <code>nginx</code>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public --expand -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-05-02T09:47:22Z

AkhilVarkey: /* Hardening checklist */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run <code>/var/www/get-riot</code> and reload <code>nginx</code>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Loomio/Archive 1

2019-03-24T14:56:03Z

AkhilVarkey: /* Loomio Deployment Reference */

Loomio/Archive 1

2019-03-24T14:55:07Z

AkhilVarkey:

[https://freesoftwareindia.org/ Loomio] instance at https://codema.fsci.org.in. Loomio is a '''co'''llaborative '''de'''cision-'''ma'''king platform (hence we're using the name '''codema''') where users can initiate discussions and put up proposals.

== Hosting ==
We were now on a [https://www.scaleway.com/virtual-cloud-servers START1-S instance] virtual cloud server with the following specs:

* 2 x86-64 bit Cores
* 2 GB Memory
* 50GB SSD Disk
* 200 Mbit/s Bandwidth
* 1 Reserved IP (v4)
* €3.99/Month

== Coordination ==

* Hangout with us in our Matrix room [https://matrix.to/#/#codema:poddery.com #codema:poddery.com]

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

== History ==
We were on a [https://www.scaleway.com/virtual-cloud-servers VC1S instance] virtual cloud server with the following specs until 31/01/2019.

* 2 x86-64 bit Cores
* 2GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 200Mbit/s Unmetered bandwidth
* €2.99/Month

On 01/02/2019 our server was taken down by [https://scaleway.com Scaleway] quoting payment issues. Payment was failing even after updating the credit card details and following that our server got deleted without proper notifications from Scaleway's side. Screenshot of the email from Scaleway attached below. Fortunately we were provided with a snapshot of the server from which we were able to recover codema to a new server.

=== Loomio Deployment Reference ===

Loomio was deployed and updated on codema.fsci.org.in after following this Loomio Deployment Reference - https://github.com/loomio/loomio-deploy

=== Codema recovery process ===

''Here's a brief description of how codema was recovered after the server take down on 01/01/2019:''

Under 'Snapshots' tab in the Scaleway dashboard we were provided with the snapshot (backup) of our codema server. A system image was created from this snapshot and it was used to create a new server with similar specifications. We lost our public IP along with the old server, so a new IP was assigned to the server and then updated the DNS A record of codema.fsci.org.in to point to this new IP. Once the server was up loomio was restarted using the following commands from the loomio installation directory:

docker-compose down
docker-compose up -d

The logs were checked for errors using the following command:

docker-compose logs -f

Loomio wasn't getting started saying the port 25 was already in use. So the application using that port (which was exim4 in this case) was killed and loomio was restarted again.

[[Category: Services]]

Loomio/Archive 1

2019-03-24T14:53:06Z

AkhilVarkey: /* History */

[https://freesoftwareindia.org/ Loomio] instance at https://codema.fsci.org.in. Loomio is a '''co'''llaborative '''de'''cision-'''ma'''king platform (hence we're using the name '''codema''') where users can initiate discussions and put up proposals.

== Hosting ==
We were now on a [https://www.scaleway.com/virtual-cloud-servers START1-S instance] virtual cloud server with the following specs:

* 2 x86-64 bit Cores
* 2 GB Memory
* 50GB SSD Disk
* 200 Mbit/s Bandwidth
* 1 Reserved IP (v4)
* €3.99/Month

== Coordination ==

* Hangout with us in our Matrix room [https://matrix.to/#/#codema:poddery.com #codema:poddery.com]

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

== History ==
We were on a [https://www.scaleway.com/virtual-cloud-servers VC1S instance] virtual cloud server with the following specs until 31/01/2019.

* 2 x86-64 bit Cores
* 2GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 200Mbit/s Unmetered bandwidth
* €2.99/Month

On 01/02/2019 our server was taken down by [https://scaleway.com Scaleway] quoting payment issues. Payment was failing even after updating the credit card details and following that our server got deleted without proper notifications from Scaleway's side. Screenshot of the email from Scaleway attached below. Fortunately we were provided with a snapshot of the server from which we were able to recover codema to a new server.

=== Codema recovery process ===

''Here's a brief description of how codema was recovered after the server take down on 01/01/2019:''

Under 'Snapshots' tab in the Scaleway dashboard we were provided with the snapshot (backup) of our codema server. A system image was created from this snapshot and it was used to create a new server with similar specifications. We lost our public IP along with the old server, so a new IP was assigned to the server and then updated the DNS A record of codema.fsci.org.in to point to this new IP. Once the server was up loomio was restarted using the following commands from the loomio installation directory:

docker-compose down
docker-compose up -d

The logs were checked for errors using the following command:

docker-compose logs -f

Loomio wasn't getting started saying the port 25 was already in use. So the application using that port (which was exim4 in this case) was killed and loomio was restarted again.

[[Category: Services]]

Infrastructure

2019-03-23T14:24:17Z

AkhilVarkey: /* Our machines */

== Our machines ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|mahishasura
|http://mahishasura.pxq.in
|Test machine
|
|-
|[[Poddery_-_Diaspora,_Matrix_and_XMPP|poddery]]
|https://poddery.com
|diaspora pod, matrix homeserver and xmpp server (production)
|
|-
|lists
|https://lists.fsci.org.in
|Mailing list service (production)
|
|-
|Loomio
|https://codema.fsci.org.in
|Loomio service
|
|-
|git
|https://git.fosscommunity.in
|Gitlab instance (production)
|
|-
|Wikimedia
|https://wiki.fsci.org.in/index.php
|Wikimedia instance (production)
|
|-
|Peer tube
|https://videos.fsci.org.in
|Peer tube instance
|Currently down
|-
|Monitor
|https://monitor.fsci.org.in
|For monitoring various services, (grafana)
|Currently down
|-
|Open Source Event Manager
|https://events.fsci.org.in
|Event management service (production)
|
|-
|mayasura
|62.210.83.200 ssh:19022
|Backup server for diasp.in
|
|-
|bhasmasura
|62.210.83.200 ssh:12022 postgres:12432
|Backup server for git.fosscommunity.in
|
|-
|banasura
|62.210.83.200 ssh:11022 mysql:
|Backup server for poddery.com
|
|-
|}

== Other services endorsed by FSCI ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|Discourse
|https://freesoftwareindia.org/
|Discourse instance
|
|-
|Mastodon
|https://social.masto.host/
|Mastodon instance
|
|-
|libre infra
|https://libreinfra.org
|Email, Next Cloud services
|Public registration is closed
|-
|}

Infrastructure

2019-03-23T14:10:17Z

AkhilVarkey:

== Our machines ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|mahishasura
|http://mahishasura.pxq.in
|Test machine
|
|-
|[[Poddery_-_Diaspora,_Matrix_and_XMPP|poddery]]
|https://poddery.com
|diaspora pod, matrix homeserver and xmpp server (production)
|
|-
|lists
|https://lists.fsci.org.in
|Mailing list service (production)
|
|-
|Loomio
|https://codema.fsci.org.in
|Loomio service
|
|-
|git
|https://git.fosscommunity.in
|Gitlab instance (production)
|
|-
|Wikimedia
|https://wiki.fsci.org.in/index.php
|Wikimedia instance (production)
|
|-
|Peer tube
|https://videos.fsci.org.in
|Peer tube instance
|Currently down
|-
|Monitor
|https://monitor.fsci.org.in
|For monitoring various services, (grafana)
|Currently down
|-
|Open Source Event Manager
|https://events.fsci.org.in
|Event management service (production)
|
|-
|mayasura
|62.210.83.200 ssh:
|Backup server for diasp.in
|
|-
|bhasmasura
|62.210.83.200 ssh:12022 postgres:12432
|Backup server for git.fosscommunity.in
|
|-
|banasura
|62.210.83.200 ssh:11022 mysql:
|Backup server for poddery.com
|
|-
|}

== Other services endorsed by FSCI ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|Discourse
|https://freesoftwareindia.org/
|Discourse instance
|
|-
|Mastodon
|https://social.masto.host/
|Mastodon instance
|
|-
|libre infra
|https://libreinfra.org
|Email, Next Cloud services
|Public registration is closed
|-
|}

Infrastructure

2019-03-23T13:57:52Z

AkhilVarkey: /* Our machines */

== Our machines ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|mahishasura
|http://mahishasura.pxq.in
|Test machine
|
|-
|Poddery
|https://poddery.com
|diaspora pod, matrix homeserver and xmpp server (production)
|
|-
|lists
|https://lists.fsci.org.in
|Mailing list service (production)
|
|-
|Loomio
|https://codema.fsci.org.in
|Loomio service
|
|-
|git
|https://git.fosscommunity.in
|Gitlab instance (production)
|
|-
|Wikimedia
|https://wiki.fsci.org.in/index.php
|Wikimedia instance (production)
|
|-
|Peer tube
|https://videos.fsci.org.in
|Peer tube instance
|Currently down
|-
|Monitor
|https://monitor.fsci.org.in
|For monitoring various services, (grafana)
|Currently down
|-
|Open Source Event Manager
|https://events.fsci.org.in
|Event management service (production)
|
|-
|banasura
|62.210.83.200
|Backup server for poddery.com
|
|-
|bhasmasura
|62.210.83.200 ssh:12022 postgres:12432
|Backup server for git.fosscommunity.in
|
|-
|mayasura
|62.210.83.200 ssh:
|Backup server for diasp.in
|
|-
|}

== Other services endorsed by FSCI ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|Discourse
|https://freesoftwareindia.org/
|Discourse instance
|
|-
|Mastodon
|https://social.masto.host/
|Mastodon instance
|
|-
|libre infra
|https://libreinfra.org
|Email, Next Cloud services
|Public registration is closed
|-
|}