FSCI Wiki - User contributions [en]

Discourse

2022-10-17T17:29:47Z

Bady: Bady moved page Discourse to Discourse/Archive: No longer maintained

#REDIRECT [[Discourse/Archive]]

Discourse/Archive

2022-10-17T17:29:47Z

Bady: Bady moved page Discourse to Discourse/Archive: No longer maintained

India OS Forum - https://forum.indiaos.in - Maintained by Zerodha and Frappe Technologies

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2022-07-28T17:18:06Z

Bady: Add workaround to access XMPP service when diaspora account is closed

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Budget

2021-12-02T09:23:04Z

Bady: /* videos.fsci.org.in */

Budget details of FSCI managed services. See [https://www.loomio.org/d/7znkc0RX/annual-budget-for-fsci-looking-for-volunteers this discussion for the background].

== Domains ==

== Services ==

=== poddery.com ===

Hosting charges: €25.21 per month.

See the [[poddery]] page for more details.

=== codema.in ===

Hosting charges: €2.99 * 12 = €35.88 per year (₹239.04 * 12 = ₹2868.48, calculated as of 28/06/2018)

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €35.88
| €35.88
|-
| December 2019
| €2.19
|
| €33.69
|-
| January 2020
| €1.76
|
| €31.93
|-
| February 2020
| €2.99
|
| €28.94
|-
| March 2020
| €3.01
|
| €25.93
|-
| April 2020
| €2.99
|
| €22.94
|-
| May 2020
| €2.99
|
| €19.95
|-
| June 2020
| €2.99
|
| €16.96
|-
| July 2020
| €2.99
|
| €13.97
|-
| August 2020*
| €4.99
|
| €8.98
|-
| September 2020
| €4.99
|
| €3.99
|-
| October 2020
| €4.99
|
| -€1.00
|-
| November 2020
| €4.99
|
| -€5.99
|-
| December 2020
| €4.99
|
| -€10.98
|-
| January 2021
| €4.99
|
| -€15.97
|-
| February 2021
| €4.99
|
| -€20.96
|-
| March 2021
| €4.99
| €42
| €16.05
|-
| April 2021
| €4.99
|
| €11.06
|-
| May 2021
| €4.99
|
| €6.07
|-
| June 2021
| €4.99
|
| €1.08
|-
| July 2021
| €4.99
|
| -€3.91
|-
| August 2021
| €4.99
|
| -€8.90
|-
| September 2021
| €4.99
|
| -€13.89
|-
| October 2021
| €4.99
|
| -€18.88
|-
| November 2021
| €4.99
|
| -€23.87
|}

* *Scaleway increased hosting charges from €2.99 to €4.99 from August 2020

=== lists.fsci.org.in ===

{| class="wikitable"
|+ mailman3 expense
|-
! Month
! Debit
! Credit
! Balance
|-
| June 2018
|
| ₹ 2855.00
| ₹ 2855.00
|-
| July 2018
| ₹ 53.62
|
| ₹ 2801.38
|-
| August 2018
| ₹ 279.82
|
| ₹ 2521.56
|-
| September 2018
| ₹ 259.20
|
| ₹ 2262.36
|-
| October 2018
| ₹ 265.07
|
| ₹ 1997.29
|-
| November 2018
| ₹ 261.14
|
| ₹ 1736.15
|-
| December 2018
| ₹ 250.25
|
| ₹ 1485.90
|-
| January 2019
| ₹ 253.69
|
| ₹ 1232.21
|-
| February 2019
| ₹ 389.31
|
| ₹ 843.21
|-
| March 2019
| ₹ 363.68
| ₹ 4670.00
| ₹ 5149.53
|-
| April 2019
| ₹ 317.53
|
| ₹ 4832.00
|-
| May 2019
| ₹ 317.53
|
| ₹ 4514.47
|-
| June 2019
| ₹ 317.53
|
| ₹ 4196.94
|-
| July 2019
| ₹ 317.53
|
| ₹ 3879.41
|-
| August 2019
| ₹ 317.53
|
| ₹ 3561.88
|-
| September 2019
| ₹ 317.53
|
| ₹ 3244.35
|-
| October 2019
| ₹ 317.53
|
| ₹ 2926.77
|-
| November 2019
| ₹ 316.8
|
| ₹ 2609.97
|-
| December 2019
| ₹ 318.8
|
| ₹ 2291.17
|-
| January 2020
| ₹ 314.8
|
| ₹ 1976.37
|-
| February 2020
| ₹ 317.5
|
| ₹ 1658.87
|-
| March 2020
| ₹ 332.7
|
| ₹ 1326.17
|-
| April 2020
| ₹ 329.9
|
| ₹ 996.27
|-
| May 2020
| ₹ 335.8
|
| ₹ 660.47
|-
| June 2020
| ₹ 263.01
|
| ₹ 397.46
|}

=== videos.fsci.org.in ===

Hosting charges: €3.49 * 12 = €41.88 per year

{| class="wikitable"
|+ peertube expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €47.88
| €47.88
|-
| July 2020
| €3.49
|
| €44.39
|-
| August 2020
| €3.49
|
| €40.9
|-
| September 2020
| €3.49
|
| €37.41
|-
| October 2020
| €3.49
|
| €33.92
|-
| November 2020
| €3.49
|
| €30.43
|-
| December 2020
| €3.49
|
| €26.94
|-
| January 2021
| €3.49
|
| €23.45
|-
| February 2021
| €3.49
|
| €19.96
|-
| March 2021*
| €3.49
| €42 + €15
| €73.47
|-
| April 2021
| €5.09
|
| €68.38
|-
| May 2021
| €5.09
|
| €63.29
|-
| June 2021
| €5.09
|
| €58.20
|-
| July 2021
| €5.09
|
| €53.11
|-
| August 2021
| €5.09
|
| €48.02
|-
| September 2021
| €5.09
|
| €42.93
|-
| October 2021
| €5.09
|
| €37.84
|-
| November 2021
| €5.09
|
| €32.75
|}

* *Additional 40G storage space added which adds €1.6 to the monthly cost. The total monthly cost is now €5.09. Got €15 sponsorship for the storage space and €42 for hosting charges for one year.

=== git.fosscommunity.in ===

Hosting charges: $47.28 per month

{| class="wikitable"
|+ git.fosscommunity.in expense
|-
! Month
! Debit
! Credit
! Balance
|-
| October 2021
| $30.00
|
| 0
|-
| November 2021
| $104.56
| $610
| $505.44
|}

[[Category: Services]]

Budget

2021-12-02T09:01:35Z

Bady: /* codema.in */

Budget details of FSCI managed services. See [https://www.loomio.org/d/7znkc0RX/annual-budget-for-fsci-looking-for-volunteers this discussion for the background].

== Domains ==

== Services ==

=== poddery.com ===

Hosting charges: €25.21 per month.

See the [[poddery]] page for more details.

=== codema.in ===

Hosting charges: €2.99 * 12 = €35.88 per year (₹239.04 * 12 = ₹2868.48, calculated as of 28/06/2018)

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €35.88
| €35.88
|-
| December 2019
| €2.19
|
| €33.69
|-
| January 2020
| €1.76
|
| €31.93
|-
| February 2020
| €2.99
|
| €28.94
|-
| March 2020
| €3.01
|
| €25.93
|-
| April 2020
| €2.99
|
| €22.94
|-
| May 2020
| €2.99
|
| €19.95
|-
| June 2020
| €2.99
|
| €16.96
|-
| July 2020
| €2.99
|
| €13.97
|-
| August 2020*
| €4.99
|
| €8.98
|-
| September 2020
| €4.99
|
| €3.99
|-
| October 2020
| €4.99
|
| -€1.00
|-
| November 2020
| €4.99
|
| -€5.99
|-
| December 2020
| €4.99
|
| -€10.98
|-
| January 2021
| €4.99
|
| -€15.97
|-
| February 2021
| €4.99
|
| -€20.96
|-
| March 2021
| €4.99
| €42
| €16.05
|-
| April 2021
| €4.99
|
| €11.06
|-
| May 2021
| €4.99
|
| €6.07
|-
| June 2021
| €4.99
|
| €1.08
|-
| July 2021
| €4.99
|
| -€3.91
|-
| August 2021
| €4.99
|
| -€8.90
|-
| September 2021
| €4.99
|
| -€13.89
|-
| October 2021
| €4.99
|
| -€18.88
|-
| November 2021
| €4.99
|
| -€23.87
|}

* *Scaleway increased hosting charges from €2.99 to €4.99 from August 2020

=== lists.fsci.org.in ===

{| class="wikitable"
|+ mailman3 expense
|-
! Month
! Debit
! Credit
! Balance
|-
| June 2018
|
| ₹ 2855.00
| ₹ 2855.00
|-
| July 2018
| ₹ 53.62
|
| ₹ 2801.38
|-
| August 2018
| ₹ 279.82
|
| ₹ 2521.56
|-
| September 2018
| ₹ 259.20
|
| ₹ 2262.36
|-
| October 2018
| ₹ 265.07
|
| ₹ 1997.29
|-
| November 2018
| ₹ 261.14
|
| ₹ 1736.15
|-
| December 2018
| ₹ 250.25
|
| ₹ 1485.90
|-
| January 2019
| ₹ 253.69
|
| ₹ 1232.21
|-
| February 2019
| ₹ 389.31
|
| ₹ 843.21
|-
| March 2019
| ₹ 363.68
| ₹ 4670.00
| ₹ 5149.53
|-
| April 2019
| ₹ 317.53
|
| ₹ 4832.00
|-
| May 2019
| ₹ 317.53
|
| ₹ 4514.47
|-
| June 2019
| ₹ 317.53
|
| ₹ 4196.94
|-
| July 2019
| ₹ 317.53
|
| ₹ 3879.41
|-
| August 2019
| ₹ 317.53
|
| ₹ 3561.88
|-
| September 2019
| ₹ 317.53
|
| ₹ 3244.35
|-
| October 2019
| ₹ 317.53
|
| ₹ 2926.77
|-
| November 2019
| ₹ 316.8
|
| ₹ 2609.97
|-
| December 2019
| ₹ 318.8
|
| ₹ 2291.17
|-
| January 2020
| ₹ 314.8
|
| ₹ 1976.37
|-
| February 2020
| ₹ 317.5
|
| ₹ 1658.87
|-
| March 2020
| ₹ 332.7
|
| ₹ 1326.17
|-
| April 2020
| ₹ 329.9
|
| ₹ 996.27
|-
| May 2020
| ₹ 335.8
|
| ₹ 660.47
|-
| June 2020
| ₹ 263.01
|
| ₹ 397.46
|}

=== videos.fsci.org.in ===

Hosting charges: €3.49 * 12 = €41.88 per year

{| class="wikitable"
|+ peertube expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €47.88
| €47.88
|-
| July 2020
| €3.49
|
| €44.39
|-
| August 2020
| €3.49
|
| €40.9
|-
| September 2020
| €3.49
|
| €37.41
|-
| October 2020
| €3.49
|
| €33.92
|-
| November 2020
| €3.49
|
| €30.43
|-
| December 2020
| €3.49
|
| €26.94
|-
| January 2021
| €3.49
|
| €23.45
|-
| February 2021
| €3.49
|
| €19.96
|-
| March 2021*
| €3.49
| €42 + €15
| €73.47
|}

* *Additional 40G storage space added which adds €1.6 to the monthly cost. The total monthly cost is now €5.09. Got €15 sponsorship for the storage space and €42 for hosting charges for one year.

=== git.fosscommunity.in ===

Hosting charges: $47.28 per month

{| class="wikitable"
|+ git.fosscommunity.in expense
|-
! Month
! Debit
! Credit
! Balance
|-
| October 2021
| $30.00
|
| 0
|-
| November 2021
| $104.56
| $610
| $505.44
|}

[[Category: Services]]

Infrastructure

2021-04-01T14:01:41Z

Bady: Add jitsi

== Our machines ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|mahishasura
|http://mahishasura.pxq.in
|Test machine
|
|-
|[[Poddery_-_Diaspora,_Matrix_and_XMPP|poddery]]
|https://poddery.com
|diaspora pod, matrix homeserver and xmpp server (production)
|
|-
|lists
|https://lists.fsci.in
|Mailing list service (production)
|
|-
|Loomio
|https://codema.in
|Loomio service
|
|-
|git
|https://git.fosscommunity.in
|Gitlab instance (production)
|
|-
|Mediawiki
|https://wiki.fsci.org.in/index.php
|Mediawiki instance (production)
|
|-
|Peer tube
|https://videos.fsci.in
|Peer tube instance
|
|-
|Jitsi
|https://meet.fsci.in
|Jitsi instance
|
|-
|Monitor
|https://monitor.fsci.in
|For monitoring various services, (grafana)
|
|-
|Open Source Event Manager
|https://events.fsci.org.in
|Event management service (production)
|Currently down
|-
|mayasura
|62.210.83.200 ssh:19022
|Backup server for diasp.in
|
|-
|bhasmasura
|62.210.83.200 ssh:12022 postgres:12432
|Backup server for git.fosscommunity.in
|
|-
|banasura
|62.210.83.200 ssh:11022 mysql:
|Backup server for poddery.com
|
|-
|}

== Other services endorsed by FSCI ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|Discourse
|https://freesoftwareindia.org/
|Discourse instance
|
|-
|Mastodon
|https://social.masto.host/
|Mastodon instance
|
|-
|libre infra
|https://libreinfra.org
|Email, Next Cloud services
|Public registration is closed
|-
|}

Infrastructure

2021-04-01T13:53:46Z

Bady: Update domain names and uptime status

== Our machines ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|mahishasura
|http://mahishasura.pxq.in
|Test machine
|
|-
|[[Poddery_-_Diaspora,_Matrix_and_XMPP|poddery]]
|https://poddery.com
|diaspora pod, matrix homeserver and xmpp server (production)
|
|-
|lists
|https://lists.fsci.in
|Mailing list service (production)
|
|-
|Loomio
|https://codema.in
|Loomio service
|
|-
|git
|https://git.fosscommunity.in
|Gitlab instance (production)
|
|-
|Mediawiki
|https://wiki.fsci.org.in/index.php
|Mediawiki instance (production)
|
|-
|Peer tube
|https://videos.fsci.in
|Peer tube instance
|
|-
|Monitor
|https://monitor.fsci.in
|For monitoring various services, (grafana)
|
|-
|Open Source Event Manager
|https://events.fsci.org.in
|Event management service (production)
|Currently down
|-
|mayasura
|62.210.83.200 ssh:19022
|Backup server for diasp.in
|
|-
|bhasmasura
|62.210.83.200 ssh:12022 postgres:12432
|Backup server for git.fosscommunity.in
|
|-
|banasura
|62.210.83.200 ssh:11022 mysql:
|Backup server for poddery.com
|
|-
|}

== Other services endorsed by FSCI ==

{| class="wikitable" style="margin:auto"
! Host name
! Domain
! Purpose
! Remarks
|-
|Discourse
|https://freesoftwareindia.org/
|Discourse instance
|
|-
|Mastodon
|https://social.masto.host/
|Mastodon instance
|
|-
|libre infra
|https://libreinfra.org
|Email, Next Cloud services
|Public registration is closed
|-
|}

Budget

2021-03-30T19:20:16Z

Bady: Fix a minor typo

Budget details of FSCI managed services. See [https://www.loomio.org/d/7znkc0RX/annual-budget-for-fsci-looking-for-volunteers this discussion for the background].

== Domains ==

== Services ==

=== poddery.com ===

Hosting charges: €25.21 per month.

See the [[poddery]] page for more details.

=== codema.in ===

Hosting charges: €2.99 * 12 = €35.88 per year (₹239.04 * 12 = ₹2868.48, calculated as of 28/06/2018)

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €35.88
| €35.88
|-
| December 2019
| €2.19
|
| €33.69
|-
| January 2020
| €1.76
|
| €31.93
|-
| February 2020
| €2.99
|
| €28.94
|-
| March 2020
| €3.01
|
| €25.93
|-
| April 2020
| €2.99
|
| €22.94
|-
| May 2020
| €2.99
|
| €19.95
|-
| June 2020
| €2.99
|
| €16.96
|-
| July 2020
| €2.99
|
| €13.97
|-
| August 2020*
| €4.99
|
| €8.98
|-
| September 2020
| €4.99
|
| €3.99
|-
| October 2020
| €4.99
|
| -€1.00
|-
| November 2020
| €4.99
|
| -€5.99
|-
| December 2020
| €4.99
|
| -€10.98
|-
| January 2021
| €4.99
|
| -€15.97
|-
| February 2021
| €4.99
|
| -€20.96
|-
| March 2021
| €4.99
| €42
| €16.05
|}

* *Scaleway increased hosting charges from €2.99 to €4.99 from August 2020

=== lists.fsci.org.in ===

{| class="wikitable"
|+ mailman3 expense
|-
! Month
! Debit
! Credit
! Balance
|-
| June 2018
|
| ₹ 2855.00
| ₹ 2855.00
|-
| July 2018
| ₹ 53.62
|
| ₹ 2801.38
|-
| August 2018
| ₹ 279.82
|
| ₹ 2521.56
|-
| September 2018
| ₹ 259.20
|
| ₹ 2262.36
|-
| October 2018
| ₹ 265.07
|
| ₹ 1997.29
|-
| November 2018
| ₹ 261.14
|
| ₹ 1736.15
|-
| December 2018
| ₹ 250.25
|
| ₹ 1485.90
|-
| January 2019
| ₹ 253.69
|
| ₹ 1232.21
|-
| February 2019
| ₹ 389.31
|
| ₹ 843.21
|-
| March 2019
| ₹ 363.68
| ₹ 4670.00
| ₹ 5149.53
|-
| April 2019
| ₹ 317.53
|
| ₹ 4832.00
|-
| May 2019
| ₹ 317.53
|
| ₹ 4514.47
|-
| June 2019
| ₹ 317.53
|
| ₹ 4196.94
|-
| July 2019
| ₹ 317.53
|
| ₹ 3879.41
|-
| August 2019
| ₹ 317.53
|
| ₹ 3561.88
|-
| September 2019
| ₹ 317.53
|
| ₹ 3244.35
|-
| October 2019
| ₹ 317.53
|
| ₹ 2926.77
|-
| November 2019
| ₹ 316.8
|
| ₹ 2609.97
|-
| December 2019
| ₹ 318.8
|
| ₹ 2291.17
|-
| January 2020
| ₹ 314.8
|
| ₹ 1976.37
|-
| February 2020
| ₹ 317.5
|
| ₹ 1658.87
|-
| March 2020
| ₹ 332.7
|
| ₹ 1326.17
|-
| April 2020
| ₹ 329.9
|
| ₹ 996.27
|-
| May 2020
| ₹ 335.8
|
| ₹ 660.47
|-
| June 2020
| ₹ 263.01
|
| ₹ 397.46
|}

=== videos.fsci.org.in ===

Hosting charges: €3.49 * 12 = €41.88 per year

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €47.88
| €47.88
|-
| July 2020
| €3.49
|
| €44.39
|-
| August 2020
| €3.49
|
| €40.9
|-
| September 2020
| €3.49
|
| €37.41
|-
| October 2020
| €3.49
|
| €33.92
|-
| November 2020
| €3.49
|
| €30.43
|-
| December 2020
| €3.49
|
| €26.94
|-
| January 2021
| €3.49
|
| €23.45
|-
| February 2021
| €3.49
|
| €19.96
|-
| March 2021*
| €3.49
| €42 + €15
| €73.47
|}

* *Additional 40G storage space added which adds €1.6 to the monthly cost. The total monthly cost is now €5.09. Got €15 sponsorship for the storage space and €42 for hosting charges for one year.

[[Category: Services]]

Budget

2021-03-30T19:19:25Z

Bady:

Budget details of FSCI managed services. See [https://www.loomio.org/d/7znkc0RX/annual-budget-for-fsci-looking-for-volunteers this discussion for the background].

== Domains ==

== Services ==

=== poddery.com ===

Hosting charges: €25.21 per month.

See the [[poddery]] page for more details.

=== codema.in ===

Hosting charges: €2.99 * 12 = €35.88 per year (₹239.04 * 12 = ₹2868.48, calculated as of 28/06/2018)

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €35.88
| €35.88
|-
| December 2019
| €2.19
|
| €33.69
|-
| January 2020
| €1.76
|
| €31.93
|-
| February 2020
| €2.99
|
| €28.94
|-
| March 2020
| €3.01
|
| €25.93
|-
| April 2020
| €2.99
|
| €22.94
|-
| May 2020
| €2.99
|
| €19.95
|-
| June 2020
| €2.99
|
| €16.96
|-
| July 2020
| €2.99
|
| €13.97
|-
| August 2020*
| €4.99
|
| €8.98
|-
| September 2020
| €4.99
|
| €3.99
|-
| October 2020
| €4.99
|
| -€1.00
|-
| November 2020
| €4.99
|
| -€5.99
|-
| December 2020
| €4.99
|
| -€10.98
|-
| January 2021
| €4.99
|
| -€15.97
|-
| February 2021
| €4.99
|
| -€20.96
|-
| March 2021
| €4.99
| €42
| €16.05
|}

* *Scaleway increased hosting charges from €2.99 to €4.99 from August 2020

=== lists.fsci.org.in ===

{| class="wikitable"
|+ mailman3 expense
|-
! Month
! Debit
! Credit
! Balance
|-
| June 2018
|
| ₹ 2855.00
| ₹ 2855.00
|-
| July 2018
| ₹ 53.62
|
| ₹ 2801.38
|-
| August 2018
| ₹ 279.82
|
| ₹ 2521.56
|-
| September 2018
| ₹ 259.20
|
| ₹ 2262.36
|-
| October 2018
| ₹ 265.07
|
| ₹ 1997.29
|-
| November 2018
| ₹ 261.14
|
| ₹ 1736.15
|-
| December 2018
| ₹ 250.25
|
| ₹ 1485.90
|-
| January 2019
| ₹ 253.69
|
| ₹ 1232.21
|-
| February 2019
| ₹ 389.31
|
| ₹ 843.21
|-
| March 2019
| ₹ 363.68
| ₹ 4670.00
| ₹ 5149.53
|-
| April 2019
| ₹ 317.53
|
| ₹ 4832.00
|-
| May 2019
| ₹ 317.53
|
| ₹ 4514.47
|-
| June 2019
| ₹ 317.53
|
| ₹ 4196.94
|-
| July 2019
| ₹ 317.53
|
| ₹ 3879.41
|-
| August 2019
| ₹ 317.53
|
| ₹ 3561.88
|-
| September 2019
| ₹ 317.53
|
| ₹ 3244.35
|-
| October 2019
| ₹ 317.53
|
| ₹ 2926.77
|-
| November 2019
| ₹ 316.8
|
| ₹ 2609.97
|-
| December 2019
| ₹ 318.8
|
| ₹ 2291.17
|-
| January 2020
| ₹ 314.8
|
| ₹ 1976.37
|-
| February 2020
| ₹ 317.5
|
| ₹ 1658.87
|-
| March 2020
| ₹ 332.7
|
| ₹ 1326.17
|-
| April 2020
| ₹ 329.9
|
| ₹ 996.27
|-
| May 2020
| ₹ 335.8
|
| ₹ 660.47
|-
| June 2020
| ₹ 263.01
|
| ₹ 397.46
|}

=== videos.fsci.org.in ===

Hosting charges: €3.49 * 12 = €41.88 per year

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €47.88
| €47.88
|-
| July 2020
| €3.49
|
| €44.39
|-
| August 2020
| €3.49
|
| €40.9
|-
| September 2020
| €3.49
|
| €37.41
|-
| October 2020
| €3.49
|
| €33.92
|-
| November 2020
| €3.49
|
| €30.43
|-
| December 2020
| €3.49
|
| €26.94
|-
| January 2021
| €3.49
|
| €23.45
|-
| February 2021
| €3.49
|
| €19.96
|-
| March 2021*
| €3.49
| €42 + €15
| €73.47
|}

* *Additional 40G storage space added which add €1.6 to the monthly cost. The total monthly cost is now €5.09. Got €15 sponsorship for the storage space and €42 for hosting charges for one year.

[[Category: Services]]

Budget

2021-03-30T19:03:30Z

Bady: Update codema budget

Budget details of FSCI managed services. See [https://www.loomio.org/d/7znkc0RX/annual-budget-for-fsci-looking-for-volunteers this discussion for the background].

== Domains ==

== Services ==

=== poddery.com ===

Hosting charges: €25.21 per month.

See the [[poddery]] page for more details.

=== codema.in ===

Hosting charges: €2.99 * 12 = €35.88 per year (₹239.04 * 12 = ₹2868.48, calculated as of 28/06/2018)

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €35.88
| €35.88
|-
| December 2019
| €2.19
|
| €33.69
|-
| January 2020
| €1.76
|
| €31.93
|-
| February 2020
| €2.99
|
| €28.94
|-
| March 2020
| €3.01
|
| €25.93
|-
| April 2020
| €2.99
|
| €22.94
|-
| May 2020
| €2.99
|
| €19.95
|-
| June 2020
| €2.99
|
| €16.96
|-
| July 2020
| €2.99
|
| €13.97
|-
| August 2020*
| €4.99
|
| €8.98
|-
| September 2020
| €4.99
|
| €3.99
|-
| October 2020
| €4.99
|
| -€1.00
|-
| November 2020
| €4.99
|
| -€5.99
|-
| December 2020
| €4.99
|
| -€10.98
|-
| January 2021
| €4.99
|
| -€15.97
|-
| February 2021
| €4.99
|
| -€20.96
|-
| March 2021
| €4.99
| €42
| €16.05
|}

* *Scaleway increased hosting charges from €2.99 to €4.99 from August 2020

=== lists.fsci.org.in ===

{| class="wikitable"
|+ mailman3 expense
|-
! Month
! Debit
! Credit
! Balance
|-
| June 2018
|
| ₹ 2855.00
| ₹ 2855.00
|-
| July 2018
| ₹ 53.62
|
| ₹ 2801.38
|-
| August 2018
| ₹ 279.82
|
| ₹ 2521.56
|-
| September 2018
| ₹ 259.20
|
| ₹ 2262.36
|-
| October 2018
| ₹ 265.07
|
| ₹ 1997.29
|-
| November 2018
| ₹ 261.14
|
| ₹ 1736.15
|-
| December 2018
| ₹ 250.25
|
| ₹ 1485.90
|-
| January 2019
| ₹ 253.69
|
| ₹ 1232.21
|-
| February 2019
| ₹ 389.31
|
| ₹ 843.21
|-
| March 2019
| ₹ 363.68
| ₹ 4670.00
| ₹ 5149.53
|-
| April 2019
| ₹ 317.53
|
| ₹ 4832.00
|-
| May 2019
| ₹ 317.53
|
| ₹ 4514.47
|-
| June 2019
| ₹ 317.53
|
| ₹ 4196.94
|-
| July 2019
| ₹ 317.53
|
| ₹ 3879.41
|-
| August 2019
| ₹ 317.53
|
| ₹ 3561.88
|-
| September 2019
| ₹ 317.53
|
| ₹ 3244.35
|-
| October 2019
| ₹ 317.53
|
| ₹ 2926.77
|-
| November 2019
| ₹ 316.8
|
| ₹ 2609.97
|-
| December 2019
| ₹ 318.8
|
| ₹ 2291.17
|-
| January 2020
| ₹ 314.8
|
| ₹ 1976.37
|-
| February 2020
| ₹ 317.5
|
| ₹ 1658.87
|-
| March 2020
| ₹ 332.7
|
| ₹ 1326.17
|-
| April 2020
| ₹ 329.9
|
| ₹ 996.27
|-
| May 2020
| ₹ 335.8
|
| ₹ 660.47
|-
| June 2020
| ₹ 263.01
|
| ₹ 397.46
|}

=== videos.fsci.org.in ===

Hosting charges: €3.49 * 12 = €41.88 per year

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €47.88
| €47.88
|-
| July 2020
| €3.49
|
| €44.39
|-
| August 2020
| €3.49
|
| €40.9
|-
| September 2020
| €3.49
|
| €37.41
|-
| October 2020
| €3.49
|
| €33.92
|-
| November 2020
| €3.49
|
| €30.43
|-
| December 2020
| €3.49
|
| €26.94
|-
| January 2021
| €3.49
|
| €23.45
|-
| February 2021
| €3.49
|
| €19.96
|-
| March 2021*
| €3.49
| €42 + €15
| €73.47
|}

* *Additional 40G storage space added which add 1.6 to the monthly cost. The total monthly cost is now €5.09. Got €15 sponsorship for the storage space and €42 for hosting charges for one year.

[[Category: Services]]

Budget

2021-03-30T18:54:14Z

Bady: Update peertube budget with sponsorship details

Budget details of FSCI managed services. See [https://www.loomio.org/d/7znkc0RX/annual-budget-for-fsci-looking-for-volunteers this discussion for the background].

== Domains ==

== Services ==

=== poddery.com ===

Hosting charges: €25.21 per month.

See the [[poddery]] page for more details.

=== codema.in ===

Hosting charges: €2.99 * 12 = €35.88 per year (₹239.04 * 12 = ₹2868.48, calculated as of 28/06/2018)

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €35.88
| €35.88
|-
| December 2019
| €2.19
|
| €33.69
|-
| January 2020
| €1.76
|
| €31.93
|-
| February 2020
| €2.99
|
| €28.94
|-
| March 2020
| €3.01
|
| €25.93
|-
| April 2020
| €2.99
|
| €22.94
|-
| May 2020
| €2.99
|
| €19.95
|-
| June 2020
| €2.99
|
| €16.96
|-
| July 2020
| €2.99
|
| €13.97
|-
| August 2020*
| €4.99
|
| €8.98
|-
| September 2020
| €4.99
|
| €3.99
|}

* *Scaleway increased hosting charges from €2.99 to €4.99 from August 2020

=== lists.fsci.org.in ===

{| class="wikitable"
|+ mailman3 expense
|-
! Month
! Debit
! Credit
! Balance
|-
| June 2018
|
| ₹ 2855.00
| ₹ 2855.00
|-
| July 2018
| ₹ 53.62
|
| ₹ 2801.38
|-
| August 2018
| ₹ 279.82
|
| ₹ 2521.56
|-
| September 2018
| ₹ 259.20
|
| ₹ 2262.36
|-
| October 2018
| ₹ 265.07
|
| ₹ 1997.29
|-
| November 2018
| ₹ 261.14
|
| ₹ 1736.15
|-
| December 2018
| ₹ 250.25
|
| ₹ 1485.90
|-
| January 2019
| ₹ 253.69
|
| ₹ 1232.21
|-
| February 2019
| ₹ 389.31
|
| ₹ 843.21
|-
| March 2019
| ₹ 363.68
| ₹ 4670.00
| ₹ 5149.53
|-
| April 2019
| ₹ 317.53
|
| ₹ 4832.00
|-
| May 2019
| ₹ 317.53
|
| ₹ 4514.47
|-
| June 2019
| ₹ 317.53
|
| ₹ 4196.94
|-
| July 2019
| ₹ 317.53
|
| ₹ 3879.41
|-
| August 2019
| ₹ 317.53
|
| ₹ 3561.88
|-
| September 2019
| ₹ 317.53
|
| ₹ 3244.35
|-
| October 2019
| ₹ 317.53
|
| ₹ 2926.77
|-
| November 2019
| ₹ 316.8
|
| ₹ 2609.97
|-
| December 2019
| ₹ 318.8
|
| ₹ 2291.17
|-
| January 2020
| ₹ 314.8
|
| ₹ 1976.37
|-
| February 2020
| ₹ 317.5
|
| ₹ 1658.87
|-
| March 2020
| ₹ 332.7
|
| ₹ 1326.17
|-
| April 2020
| ₹ 329.9
|
| ₹ 996.27
|-
| May 2020
| ₹ 335.8
|
| ₹ 660.47
|-
| June 2020
| ₹ 263.01
|
| ₹ 397.46
|}

=== videos.fsci.org.in ===

Hosting charges: €3.49 * 12 = €41.88 per year

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €47.88
| €47.88
|-
| July 2020
| €3.49
|
| €44.39
|-
| August 2020
| €3.49
|
| €40.9
|-
| September 2020
| €3.49
|
| €37.41
|-
| October 2020
| €3.49
|
| €33.92
|-
| November 2020
| €3.49
|
| €30.43
|-
| December 2020
| €3.49
|
| €26.94
|-
| January 2021
| €3.49
|
| €23.45
|-
| February 2021
| €3.49
|
| €19.96
|-
| March 2021*
| €3.49
| €42 + €15
| €73.47
|}

* *Additional 40G storage space added which add 1.6 to the monthly cost. The total monthly cost is now €5.09. Got €15 sponsorship for the storage space and €42 for hosting charges for one year.

[[Category: Services]]

Budget

2021-03-30T18:47:59Z

Bady: Update peertube budget

Budget

2021-03-30T18:20:14Z

Bady: Add cost of poddery

Budget

2021-03-15T15:22:01Z

Bady: Update videos.fsci.in budget

Budget details of FSCI managed services. See [https://www.loomio.org/d/7znkc0RX/annual-budget-for-fsci-looking-for-volunteers this discussion for the background].

== Domains ==

== Services ==

=== poddery.com ===
=== codema.in ===

Hosting charges: €2.99 * 12 = €35.88 per year (₹239.04 * 12 = ₹2868.48, calculated as of 28/06/2018)

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €35.88
| €35.88
|-
| December 2019
| €2.19
|
| €33.69
|-
| January 2020
| €1.76
|
| €31.93
|-
| February 2020
| €2.99
|
| €28.94
|-
| March 2020
| €3.01
|
| €25.93
|-
| April 2020
| €2.99
|
| €22.94
|-
| May 2020
| €2.99
|
| €19.95
|-
| June 2020
| €2.99
|
| €16.96
|-
| July 2020
| €2.99
|
| €13.97
|-
| August 2020*
| €4.99
|
| €8.98
|-
| September 2020
| €4.99
|
| €3.99
|}

* *Scaleway increased hosting charges from €2.99 to €4.99 from August 2020

=== lists.fsci.org.in ===

{| class="wikitable"
|+ mailman3 expense
|-
! Month
! Debit
! Credit
! Balance
|-
| June 2018
|
| ₹ 2855.00
| ₹ 2855.00
|-
| July 2018
| ₹ 53.62
|
| ₹ 2801.38
|-
| August 2018
| ₹ 279.82
|
| ₹ 2521.56
|-
| September 2018
| ₹ 259.20
|
| ₹ 2262.36
|-
| October 2018
| ₹ 265.07
|
| ₹ 1997.29
|-
| November 2018
| ₹ 261.14
|
| ₹ 1736.15
|-
| December 2018
| ₹ 250.25
|
| ₹ 1485.90
|-
| January 2019
| ₹ 253.69
|
| ₹ 1232.21
|-
| February 2019
| ₹ 389.31
|
| ₹ 843.21
|-
| March 2019
| ₹ 363.68
| ₹ 4670.00
| ₹ 5149.53
|-
| April 2019
| ₹ 317.53
|
| ₹ 4832.00
|-
| May 2019
| ₹ 317.53
|
| ₹ 4514.47
|-
| June 2019
| ₹ 317.53
|
| ₹ 4196.94
|-
| July 2019
| ₹ 317.53
|
| ₹ 3879.41
|-
| August 2019
| ₹ 317.53
|
| ₹ 3561.88
|-
| September 2019
| ₹ 317.53
|
| ₹ 3244.35
|-
| October 2019
| ₹ 317.53
|
| ₹ 2926.77
|-
| November 2019
| ₹ 316.8
|
| ₹ 2609.97
|-
| December 2019
| ₹ 318.8
|
| ₹ 2291.17
|-
| January 2020
| ₹ 314.8
|
| ₹ 1976.37
|-
| February 2020
| ₹ 317.5
|
| ₹ 1658.87
|-
| March 2020
| ₹ 332.7
|
| ₹ 1326.17
|-
| April 2020
| ₹ 329.9
|
| ₹ 996.27
|-
| May 2020
| ₹ 335.8
|
| ₹ 660.47
|-
| June 2020
| ₹ 263.01
|
| ₹ 397.46
|}

=== videos.fsci.org.in ===

Hosting charges: €3.49 * 12 = €41.88 per year

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €47.88
| €47.88
|-
| July 2020
| €3.49
|
| €44.39
|-
| August 2020
| €3.49
|
| €40.9
|-
| September 2020
| €3.49
|
| €37.41
|-
| October 2020
| €3.49
|
| €33.92
|-
| November 2020
| €3.49
|
| €30.43
|-
| December 2020
| €3.49
|
| €26.94
|-
| January 2021
| €3.49
|
| €23.45
|-
| February 2021
| €3.49
|
| €19.96
|-
| March 2021
| €3.49
|
| €16.47
|}

[[Category: Services]]

Poddery - Diaspora, Matrix and XMPP

2021-01-29T10:22:26Z

Bady: Fix synapse workers doc link

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Relaod tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Learn Debian Packaging

2021-01-27T11:13:00Z

Bady: Fix links

We are guiding some learners of [https://camp.fsf.org.in Free Software Camp] to learn Debian Packaging and this page will be used to track the progress of the tasks.

== Setting up a Debian Sid environment for packaging ==

* [https://wiki.debian.org/Packaging/Pre-Requisites Different options for setting up a Debian Sid environment]

== Learn basics of Packaging ==

Understand the basic concepts using debmake/dh_make (getting source tarballs, creating source package, building the binary package, making it lintian clean)

* [https://wiki.abrahamraji.in/simple-packaging-tutorial/ Abraham Raji's simple packaging tutorial]
* [https://wiki.debian.org/SimplePackagingTutorial Simple Packaging Tutorial on Debian Wiki]

Once you understand the basic concepts, use npm2deb to automate some of those tasks like getting source tarball, a better debian directory template than the ones created by dh_make/debmake as npm2deb knows more details specific to node modules. You will still have to fix the remaining issues flagged by lintian.

* [https://wiki.debian.org/Javascript/Nodejs/Npm2Deb/Tutorial npm2deb Tutorial on Debian Wiki]

By this time you should know creating lintian clean packages for simple modules and building it in a clean environment like sbuild.

== Packaging more complicated modules ==

Once you get a clear picture of packaging a simple module, we can move to the next stage of packaging more complicated modules that will involve things like, modifying some upstream files, removing some files from source tarball, generating some files from source, getting the source tarball from a git commit etc.

* [https://wiki.debian.org/Javascript/Nodejs/Npm2Deb/AdvancedTutorial Advanced tutorial for more complicated modules]

Budget

2020-10-07T14:52:42Z

Bady: Add codema expenses

Budget details of FSCI managed services. See [https://www.loomio.org/d/7znkc0RX/annual-budget-for-fsci-looking-for-volunteers this discussion for the background].

== Domains ==

== Services ==

=== poddery.com ===
=== codema.in ===

Hosting charges: €2.99 * 12 = €35.88 per year (₹239.04 * 12 = ₹2868.48, calculated as of 28/06/2018)

{| class="wikitable"
|+ codema expense
|-
! Month
! Debit
! Credit
! Balance
|-
| January 2018
|
| €35.88
| €35.88
|-
| December 2019
| €2.19
|
| €33.69
|-
| January 2020
| €1.76
|
| €31.93
|-
| February 2020
| €2.99
|
| €28.94
|-
| March 2020
| €3.01
|
| €25.93
|-
| April 2020
| €2.99
|
| €22.94
|-
| May 2020
| €2.99
|
| €19.95
|-
| June 2020
| €2.99
|
| €16.96
|-
| July 2020
| €2.99
|
| €13.97
|-
| August 2020*
| €4.99
|
| €8.98
|-
| September 2020
| €4.99
|
| €3.99
|}

* *Scaleway increased hosting charges from €2.99 to €4.99 from August 2020

=== lists.fsci.org.in ===

{| class="wikitable"
|+ mailman3 expense
|-
! Month
! Debit
! Credit
! Balance
|-
| June 2018
|
| ₹ 2855.00
| ₹ 2855.00
|-
| July 2018
| ₹ 53.62
|
| ₹ 2801.38
|-
| August 2018
| ₹ 279.82
|
| ₹ 2521.56
|-
| September 2018
| ₹ 259.20
|
| ₹ 2262.36
|-
| October 2018
| ₹ 265.07
|
| ₹ 1997.29
|-
| November 2018
| ₹ 261.14
|
| ₹ 1736.15
|-
| December 2018
| ₹ 250.25
|
| ₹ 1485.90
|-
| January 2019
| ₹ 253.69
|
| ₹ 1232.21
|-
| February 2019
| ₹ 389.31
|
| ₹ 843.21
|-
| March 2019
| ₹ 363.68
| ₹ 4670.00
| ₹ 5149.53
|-
| April 2019
| ₹ 317.53
|
| ₹ 4832.00
|-
| May 2019
| ₹ 317.53
|
| ₹ 4514.47
|-
| June 2019
| ₹ 317.53
|
| ₹ 4196.94
|-
| July 2019
| ₹ 317.53
|
| ₹ 3879.41
|-
| August 2019
| ₹ 317.53
|
| ₹ 3561.88
|-
| September 2019
| ₹ 317.53
|
| ₹ 3244.35
|-
| October 2019
| ₹ 317.53
|
| ₹ 2926.77
|-
| November 2019
| ₹ 316.8
|
| ₹ 2609.97
|-
| December 2019
| ₹ 318.8
|
| ₹ 2291.17
|-
| January 2020
| ₹ 314.8
|
| ₹ 1976.37
|-
| February 2020
| ₹ 317.5
|
| ₹ 1658.87
|-
| March 2020
| ₹ 332.7
|
| ₹ 1326.17
|-
| April 2020
| ₹ 329.9
|
| ₹ 996.27
|-
| May 2020
| ₹ 335.8
|
| ₹ 660.47
|-
| June 2020
| ₹ 263.01
|
| ₹ 397.46
|}

=== videos.fsci.org.in ===

Hosting charges: ₹318.99 * 12 = ₹3827.88 per year
[[Category: Services]]

Poddery - Diaspora, Matrix and XMPP

2019-05-10T13:57:27Z

Bady: /* Chat/Matrix */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public --expand -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-05-10T13:54:26Z

Bady: /* Chat/Matrix */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot web client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public --expand -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-05-10T13:50:58Z

Bady: /* Riot-web Updation */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public --expand -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-05-10T13:49:14Z

Bady: /* Workers */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like v1.0.0):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public --expand -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-05-10T11:05:03Z

Bady: /* Riot-web Updation */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =
== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Synapse Updation ===
* First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like v1.0.0):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public --expand -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-05-01T18:54:02Z

Bady: /* Disk Partitioning */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T18:49:47Z

Bady: /* Configuration and Maintenance */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T09:50:58Z

Bady: /* Workers */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T09:48:54Z

Bady: /* Backend Services */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T09:45:49Z

Bady: /* Diaspora */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T09:39:19Z

Bady: /* Chat/XMPP */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T09:33:21Z

Bady: Added Matrix and XMPP configuration

Poddery - Diaspora, Matrix and XMPP

2019-05-01T07:26:33Z

Bady: /* Contact */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T07:18:28Z

Bady: /* Contact */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T07:18:00Z

Bady: /* Contact */

Poddery - Diaspora, Matrix and XMPP

2019-05-01T07:14:54Z

Bady: /* Contact */

Poddery - Diaspora, Matrix and XMPP

2019-04-30T22:08:11Z

Bady: /* Configuration and Maintenance */

Poddery - Diaspora, Matrix and XMPP

2019-04-30T22:02:29Z

Bady: /* Disk Partitioning */

Poddery - Diaspora, Matrix and XMPP

2019-04-30T22:01:08Z

Bady: Updated according to the current setup in Hetzner server

Poddery/Archive

2019-04-30T19:15:06Z

Bady: Archive of Poddery setup before migrating to Hetzner

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* Create partitions root, boot and swap.
* Setup RAID 1:
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n log /dev/data -L <size_of_disk> # currently 50G
sudo lvcreate -n db /dev/data -L <size_of_disk> #currently 500G
sudo lvcreate -n diaspora /dev/data -l 100%FREE
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

==== Set Nginx Conf for BOSH URLS ====

* Add this configuration in nginx configuration file to enable the BOSH url to make JSXC Working.

'''Nginx'''

upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

Plz look [https://wiki.diasporafoundation.org/Integration/Chat#Nginx here] for more details. And apache settings [https://github.com/jsxc/jsxc/wiki/Prepare-apache here] :)

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database

==== Workers ====

For scalability, we are running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect `synapse.app.appservice` is running on poddery.com

A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers. (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something.)

The worker config can be found at <code>/etc/matrix-synapse/workers</code>

Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>

These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:

enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

==== Upgrade ====

First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra need to be done. Then, just run <code>/root/upgrade-synapse</code>

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

=== Riot-web Updation ===
https://chat.poddery.com/#/welcome
Backup current riot-web folder from riot to riot-backup
wget https://github.com/vector-im/riot-web/releases/download/v1.0.1/riot-v1.0.1.tar.gz
tar -xvf riot-v1.01.tar.gz
cp -r riot-v1.0.1/* /var/www/riot/
rm -rf ./riot-v1.0.1*
Transfer the old config.json,home.html,home-status.html from riot-backup to /var/www/riot/
systemctl restart nginx

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== exim ===

For sending emails.
sudo dpkg-reconfigure exim4-config

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

NOTE: This service has been disabled since the community decided that XMPP service no longer needs to be served via port 443, see this [https://www.loomio.org/d/xSiI8FGT/xmpp-service-on-port-443-and-sslh-complexity loomio post] for more details.

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues issue tracker] - we use this to track progress of tasks

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-04-01T12:26:19Z

Bady: /* System health check */

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* Create partitions root, boot and swap.
* Setup RAID 1:
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n log /dev/data -L <size_of_disk> # currently 50G
sudo lvcreate -n db /dev/data -L <size_of_disk> #currently 500G
sudo lvcreate -n diaspora /dev/data -l 100%FREE
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

==== Set Nginx Conf for BOSH URLS ====

* Add this configuration in nginx configuration file to enable the BOSH url to make JSXC Working.

'''Nginx'''

upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

Plz look [https://wiki.diasporafoundation.org/Integration/Chat#Nginx here] for more details. And apache settings [https://github.com/jsxc/jsxc/wiki/Prepare-apache here] :)

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database

==== Workers ====

For scalability, we are running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect `synapse.app.appservice` is running on poddery.com

A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers. (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something.)

The worker config can be found at <code>/etc/matrix-synapse/workers</code>

Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>

These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:

enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

==== Upgrade ====

First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra need to be done. Then, just run <code>/root/upgrade_synapse</code>

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

=== Riot-web Updation ===
https://chat.poddery.com/#/welcome
Backup current riot-web folder from riot to riot-backup
wget https://github.com/vector-im/riot-web/releases/download/v1.0.1/riot-v1.0.1.tar.gz
tar -xvf riot-v1.01.tar.gz
cp -r riot-v1.0.1/* /var/www/riot/
rm -rf ./riot-v1.0.1*
Transfer the old config.json,home.html,home-status.html from riot-backup to /var/www/riot/
systemctl restart nginx

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== exim ===

For sending emails.
sudo dpkg-reconfigure exim4-config

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

NOTE: This service has been disabled since the community decided that XMPP service no longer needs to be served via port 443, see this [https://www.loomio.org/d/xSiI8FGT/xmpp-service-on-port-443-and-sslh-complexity loomio post] for more details.

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues issue tracker] - we use this to track progress of tasks

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2019-04-01T12:15:33Z

Bady: Add a new raid array to software RAID

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* Create partitions root, boot and swap.
* Setup RAID 1:
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n diaspora /dev/data -L <size_of_disk>
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

==== Set Nginx Conf for BOSH URLS ====

* Add this configuration in nginx configuration file to enable the BOSH url to make JSXC Working.

'''Nginx'''

upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

Plz look [https://wiki.diasporafoundation.org/Integration/Chat#Nginx here] for more details. And apache settings [https://github.com/jsxc/jsxc/wiki/Prepare-apache here] :)

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database

==== Workers ====

For scalability, we are running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect `synapse.app.appservice` is running on poddery.com

A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers. (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something.)

The worker config can be found at <code>/etc/matrix-synapse/workers</code>

Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>

These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:

enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

==== Upgrade ====

First check [https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst synapse/UPGRADE.rst] to see if anything extra need to be done. Then, just run <code>/root/upgrade_synapse</code>

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

=== Riot-web Updation ===
https://chat.poddery.com/#/welcome
Backup current riot-web folder from riot to riot-backup
wget https://github.com/vector-im/riot-web/releases/download/v1.0.1/riot-v1.0.1.tar.gz
tar -xvf riot-v1.01.tar.gz
cp -r riot-v1.0.1/* /var/www/riot/
rm -rf ./riot-v1.0.1*
Transfer the old config.json,home.html,home-status.html from riot-backup to /var/www/riot/
systemctl restart nginx

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== exim ===

For sending emails.
sudo dpkg-reconfigure exim4-config

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

NOTE: This service has been disabled since the community decided that XMPP service no longer needs to be served via port 443, see this [https://www.loomio.org/d/xSiI8FGT/xmpp-service-on-port-443-and-sslh-complexity loomio post] for more details.

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues issue tracker] - we use this to track progress of tasks

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]

Loomio/Archive 1

2019-02-02T17:04:44Z

Bady:

[https://freesoftwareindia.org/ Loomio] instance at https://codema.fsci.org.in. Loomio is a '''co'''llaborative '''de'''cision-'''ma'''king platform (hence we're using the name '''codema''') where users can initiate discussions and put up proposals.

== Hosting ==
We were now on a [https://www.scaleway.com/virtual-cloud-servers START1-S instance] virtual cloud server with the following specs:

* 2 x86-64 bit Cores
* 2 GB Memory
* 50GB SSD Disk
* 200 Mbit/s Bandwidth
* 1 Reserved IP (v4)
* €3.99/Month

== Coordination ==

* Hangout with us in our Matrix room [https://matrix.to/#/#codema:poddery.com #codema:poddery.com]

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

== History ==
We were on a [https://www.scaleway.com/virtual-cloud-servers VC1S instance] virtual cloud server with the following specs until 31/01/2019.

* 2 x86-64 bit Cores
* 2GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 200Mbit/s Unmetered bandwidth
* €2.99/Month

On 01/02/2019 our server was taken down by [https://scaleway.com Scaleway] quoting payment issues. Payment was failing even after updating the credit card details and following that our server got deleted without proper notifications from Scaleway's side. Screenshot of the email from Scaleway attached below. Fortunately we were provided with a snapshot of the server from which we were able to recover codema to a new server.

[[File:]]

=== Codema recovery process ===

''Here's a brief description of how codema was recovered after the server take down on 01/01/2019:''

Under 'Snapshots' tab in the Scaleway dashboard we were provided with the snapshot (backup) of our codema server. A system image was created from this snapshot and it was used to create a new server with similar specifications. We lost our public IP along with the old server, so a new IP was assigned to the server and then updated the DNS A record of codema.fsci.org.in to point to this new IP. Once the server was up loomio was restarted using the following commands from the loomio installation directory:

docker-compose down
docker-compose up -d

The logs were checked for errors using the following command:

docker-compose logs -f

Loomio wasn't getting started saying the port 25 was already in use. So the application using that port (which was exim4 in this case) was killed and loomio was restarted again.

[[Category: Services]]

Poddery - Diaspora, Matrix and XMPP

2018-10-14T10:17:44Z

Bady: /* sslh */

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n diaspora /dev/data -L <size_of_disk>
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

==== Set Nginx Conf for BOSH URLS ====

* Add this configuration in nginx configuration file to enable the BOSH url to make JSXC Working.

'''Nginx'''

upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

Plz look [https://wiki.diasporafoundation.org/Integration/Chat#Nginx here] for more details. And apache settings [https://github.com/jsxc/jsxc/wiki/Prepare-apache here] :)

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database

==== Workers ====

For scalability, we are running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect `synapse.app.appservice` is running on poddery.com

A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers. (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something.)

The worker config can be found at <code>/etc/matrix-synapse/workers</code>

Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>

These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:

enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== exim ===

For sending emails.
sudo dpkg-reconfigure exim4-config

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

NOTE: This service has been disabled since the community decided that XMPP service no longer needs to be served via port 443, see this [https://www.loomio.org/d/xSiI8FGT/xmpp-service-on-port-443-and-sslh-complexity loomio post] for more details.

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues issue tracker] - we use this to track progress of tasks

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2018-10-14T10:14:27Z

Bady: /* sslh */

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n diaspora /dev/data -L <size_of_disk>
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

==== Set Nginx Conf for BOSH URLS ====

* Add this configuration in nginx configuration file to enable the BOSH url to make JSXC Working.

'''Nginx'''

upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

Plz look [https://wiki.diasporafoundation.org/Integration/Chat#Nginx here] for more details. And apache settings [https://github.com/jsxc/jsxc/wiki/Prepare-apache here] :)

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database

==== Workers ====

For scalability, we are running [https://github.com/matrix-org/synapse/blob/master/docs/workers.rst workers]. Currently all workers specified in that page, expect `synapse.app.appservice` is running on poddery.com

A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers. (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something.)

The worker config can be found at <code>/etc/matrix-synapse/workers</code>

Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>

These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:

enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

These services must be enabled, and added to <code>Requires</code> and <code>Before</code> sections of the original <code>matrix-synapse.service</code>:
matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== exim ===

For sending emails.
sudo dpkg-reconfigure exim4-config

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

NOTE: This service has been disabled since the community decided that XMPP service no longer needs to be served via port 443, see this loomio post for more details.

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]
* [https://git.fosscommunity.in/community/poddery.com/issues issue tracker] - we use this to track progress of tasks

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]

Budget

2018-06-29T16:52:13Z

Bady: /* Services */

Budget details of FSCI managed services.

== Domains ==

== Services ==

=== poddery.com ===
=== codema.fsci.org.in ===

Hosting charges: ₹239.04 * 12 = ₹2868.48 per year
=== lists.fsci.org.in ===
=== videos.fsci.org.in ===

Hosting charges: ₹318.99 * 12 = ₹3827.88 per year
[[Category: Services]]

Loomio/Archive 1

2018-06-26T09:39:07Z

Bady: /* Hosting */

Loomio/Archive 1

2018-06-25T16:58:41Z

Bady:

Loomio/Archive 1

2018-06-25T16:55:43Z

Bady:

Poddery - Diaspora, Matrix and XMPP

2018-03-25T10:48:50Z

Bady: /* exim4 */

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n diaspora /dev/data -L <size_of_disk>
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database (Note: PyPi package is outdated, hence install from git)

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== exim ===

For sending emails.
sudo dpkg-reconfigure exim4-config

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2018-03-25T10:05:45Z

Bady: /* Chat/Matrix */

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n diaspora /dev/data -L <size_of_disk>
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database (Note: PyPi package is outdated, hence install from git)

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== exim4 ===

For sending emails.
sudo dpkg-reconfigure exim4-config

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2018-03-25T09:19:39Z

Bady: /* Backend Services */

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n diaspora /dev/data -L <size_of_disk>
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database.

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== exim4 ===

For sending emails.
sudo dpkg-reconfigure exim4-config

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com

# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

Make sure the certificates used by prosody are symbolic links to letsencrypt default location.

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2018-02-16T17:44:28Z

Bady: /* Handling critical data */

We run Diaspora, XMPP and Matrix services at [https://poddery.com poddery.com]. Diaspora username and password can be used to access XMPP and Matrix services. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
We are on a [https://www.scaleway.com/baremetal-cloud-servers/ C2S instance of scaleway.com bare metal cloud server].

* 4 Dedicated x86 64bit Cores
* 8GB Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* 300Mbit/s Unmetered bandwidth
* 2.5Gbit/s Internal bandwidth
* €11.99 Per Month

Due to performance issues we are migrating to a new server ([https://www.scaleway.com/baremetal-cloud-servers/ C2M instance of scaleway.com]) with the following specs:

* '''8''' Dedicated x86 64bit Cores
* '''16GB''' Memory
* 50GB SSD Disk
* 1 Flexible Public IPv4
* '''500Mbit/s''' Unmeterd bandwidth
* '''5Gbit/s''' Internal bandwidth
* '''€17.99''' Per Month
* Extra '''150GB''' SSD
* Total '''€20.99''' Per Month

== Operating System ==

We run Debian 9 Stretch image provided by Scaleway, with latest security updates applied.

=== Hardening checklist ===
* SSH password login disabled (allow only key based logins)
* root SSH login disabled (use a normal user with sudo)
'''/etc/ssh/sshd_config:'''
...
PermitRootLogin no
...
PasswordAuthentication no
...
* Firewall enabled with only the ports we need opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial])
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
Currently ufw is disabled as it is crashing the server.

* fail2ban configured against brute force attacks
'''/etc/ssh/sshd_config:'''
...
LogLevel VERBOSE
...

sudo systemctl restart ssh
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Check '''/var/log/fail2ban.log''' for logs

Unban an IP:
sudo fail2ban-client set sshd unbanip <banned_ip>

Here sshd is the defaut jail name, change it if you are using a different jail.

=== System health check ===

* There should be a data disk attached (added from cloud.scaleway.com)
* The attached disk (/dev/nbdX) should be an lvm physical volume. We cannot use it directly for encryption, so we use lvm.
# Make sure '''lvm2''' and '''udev''' packages are installed
sudo apt-get install lvm2 udev

# Replace X with valid number according to '''lsblk'''
sudo pvcreate /dev/nbdX
* /dev/data is an lvm volume group created from /dev/nbdX
sudo vgcreate data /dev/nbdX
* /dev/data/diaspora is an lvm logical volume
sudo lvcreate -n diaspora /dev/data -L <size_of_disk>
* /dev/mapper/diaspora is an encrypted device
# Make sure '''cryptsetup''' package is installed
sudo apt-get install cryptsetup

# Give disk encryption password as specified in the [[#Server_Access|access repo]]
sudo cryptsetup luksFormat /dev/data/diaspora
sudo cryptsetup luksOpen /dev/data/diaspora diaspora
* /dev/mapper/diaspora is an ext4 file system
sudo mkfs.ext4 /dev/mapper/diaspora
* /var/lib/diaspora should be mounted. All [[#Handling_critical_data|critical data]] should be on /var/lib/diaspora.
sudo mkdir /var/lib/diaspora
sudo mount /dev/mapper/diaspora /var/lib/diaspora

== User Visible Services ==
=== Diaspora ===

* We use diaspora-installer-mysql package from https://people.debian.org/~praveen/diaspora/README
* See [https://salsa.debian.org/ruby-team/diaspora-installer/blob/debian/0.6.6.0+debian1/README /usr/share/doc/diaspora-common/README] for package specific configuration.
* [https://poddery.com/statistics live statistics]

=== Chat/XMPP ===

* We use Prosody and steps for setting up Prosody is given at -> https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP
sudo mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

sudo chown -R root:ssl-cert /etc/letsencrypt
sudo chmod g+r -R /etc/letsencrypt
sudo chmod g+x /etc/letsencrypt/{archive,live}

sudo systemctl restart prosody
* We have enabled all XEPs conversations expect. We use sslh to multiplex Diaspora and Prosody on port 443. See [https://wiki.debian.org/InstallingProsody#XMPP_over_HTTPS XMPP over HTTPS] section of the Installing Prosody article in Debian Wiki for sample sslh configuration.

=== Chat/Matrix ===

* We use Synapse server for setting up the Matrix server.
* We will be using https://github.com/matrix-org/synapse/#synapse-installation for setting up this instance
* We use nginx reverse proxy to send requests that has ''/_matrix/*'' in url to synapse on 8008. See /etc/nginx/sites-enabled/diaspora
* We use https://git.fosscommunity.in/necessary129/synapse-diaspora-auth to authenticate synapse with Diaspora database.

=== Homepage ===

Homepage and other static pages are maintained in our Gitlab instance. You can change it directly in the master branch or send pull requests. You can edit it via web as well.

* poddery.com -> https://git.fosscommunity.in/community/poddery.com
# Make sure '''git''' and '''acl''' packages are installed
sudo apt-get install git acl

# Grant rwx permissions for the ssh user to /usr/share/diaspora/public
sudo setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
cd /usr/share/diaspora/public/save
git submodule init
git submodule update
save.poddery.com repo is maintained as a sub module in poddery.com repo. See this tutorial -> https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ for working with git submodules.

== Backend Services ==
=== nginx ===

Front-end for Diaspora and Matrix.

=== PostgreSQL ===

Backend for Matrix.

=== MySQL ===

Backend for Diaspora.

'''TODO''': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== sslh ===

Port multiplexer to allow XMPP and Diaspora to share 443 port. This allows us to fool stupid firewalls which blocks all ports except 80 and 443 (hence XMPP).

=== SSL/TLS certificates ===

# letsencrypt certonly --webroot -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com
# cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
# cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key
# chown -R root:ssl-cert /etc/letsencrypt
# chmod g+r -R /etc/letsencrypt
# chmod g+x /etc/letsencrypt/*

# ls -l /etc/prosody/certs/
total 0
lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem
lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem

# crontab -e
30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log
32 2 * * 1 /etc/init.d/nginx reload
34 2 * * 1 /etc/init.d/prosody reload

=== Handling critical data ===
sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /var/lib/diaspora
sudo ln /var/lib/diaspora/mysql /var/lib/mysql
sudo mkdir /var/lib/diaspora/uploads
sudo chown -R diaspora: /var/lib/diaspora/uploads
sudo ln -s /var/lib/diaspora/uploads /usr/share/diaspora/public/uploads

=== Services health check ===

Sample output - Look for "Active: active (running)"

systemctl status nginx # Our web server front-end for Diaspora, XMPP and Matrix
nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
Active: active (running) since Fri 2018-01-05 07:17:02 UTC; 4 weeks 1 days ago
Process: 5063 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 13140 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 5071 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 5067 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 5072 (nginx)
CGroup: /system.slice/nginx.service
├─ 5072 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─13149 nginx: worker process
├─13150 nginx: worker process
├─13151 nginx: worker process
└─13153 nginx: worker process

systemctl status diaspora # Diaspora service
diaspora.service - LSB: Diaspora application server
Loaded: loaded (/etc/init.d/diaspora)
Active: active (running) since Fri 2018-01-05 07:21:29 UTC; 4 weeks 1 days ago
Process: 5146 ExecStop=/etc/init.d/diaspora stop (code=exited, status=0/SUCCESS)
Process: 5167 ExecStart=/etc/init.d/diaspora start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/diaspora.service
├─ 850 unicorn worker[0] -c config/unicorn.rb -D
├─ 5174 sudo -u diaspora -E -H ./script/server
├─ 5175 eye monitoring v0.9.1 [diaspora] (in /usr/share/diaspora)
├─ 5211 sidekiq 4.2.9 diaspora [0 of 25 busy]
├─ 5222 unicorn master -c config/unicorn.rb -D
└─31717 unicorn worker[1] -c config/unicorn.rb -D

systemctl status matrix-synapse.service # Synapse Matrix Server
matrix-synapse.service - Synapse Matrix homeserver
Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled)
Active: active (running) since Sat 2018-01-13 05:38:55 UTC; 3 weeks 1 days ago
Process: 15800 ExecStartPre=/var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
Main PID: 15808 (python2.7)
CGroup: /system.slice/matrix-synapse.service
└─15808 /var/lib/matrix-synapse/synapse/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

systemctl status prosody # Prosody XMPP Server
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Fri 2018-01-05 07:35:41 UTC; 4 weeks 1 days ago
Process: 6218 ExecStop=/etc/init.d/prosody stop (code=exited, status=0/SUCCESS)
Process: 6483 ExecReload=/etc/init.d/prosody reload (code=exited, status=0/SUCCESS)
Process: 6223 ExecStart=/etc/init.d/prosody start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/prosody.service
└─6231 /usr/bin/lua5.1 /usr/bin/prosody

systemctl status sslh # SSL/SSH multiplexer which allow us to provide multiple services via 443 port (to bypass stupid firewalls)
sslh.service - SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled)
Active: active (running) since Fri 2018-01-05 07:29:27 UTC; 4 weeks 1 days ago
Docs: man:sslh(8)
Main PID: 5444 (sslh)
CGroup: /system.slice/sslh.service
├─ 713 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 830 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1672 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 1673 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3514 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3875 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3876 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 3896 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 4965 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5395 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5444 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5445 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 5963 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6617 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6774 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 6957 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7063 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─ 7083 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
├─25613 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg
└─27481 /usr/sbin/sslh --foreground -F /etc/sslh/sslh.cfg

= Coordination =

*[https://www.loomio.org/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group loomio group] - we use this for decision making.
* Hangout with us in our Matrix room [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com]

=== Contact ===

Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)

The following people have their GPG keys in the password file.

Praveen Arimbrathodiyil (piratepin) (ID: 0xCE1F9C674512), Balasankar C (ID: 0x96EDAB9B2E6B7171), Manu Krishnan T V (ID: 0x5D0064186AF037D9), Fayad Fami (fayad) (ID: 0x51C954405D432381), Abhijith PA (ID: 0x863D4DF2ED9C28EF), Syam G Krishnan (sgk) (ID: 0x6EF48CCD865A1FFC), Sagar Ippalpalli (ID: 0xFD49D0BC6FEAECDA), Pirate Bady (piratesin) (ID: 0x92FDAB42A95FF20C), Kannan (ID: 0x0B1955F40C691CCE)

We recommend you setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GPG Plugin] for transparent editing. If you are new to GPG, then follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===

Maintained in a private git repo at -> https://git.fosscommunity.in/community/access

= Setting up Backup =

Backup was setup on a Scaleway C1 VPS (4 core ARM processor with 2GB RAM). '''TODO: C1 server was crashing frequently and we need to setup backup again on VPS provided by Manu'''.

Hostname (IP): backup.poddery.com (No public ip, access via scaleway.com web console). If you restart this machine, you may want to add poddery.com private ip in /etc/hosts

# apt-get install lvm2 cryptsetup

Directly creating luks volume on /dev/nbd1 is not working, so we use a logical volume

# pvcreate /dev/nbd1
# vgcreate data /dev/nbd1
# lvcreate -n diaspora -L 46.5G /dev/data

# cryptsetup luksFormat /dev/data/diaspora
# cryptsetup luksOpen /dev/data/diaspora diaspora

and update /etc/crypttab
# <target name> <source device> <key file> <options>
diaspora /dev/data/diaspora none luks

# mkfs.ext4 /dev/mapper/diaspora
# mkdir /var/lib/diaspora
and update /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/diaspora /var/lib/diaspora ext4 defaults 0 2

# mount -a
# apt-get install mysql-server

Move MySQL data directory to encrypted volume
# /etc/init.d/mysql stop
# mv /var/lib/mysql /var/lib/diaspora/
# ln -s /var/lib/diaspora/mysql /var/lib/mysql

Follow steps in https://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html for replication

Follow steps in https://www.howtoforge.com/how-to-set-up-mysql-database-replication-with-ssl-encryption-on-centos-5.4 for ssl (but ssl support is disabled in debian)

Follow steps in http://www.networkcomputing.com/storage/how-set-ssh-encrypted-mysql-replication/1111882674 to use ssh port forwarding to have encrypted replication

# adduser sshtunnel --disabled-login
# su sshtunnel

Generate SSH key pair and copy public key to target system
$ ssh-keygen -t rsa
$ ssh -f sshtunnel@poddery.com -L 7777:127.0.0.1:3306 -N

Test the connectivity
# mysql -u poddery_backup -p -P 7777 -h 127.0.0.1

Uploads are rsynced every hour

# crontab -e
# m h dom mon dow command
0 * * * * pgrep rsync || rsync -av --delete root@poddery.com:/var/lib/diaspora/uploads/ /var/lib/diaspora/uploads/ >/var/lib/diaspora/rsync-uploads.log

'''Note:''' Since we are not using a public ip (saves us money), backup.poddery.com connects to poddery.com via private ip. So if poddery.com is rebooted, the new ip address should be updated in /etc/hosts file of backup.poddery.com. To connect, use the web console from scaleway.com

= Add more disk space =

# Power off the machine with "ARCHIVE" option. It may take upto an hour for shutdown to complete on backup.poddery.com and poddery.com
# Add more disk from scaleway.com control panel . Volumes -> CREATE VOLUME
# Attach the newly created volume to server from Server page
# Power on the server
# Create physical volume (pvcreate /dev/nbdN)
# Expand volume group (vgextend data /dev/nbdN)
# Expand logical volume (lvresize --size=186G data/diaspora)
# Expand encrypted partition (cryptsetup resize diaspora)
# Resize file system (resize2fs /dev/mapper/diaspora)

= Maintenance history =
This section holds maintenance/issue history for future tracking.

'''When updating diaspora-installer-mysql packages, remember to recreate /usr/share/diaspora/public/uploads symlink to /var/lib/diaspora/uploads'''.

1. Prosody error - Failed to load private key

certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Previous error (see logs), or other system error. (for poddery.com)
tls error Unable to initialize TLS: error loading private key (system lib)
certmanager error SSL/TLS: Failed to load '/etc/letsencrypt/live/poddery.com/privkey.pem': Check that the permissions allow Prosody to read this file.

This error is usually when ssl certificate in freshly installed or renewed. Prosody user is unable to access the key file due to lack of privileges.

Note that Poddery uses Letsencrypt for ssl.

Fix:

* Make sure that prosody user is in 'certs' group (this group may also be called ssl-certs as setup by Letencrypt)
* /etc/letsencrypt/ is the ssl directory.
* Prosody user should have permissions to all folders importantly archive and live folders in /etc/letsencrypt. Permissions to each folder must be 750.
* Troubleshoot by checking if you can switch to each folder in /etc/letsencrypt as prosody user and cat the files.

'''If replication fails, you can restart it following the instructions here'''

https://dba.stackexchange.com/questions/69394/mysql-replication-error-1594

[[Category:Services]]