FSCI Wiki - User contributions [en]

NUUP

2026-01-12T16:23:27Z

Ravi: /* Directly Opening a specific screen */ Reword

[[File:*99- USSD.png|alt=USSD Sample|thumb|*99# USSD]]

The National Unified USSD Platform (NUUP) lets you use UPI by dialling *99#, without having to install a proprietary app.

The service is run by the National Payments Corporation of India (NPCI), the government-backed private company that also operates the UPI ecosystem itself.

== *99# and BHIM ==
Besides NUUP, NPCI also operates BHIM which is a proprietary app for Android smartphones. The two are linked in the sense that dialling *99# is essentially a different way of accessing your BHIM account. Practically, this means that if you run into issues setting up UPI payments with *99#, you can often work around this by borrowing a proprietary Android phone from someone and installing BHIM on that to link your bank and set up UPI payments. Once done, you can uninstall the app and make payments directly via *99#.

== Provider support ==
While NUUP is theoretically supposed to work on all USSD supporting carriers in India (all the provider apart from Jio), the service is not very well maintained and keeps going down, or, in some cases, doesn't work at all.

{| class="wikitable"
|+ Provider support for *99#
|-
! Provider !! Officially supported !! Actually working
|-
| Airtel || {{Tick}} || {{Tick}}
|-
| BSNL || {{Tick}} || {{Cross}}
|-
| Jio || {{Cross}} || {{Cross}}
|-
| MTNL || ||
|-
| Vi || {{Tick}} || {{Tick}} but can pay only to mobile numbers and not full upi ids. collect requests works.
|}

[[File:Nuup-vi.png|alt=USSD failure with vi|thumb|Vi failing with UPI ID]]

== Bank support ==
The NPCI [https://www.npci.org.in/what-we-do/99/live-members lists 103 banks] supporting *99# payments. However, each of them has their own slightly different interface to the service, and not all of them work seamlessly.

This section provides crowdsourced information on which banks have been known to work or not, as experienced by the FSCI community.

{| class="wikitable sortable"
|+ Bank feature support on *99#
|-
! Bank !! Available on service !! Registration (if UPI PIN not set) !! Registration (if UPI PIN already set) !! Accepting payment request !! Receiving money!! Payment to beneficiary !! Payment to UPI ID !! Payment to Mobile Number !! Payment to Account/IFSC !! Setting UPI ID
|-
| City Union Bank || {{Tick}} || {{Cross}} Sends an SMS regarding UPI activation as well as an OTP, but USSD only says to "keep your debit card ready" and doesn't accept further input || {{Tick}} || {{Tick}} || {{Tick}} || || {{Cross}} truncates at `@` and claims "invalid UPI ID" || || || {{Tick}}
|-
| Federal Bank || {{Tick}} || || {{Tick}} || || || || yes|| yes|| ||
|-
| Kotak Bank || {{Tick}} || || {{Tick}} || || || || yes|| yes|| ||
|-
| ICICI || {{Tick}} || {{Tick}} || {{Tick}} || || || || {{Tick}} on Airtel, but {{Cross}} on vi, which truncates at `@` and claims "invalid UPI ID" || {{Tick}} || ||
|-
| India Post Payments Bank || {{Cross}} Supposedly listed, but not actually present || {{Cross}} || {{Cross}} || {{Cross}} || || {{Cross}} || {{Cross}} || {{Cross}} || {{Cross}} || {{Cross}}
|-
| Indian Bank || {{Tick}} || {{Cross}} Allows debit card based registration but goes into loop due to bug with 6 digit PIN vs. 4 digit PIN || {{Tick}} || {{Tick}} || {{Tick}} || {{Tick}} || {{Cross}} truncates at `@` and claims "invalid UPI ID" || {{Tick}} || || {{Tick}}
|-
|Central Bank of India
|{{Tick}}
|
|
|
|
|{{Tick}}
|{{Tick}}
|{{Tick}}
|
|
|}

== City Support ==
It is observed to be working only in selected cities.

Confirmed working: Bengaluru, Kochi, Mumbai, Gurugram

Confirmed not working: Ahmedabad, Kolkata, Palakkad, Pune, Surat, Coimbatore

== Limitations ==
Dynamic QR codes generated by payment machines don't work. Only fixed QR codes / UPI ids work.

== UPI URL format ==
Currently we can only copy the whole URL in most QR scanning apps, so we have to manually extract the UPI id when making payments via *99# ussd interface or in android without google play service, upi apps can't scan directly, so you still need to extract the upi id manually).

UPI url looks like upi://pay?pa=Motilal2005@hdfcbank&pn=MOFSL&mc=6021&tr=uft_2288daf8bd45a5&tn=Pay%2520to%2520MOFSL&mode=03&am=100&cu=INR

Here UPI id is the value of pa, ie, Motilal2005@hdfcbank

See https://source.puri.sm/Librem5/millipixels/-/issues/98 for adding support to extract UPI id in millipixels.

== Collect Requests ==
Businesses can still initiate collect requests - for example payment gateways (tested with RazorPay page). You enter your UPI id on the webpage and diasl *99#, it will directly show the accept request screen, you can enter the pin to approve the request.
[[File:Upi-Collect-Request.png|alt=UPI Collect Request screenshot|thumb|UPI Collect Request]]

'''Note:''' Starting 1 October 2025, UPI collect request will [https://www.angelone.in/news/personal-finance/upi-collect-requests-to-end-from-october-1-what-phonepe-google-pay-paytm-users-must-know no longer be available for individuals] although it will still be available for businesses as well as online payments. The ostensible reason is to prevent "fraud" from people misusing the collect request functionality. Specifically, this means it will not be possible to ask people to request money in case adding via UPI ID is not working. Some UPI apps like Paytm and PhonePe have already stopped offering this feature.

Collect requests works with RazorPay page but fails with Airtel Recharge.

== Directly Opening a specific screen ==
Choices can be sent directly as well. *99*1# opens the screen to send money directly, whereas *99*1*3# takes you to the screen to enter the UPI ID. So with this the camera app itself could add a pay option if upi:// URL is detected and dial *99*1*3# and fill the UPI id, so we can emulate the scan and pay - then people will only have to enter pin manually!

== See also ==

# [[Making payments using Free Software]]

Poddery - Diaspora, Matrix and XMPP

2025-10-11T19:06:02Z

Ravi: /* TLS */ Update certificate renewal steps

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social network, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Element client (accessed by a web browser), which can be used to connect to any Matrix server without installing the Element app.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service
matrix-synapse@federation_reader.service
matrix-synapse@event_creator.service
matrix-synapse@federation_sender.service
matrix-synapse@pusher.service
matrix-synapse@user_dir.service
matrix-synapse@media_repository.service
matrix-synapse@frontend_proxy.service
matrix-synapse@client_reader.service
matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

===SSL certificate renewal===
On the 12th of October 2025, all the certificates were removed and were recreated. [https://codema.in/d/XUfAOrPW/poddery-server-certificates-recreated This thread] documents all those steps.

When renewing certificates on the poddery server, make sure to follow the following steps.

# Stop nginx by running
sudo systemctl stop nginx

# Renew certificates for all the domains
sudo certbot renew

Follow the prompts by certbot to renew certificates for all the domains.

# Start nginx after the renewal is successful
sudo systemctl start nginx

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:

* Dump of postgresql database using `pg_dump`
* `/etc/matrix-synapse` - contains config files
* `/var/lib/static/synapse/media` -- contains uploaded media files

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:<syntaxhighlight lang="bash">
eval "$(ssh-agent -s)"
</syntaxhighlight>followed by<syntaxhighlight>
ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"
</syntaxhighlight>on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:<syntaxhighlight>
ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > synapse-2025-01-07.dump
</syntaxhighlight>

====Prosody====
For backing up prosody, the following were copied:

* Dump of the database using `mysqldump`
* `/var/lib/prosody` for media files
* `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server
<syntaxhighlight lang="bash">
ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup.sql.gz
</syntaxhighlight>

Backup of `/var/lb/prosody` was taken using following steps:

* Create a tar file of prosody directory
<syntaxhighlight>
cd /var/lib && sudo tar -czvf ~user/var.lib.prosody-2025-01-07.tar.gz prosody
</syntaxhighlight>

* Make user as owner of compressed file:

<syntaxhighlight>
cd && chown user: var.lib.prosody-2025-01-07.tar.gz
</syntaxhighlight>

* Use `scp` to transfer tar file to the backup-server
<syntaxhighlight>
scp -P <port-for-ssh-on-backup-server> ./var.lib.prosody-2025-01-07.tar.gz backup-user@backup-server:directory-to-backup
</syntaxhighlight>

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2025-10-07T20:44:50Z

Ravi: Fix 1 spelling

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social network, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Element client (accessed by a web browser), which can be used to connect to any Matrix server without installing the Element app.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service
matrix-synapse@federation_reader.service
matrix-synapse@event_creator.service
matrix-synapse@federation_sender.service
matrix-synapse@pusher.service
matrix-synapse@user_dir.service
matrix-synapse@media_repository.service
matrix-synapse@frontend_proxy.service
matrix-synapse@client_reader.service
matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:

* Dump of postgresql database using `pg_dump`
* `/etc/matrix-synapse` - contains config files
* `/var/lib/static/synapse/media` -- contains uploaded media files

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:<syntaxhighlight lang="bash">
eval "$(ssh-agent -s)"
</syntaxhighlight>followed by<syntaxhighlight>
ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"
</syntaxhighlight>on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:<syntaxhighlight>
ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > synapse-2025-01-07.dump
</syntaxhighlight>

====Prosody====
For backing up prosody, the following were copied:

* Dump of the database using `mysqldump`
* `/var/lib/prosody` for media files
* `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server
<syntaxhighlight lang="bash">
ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup.sql.gz
</syntaxhighlight>

Backup of `/var/lb/prosody` was taken using following steps:

* Create a tar file of prosody directory
<syntaxhighlight>
cd /var/lib && sudo tar -czvf ~user/var.lib.prosody-2025-01-07.tar.gz prosody
</syntaxhighlight>

* Make user as owner of compressed file:

<syntaxhighlight>
cd && chown user: var.lib.prosody-2025-01-07.tar.gz
</syntaxhighlight>

* Use `scp` to transfer tar file to the backup-server
<syntaxhighlight>
scp -P <port-for-ssh-on-backup-server> ./var.lib.prosody-2025-01-07.tar.gz backup-user@backup-server:directory-to-backup
</syntaxhighlight>

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2025-10-07T20:42:59Z

Ravi: Replace Riot with Element

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Element client (accessed by a web browser), which can be used to connect to any Matrix server without installing the Element app.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service
matrix-synapse@federation_reader.service
matrix-synapse@event_creator.service
matrix-synapse@federation_sender.service
matrix-synapse@pusher.service
matrix-synapse@user_dir.service
matrix-synapse@media_repository.service
matrix-synapse@frontend_proxy.service
matrix-synapse@client_reader.service
matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:

* Dump of postgresql database using `pg_dump`
* `/etc/matrix-synapse` - contains config files
* `/var/lib/static/synapse/media` -- contains uploaded media files

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:<syntaxhighlight lang="bash">
eval "$(ssh-agent -s)"
</syntaxhighlight>followed by<syntaxhighlight>
ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"
</syntaxhighlight>on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:<syntaxhighlight>
ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > synapse-2025-01-07.dump
</syntaxhighlight>

====Prosody====
For backing up prosody, the following were copied:

* Dump of the database using `mysqldump`
* `/var/lib/prosody` for media files
* `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server
<syntaxhighlight lang="bash">
ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup.sql.gz
</syntaxhighlight>

Backup of `/var/lb/prosody` was taken using following steps:

* Create a tar file of prosody directory
<syntaxhighlight>
cd /var/lib && sudo tar -czvf ~user/var.lib.prosody-2025-01-07.tar.gz prosody
</syntaxhighlight>

* Make user as owner of compressed file:

<syntaxhighlight>
cd && chown user: var.lib.prosody-2025-01-07.tar.gz
</syntaxhighlight>

* Use `scp` to transfer tar file to the backup-server
<syntaxhighlight>
scp -P <port-for-ssh-on-backup-server> ./var.lib.prosody-2025-01-07.tar.gz backup-user@backup-server:directory-to-backup
</syntaxhighlight>

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2025-10-07T13:10:41Z

Ravi: /* Workers */ It was very hard to distinguish one service from the other in the previous list. So, I made a new line for each of the services.

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service
matrix-synapse@federation_reader.service
matrix-synapse@event_creator.service
matrix-synapse@federation_sender.service
matrix-synapse@pusher.service
matrix-synapse@user_dir.service
matrix-synapse@media_repository.service
matrix-synapse@frontend_proxy.service
matrix-synapse@client_reader.service
matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:

* Dump of postgresql database using `pg_dump`
* `/etc/matrix-synapse` - contains config files
* `/var/lib/static/synapse/media` -- contains uploaded media files

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:<syntaxhighlight lang="bash">
eval "$(ssh-agent -s)"
</syntaxhighlight>followed by<syntaxhighlight>
ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"
</syntaxhighlight>on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:<syntaxhighlight>
ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > synapse-2025-01-07.dump
</syntaxhighlight>

====Prosody====
For backing up prosody, the following were copied:

* Dump of the database using `mysqldump`
* `/var/lib/prosody` for media files
* `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server
<syntaxhighlight lang="bash">
ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup.sql.gz
</syntaxhighlight>

Backup of `/var/lb/prosody` was taken using following steps:

* Create a tar file of prosody directory
<syntaxhighlight>
cd /var/lib && sudo tar -czvf ~user/var.lib.prosody-2025-01-07.tar.gz prosody
</syntaxhighlight>

* Make user as owner of compressed file:

<syntaxhighlight>
cd && chown user: var.lib.prosody-2025-01-07.tar.gz
</syntaxhighlight>

* Use `scp` to transfer tar file to the backup-server
<syntaxhighlight>
scp -P <port-for-ssh-on-backup-server> ./var.lib.prosody-2025-01-07.tar.gz backup-user@backup-server:directory-to-backup
</syntaxhighlight>

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2025-01-08T08:54:19Z

Ravi: /* Prosody */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:

* Dump of postgresql database using `pg_dump`
* `/etc/matrix-synapse` - contains config files
* `/var/lib/static/synapse/media` -- contains uploaded media files

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:<syntaxhighlight lang="bash">
eval "$(ssh-agent -s)"
</syntaxhighlight>followed by<syntaxhighlight>
ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"
</syntaxhighlight>on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:<syntaxhighlight>
ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > synapse-2025-01-07.dump
</syntaxhighlight>

====Prosody====
For backing up prosody, the following were copied:

* Dump of the database using `mysqldump`
* `/var/lib/prosody` for media files
* `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server
<syntaxhighlight lang="bash">
ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup.sql.gz
</syntaxhighlight>

Backup of `/var/lb/prosody` was taken using following steps:

* Create a tar file of prosody directory
<syntaxhighlight>
cd /var/lib && sudo tar -czvf ~user/var.lib.prosody-2025-01-07.tar.gz prosody
</syntaxhighlight>

* Make user as owner of compressed file:

<syntaxhighlight>
cd && chown user: var.lib.prosody-2025-01-07.tar.gz
</syntaxhighlight>

* Use `scp` to transfer tar file to the backup-server
<syntaxhighlight>
scp -P <port-for-ssh-on-backup-server> ./var.lib.prosody-2025-01-07.tar.gz backup-user@backup-server:directory-to-backup
</syntaxhighlight>

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2025-01-08T08:48:07Z

Ravi: /* Prosody */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:

* Dump of postgresql database using `pg_dump`
* `/etc/matrix-synapse` - contains config files
* `/var/lib/static/synapse/media` -- contains uploaded media files

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:<syntaxhighlight lang="bash">
eval "$(ssh-agent -s)"
</syntaxhighlight>followed by<syntaxhighlight>
ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"
</syntaxhighlight>on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:<syntaxhighlight>
ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > synapse-2025-01-07.dump
</syntaxhighlight>

====Prosody====
For backing up prosody, the following were copied:

* Dump of the database using `mysqldump`
* `/var/lib/prosody` for media files
* `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server
<syntaxhighlight lang="bash">
ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup-2024-10-27.sql.gz
</syntaxhighlight>

Backup of `/var/lb/prosody` was taken using following steps:

* Create a tar file of prosody directory
<syntaxhighlight>
cd /var/lib && sudo tar -czvf ~user/var.lib.prosody-2025-01-07.tar.gz prosody
</syntaxhighlight>

* Make user as owner of compressed file:

<syntaxhighlight>
cd && chown user: var.lib.prosody-2025-01-07.tar.gz
</syntaxhighlight>

* Use `scp` to transfer tar file to the backup-server
<syntaxhighlight>
scp -P <port-for-ssh-on-backup-server> ./var.lib.prosody-2025-01-07.tar.gz backup-user@backup-server:directory-to-backup
</syntaxhighlight>

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2025-01-08T08:41:26Z

Ravi: /* Backup steps on 7th Jan 2025 */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:

* Dump of postgresql database using `pg_dump`
* `/etc/matrix-synapse` - contains config files
* `/var/lib/static/synapse/media` -- contains uploaded media files

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:<syntaxhighlight lang="bash">
eval "$(ssh-agent -s)"
</syntaxhighlight>followed by<syntaxhighlight>
ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"
</syntaxhighlight>on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:<syntaxhighlight>
ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > synapse-2025-01-07.dump
</syntaxhighlight>

====Prosody====
For backing up prosody, the following were copied:

* Dump of the database using `mysqldump`
* `/var/lib/prosody` for media files
* `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server<syntaxhighlight lang="bash">
ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup-2024-10-27.sql.gz
</syntaxhighlight>

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2025-01-08T08:35:29Z

Ravi: /* Prosody */

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:
- Dump of postgresql using `pg_dump`
- `/etc/matrix-synapse`
- `/var/lib/static/synapse/media`

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:

`eval "$(ssh-agent -s)"`

followed by

`ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"`
on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:

`ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > poddery-backups/matrix-synapse/synapse-2025-01-07.dump`

====Prosody====
The following were backed up for prosody:

* Dump of the database using `mysqldump`
* `/var/lib/prosody` for media files
* `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server<syntaxhighlight lang="bash">
ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup-2024-10-27.sql.gz
</syntaxhighlight>

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

Poddery - Diaspora, Matrix and XMPP

2025-01-08T08:31:33Z

Ravi: Add backup steps

We run decentralized and federated [https://diasporafoundation.org/ Diaspora] social netowrk, [https://xmpp.org/ XMPP] and [https://matrix.org Matrix] instant messaging services at [https://poddery.com poddery.com]. Along with Diaspora, Poddery username and password can be used to access XMPP and Matrix services as well. [https://chat.poddery.com chat.poddery.com] provides Riot client (accessed by a web browser), which can be used to connect to any Matrix server without installing a Riot app/client.

= Environment =
== Hosting ==
Poddery is hosted at [https://www.hetzner.com Hetzner] with the following specs:

* Intel Xeon E3-1246V3 Process - 4 Cores, 3.5GHz
* 4TB HDD
* 32GB DDR3 RAM

== Operating System ==
* Debian Buster

== User Visible Services ==
=== Diaspora ===
* Currently installed version is 0.7.6.1 which is available in [https://packages.debian.org/buster/diaspora-installer Debian Buster contrib]
* For live statistics see https://poddery.com/statistics

=== Chat/XMPP ===
* [https://prosody.im/ Prosody] is used as the XMPP server which is modern and lightweight.
* Currently installed version is 0.11.2 which is available in [https://packages.debian.org/buster/prosody Debian Buster].
* All XEPs are enabled which the [https://conversations.im/ Conversations app] support.

=== Chat/Matrix ===
* [https://matrix.org/docs/projects/server/synapse.html Synapse] is used as the Matrix server.
* Synapse is currently installed directly from the [https://github.com/matrix-org/synapse official GitHub repo].
* Riot-web Matrix client is hosted at https://chat.poddery.com

=== Homepage ===
Homepage and other static pages are maintained in FSCI [https://git.fosscommunity.in GitLab instance].
* poddery.com -> https://git.fosscommunity.in/community/poddery.com
* save.poddery.com -> https://git.fosscommunity.in/community/save.poddery.com
* fund.poddery.com -> https://git.fosscommunity.in/community/fund-poddery

== Backend Services ==
=== Web Server / Reverse Proxy ===
* Nginx web server which also acts as front-end (reverse proxy) for Diaspora and Matrix.

=== Database ===
* PostgreSQL for Matrix
* MySQL for Diaspora

''TODO'': Consider migrating to PostgreSQL to optimize resources (We can reduce one service and RAM usage).

=== Email ===
* Exim

=== SSL/TLS certificates ===
* Let's Encrypt

=== Firewall ===
* UFW (Uncomplicated Firewall)

=== Intrusion Prevention ===
* Fail2ban

= Coordination =
* [https://codema.in/g/2bjVXqAu/fosscommunity-in-poddery-com-maintainer-s-group Loomio group] - Mainly used for decision making
* Matrix room - [https://matrix.to/#/#poddery:poddery.com #poddery:poddery.com] also bridged to xmpp [xmpp:poddery.com-support@chat.yax.im?join poddery.com-support@chat.yax.im]
* [https://git.fosscommunity.in/community/poddery.com/issues Issue tracker] - Used for tracking progress of tasks

=== Contact ===
* Email: poddery at autistici.org (alias that reaches Akhilan, Abhijith Balan, Fayad, Balasankar, Julius, Praveen, Prasobh, Sruthi, Shirish, Vamsee and Manukrishnan)
* The following people have their GPG keys in the [[#Server_Access|access file]]:
** ID: 0xCE1F9C674512C22A - Praveen Arimbrathodiyil (piratepin)
** ID: 0xB77D2E2E23735427 - Balasankar C
** ID: 0x5D0064186AF037D9 - Manu Krishnan T V
** ID: 0x51C954405D432381 - Fayad Fami (fayad)
** ID: 0x863D4DF2ED9C28EF - Abhijith PA
** ID: 0x6EF48CCD865A1FFC - Syam G Krishnan (sgk)
** ID: 0xFD49D0BC6FEAECDA - Sagar Ippalpalli
** ID: 0x92FDAB42A95FF20C - Pirate Bady (piratesin)
** ID: 0x0B1955F40C691CCE - Kannan
** ID: 0x32FF6C6F5B7AE248 - Akhil Varkey
** ID: 0xFBB7061C27CB70C1 - Ranjith Siji
** ID: 0xEAAFE4A8F39DE34F - Kiran S Kunjumon (hacksk)
* It's recommended to setup [http://www.vim.org/scripts/script.php?script_id=3645 Vim GnuPG Plugin] for transparent editing. Those who are new to GPG can follow [https://www.madboa.com/geek/gpg-quickstart/ this guide].

=== Server Access ===
Maintained in a private git repo at https://git.fosscommunity.in/community/access

= Configuration and Maintenance =

Boot into rescue system using https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system

== Disk Partitioning ==
* RAID 1 setup on 2x2TB HDDs (<code>sda</code> and <code>sdb</code>).
mdadm --verbose --create /dev/mdX --level=mirror --raid-devices=2 /dev/sdaY /dev/sdbY
* Separate partitions for swap (<code>md0</code> - 16GB), boot (<code>md1</code> - 512MB) and root (<code>md2</code> - 50GB).
* LVM on Luks for separate encrypted data partitions for database, static files and logs.
# Setup LUKS (make sure <code>lvm2</code>, <code>udev</code> and <code>cryptsetup</code> packages are installed).
cryptsetup luksFormat /dev/mdX
# Give disk encryption password as specified in the [[#Server_Access|access repo]]
cryptsetup luksOpen /dev/mdX poddery

# LVM Setup
# Create physical volume named <code>poddery</code>
pvcreate /dev/mapper/poddery
# Create volume group named <code>data</code>
vgcreate data /dev/mapper/poddery
# Create logical volumes named <code>log</code>, <code>db</code> and <code>static</code>
lvcreate -n log /dev/data -L 50G
lvcreate -n db /dev/data -L 500G
# Assign remaining free space for static files
lvcreate -n static /dev/data -l 100%FREE

# Setup filesystem on the logical volumes
mkfs.ext4 /dev/data/log
mkfs.ext4 /dev/data/db
mkfs.ext4 /dev/data/static

# Create directories for mounting the encrypted partitions
mkdir /var/lib/db /var/lib/static /var/log/poddery

# Manually mount encrypted partitions. This is needed on each reboot as Hetzner doesn't provide a web console so that we can't decrypt the partitions during booting.
mount /dev/data/db /var/lib/db
mount /dev/data/static /var/lib/static
mount /dev/data/log /var/log/poddery

== Hardening checklist ==
* SSH password based login disabled (allow only key based logins)
* SSH login disabled for root user (use a normal user with sudo)
# Check for the following settings in /etc/ssh/sshd_config:
...
PermitRootLogin no
...
PasswordAuthentication no
...

* <code>ufw</code> firewall enabled with only the ports that needs to be opened ([https://fxdata.cloud/tutorials/set-up-a-firewall-with-ufw-on-ubuntu-16-04 ufw tutorial]):
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http/tcp
ufw allow https/tcp
ufw allow Turnserver
ufw allow XMPP
ufw allow 8448

ufw enable

# Verify everything is setup properly
ufw status

# Enable ufw logging with default mode low
ufw logging on

* <code>fail2ban</code> configured against brute force attacks:
# Check for the following line <code>/etc/ssh/sshd_config</code>
...
LogLevel VERBOSE
...

# Restart SSH and enable fail2ban
systemctl restart ssh
systemctl enable fail2ban
systemctl start fail2ban

# To unban an IP, first check <code>/var/log/fail2ban.log</code> to get the banned IP and then run the following
# Here <code>sshd</code> is the defaut jail name, change it if you are using a different jail
fail2ban-client set sshd unbanip <banned_ip>

== Diaspora ==
* Install <code>diaspora-installer</code> from Debian Buster contrib:
apt install diaspora-installer

* Move MySQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/db/
ln -s /var/lib/db/mysql /var/lib/
systemctl start mysql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/diaspora
mv /usr/share/diaspora/public/uploads /var/lib/static/diaspora
ln -s /var/lib/static/diaspora/uploads /usr/share/diaspora/public/
chown -R diaspora: /var/lib/static/diaspora

* Modify configuration files at <code>/etc/diaspora</code> and <code>/etc/diaspora.conf</code> as needed (backup of the current configuration files are available in the [[#Server_Access|access repo]]).
* Homepage configuration:
# Make sure <code>git</code> and <code>acl</code> packages are installed
# Grant <code>rwx</code> permissions for the ssh user to <code>/usr/share/diaspora/public</code>
setfacl -m "u:<ssh_user>:rwx" /usr/share/diaspora/public

# Clone poddery.com repo
cd /usr/share/diaspora/public
git clone https://git.fosscommunity.in/community/poddery.com.git
cd poddery.com && mv * .[^.]* .. #Give yes for all files when prompted
cd .. && rmdir poddery.com

* [https://save.poddery.com Save Poddery] repo is maintained as a sub module in poddery.com repo. See this [https://chrisjean.com/git-submodules-adding-using-removing-and-updating/ tutorial] for working with git submodules.
# Clone save.poddery.com repo
cd /usr/share/diaspora/public/save
git submodule init
git submodule update

== Matrix ==
* See the [https://github.com/matrix-org/synapse/blob/master/INSTALL.md official installation guide] of Synapse for installing from source.
* Nginx is used as reverse proxy to send requests that has <code>/_matrix/*</code> in URL to Synapse on port <code>8008</code>. This is configured in <code>/etc/nginx/sites-enabled/diaspora</code>.
* Shamil's [https://git.fosscommunity.in/necessary129/synapse-diaspora-auth Synapse Diaspora Auth] script is used to authenticate Synapse with Diaspora database.
* Move PostgreSQL data to encrypted partition:
# Make sure <code>/dev/data/db</code> is mounted to <code>/var/lib/db</code>
systemctl stop postgresql
systemctl disable postgresql
mv /var/lib/postgres /var/lib/db/
ln -s /var/lib/db/postgres /var/lib/
systemctl start postgresql

* Move static files to encrypted partition:
# Make sure <code>/dev/data/static</code> is mounted to <code>/var/lib/static</code>
mkdir /var/lib/static/synapse
mv /var/lib/matrix-synapse/uploads /var/lib/static/synapse/
ln -s /var/lib/static/synapse/uploads /var/lib/matrix-synapse/
mv /var/lib/matrix-synapse/media /var/lib/static/synapse/
ln -s /var/lib/static/synapse/media /var/lib/matrix-synapse/
chown -R matrix-synapse: /var/lib/static/synapse

* Install identity server <code>mxisd</code> (<code>deb</code> package available [https://github.com/kamax-matrix/mxisd/blob/master/docs/install/debian.md here])

=== Workers ===
* For scalability, Poddery is running [https://github.com/matrix-org/synapse/blob/master/docs/workers.md workers]. Currently all workers specified in that page, expect <code>synapse.app.appservice</code> is running on poddery.com
* A new service [https://gist.github.com/necessary129/5dfbb140e4727496b0ad2bf801c10fdc <code>matrix-synapse@.service</code>] is installed for the workers (Save the <code>synape_worker</code> file somewhere like <code>/usr/local/bin/</code> or something).
* The worker config can be found at <code>/etc/matrix-synapse/workers</code>
* Synapse needs to be put under a reverse proxy see <code>/etc/nginx/sites-enabled/matrix</code>. A lot of <code>/_matrix/</code> urls needs to be overridden too see <code>/etc/nginx/sites-enabled/diaspora</code>
* These lines must be added to <code>homeserver.yaml</code> as we are running <code>media_repository</code>, <code>federation_sender</code>, <code>pusher</code>, <code>user_dir</code> workers respectively:
enable_media_repo: False
send_federation: False
start_pushers: False
update_user_directory: false

* These services must be enabled:

matrix-synapse@synchrotron.service matrix-synapse@federation_reader.service matrix-synapse@event_creator.service matrix-synapse@federation_sender.service matrix-synapse@pusher.service matrix-synapse@user_dir.service matrix-synapse@media_repository.service matrix-synapse@frontend_proxy.service matrix-synapse@client_reader.service matrix-synapse@synchrotron_2.service

To load balance between the 2 synchrotrons, We are running [https://github.com/Sorunome/matrix-synchrotron-balancer matrix-synchrotron-balancer]. It has a systemd file at <code>/etc/systemd/system/matrix-synchrotron-balancer</code>. The files are in <code>/opt/matrix-synchrotron-balancer</code>

=== Synapse Updation ===
* First check [https://matrix-org.github.io/synapse/latest/upgrade synapse//latest/upgrade] to see if anything extra needs to be done. Then, just run <code>/root/upgrade-synapse</code>
* Current version of synapse can be found from https://poddery.com/_matrix/federation/v1/version

=== Riot-web Updation ===
* Just run the following (make sure to replace <code><version></code> with a proper version number like <code>v1.0.0</code>):
/var/www/get-riot <version>

== Chat/XMPP ==
* Steps for setting up Prosody is given at https://wiki.debian.org/Diaspora/XMPP
# Follow steps 1 to 6 from https://wiki.debian.org/Diaspora/XMPP and then run the following:
mysql -u root -p # Enter password from the access repo

CREATE USER 'prosody'@'localhost' IDENTIFIED BY '<passwd_in_repo>';
GRANT ALL PRIVILEGES ON diaspora_production.* TO 'prosody'@'localhost';
FLUSH PRIVILEGES;

systemctl restart prosody

* Install plugins
# Make sure <code>mercurial</code> is installed
cd /etc && hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

=== Set Nginx Conf for BOSH URLS ===
* Add the following in <code>nginx</code> configuration file to enable the BOSH URL to make JSXC Working:
upstream chat_cluster {
server localhost:5280;
}

location /http-bind {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_connect_timeout 5;
proxy_buffering off;
proxy_read_timeout 70;
keepalive_timeout 70;
send_timeout 70;
client_max_body_size 4M;
client_body_buffer_size 128K;
proxy_pass http://chat_cluster;
}

* [https://wiki.diasporafoundation.org/Integration/Chat#Nginx See here] for more details on <code>nginx</code> configuration. Alternatively, <code>apache</code> settings can be found [https://github.com/jsxc/jsxc/wiki/Prepare-apache here].

== TLS ==
* Install <code>letsencrypt</code>.
* Ensure proper permissions are set for <code>/etc/letsencrypt</code> and its contents.
chown -R root:ssl-cert /etc/letsencrypt
chmod g+r -R /etc/letsencrypt
chmod g+x /etc/letsencrypt/{archive,live}
* Generate certificates. For more details see https://certbot.eff.org.
* Make sure the certificates used by <code>diaspora</code> are symbolic links to letsencrypt default location:
ls -l /etc/diaspora/ssl
''total 0
''lrwxrwxrwx 1 root root 47 Apr 2 22:47 poddery.com-bundle.pem -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 45 Apr 2 22:48 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/diaspora/ssl/poddery.com-bundle.pem
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/diaspora/ssl/poddery.com.key

* Make sure the certificates used by <code>prosody</code> are symbolic links to letsencrypt default location:
ls -l /etc/prosody/certs/
''total 0''
''lrwxrwxrwx 1 root root 40 Mar 28 01:16 poddery.com.crt -> /etc/letsencrypt/live/poddery.com/fullchain.pem''
''lrwxrwxrwx 1 root root 33 Mar 28 01:16 poddery.com.key -> /etc/letsencrypt/live/poddery.com/privkey.pem''

# If you don't get the above output, then run the following:
cp -L /etc/letsencrypt/live/poddery.com/fullchain.pem /etc/prosody/certs/poddery.com.crt
cp -L /etc/letsencrypt/live/poddery.com/privkey.pem /etc/prosody/certs/poddery.com.key

* Note- letsencrypt executable used below is actually a symlik to /usr/bin/certbot
* Cron jobs:
crontab -e
''30 2 * * 1 letsencrypt renew >> /var/log/le-renew.log''
''32 2 * * 1 /etc/init.d/nginx reload''
''34 2 * * 1 /etc/init.d/prosody reload''

* Manually updating TLS certificate:
letsencrypt certonly --webroot --agree-tos -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save -d save.poddery.com -w /var/www/riot -d chat.poddery.com
* To include an additional subdomain such as fund.poddery.com use with --expand parameter as shown below
letsencrypt certonly --webroot --agree-tos --expand -w /usr/share/diaspora/public -d poddery.com -d www.poddery.com -d test.poddery.com -d groups.poddery.com -d fund.poddery.com -w /usr/share/diaspora/public/save/ -d save.poddery.com -w /var/www/riot/ -d chat.poddery.com

==Backup==

Backup server is provided by Manu (KVM virtual machine with 180 GB storage and 1 GB ram ).

Debian Stetch was upgraded Debian Buster before database relication of synapse database.

Documentation: https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

Currently postgres database for matrix-synapse is backed up.

===Before Replication (specific to poddery.com)===

Setup tinc vpn in the backup server

# apt install tinc

Configure tinc by creating tinc.conf and host podderybackup under label fsci.
Add tinc-up and tinc-down scripts
Copy poddery host config to backup server and podderybackup host config to poddery.com server.

Reload tinc vpn service at both poddery.com and backup servers

# systemctl reload tinc@fsci.service

Enable tinc@fsci systemd service for autostart

# systemctl enable tinc@fsci.service

The synapse database was also pruned to reduce the size before replication by following this guide - https://levans.fr/shrink-synapse-database.html
If you want to follow this guide, make sure matrix synapse server is updated to version 1.13 atleast since it introduces the Rooms API mentioned the guide.
Changes done to steps in the guide.

# jq '.rooms[] | select(.joined_local_members == 0) | .room_id' < roomlist.json | sed -e 's/"//g' > to_purge.txt

The room list obtained this way can, be looped to pass the room names as variables to the purge api.

# set +H // if you are using bash to avoid '!' in the roomname triggering the history substitution.
# for room_id in $(cat to_purge.txt); do curl --header "Authorization: Bearer <your access token>" \
-X POST -H "Content-Type: application/json" -d "{ \"room_id\": \"$room_id\" }" \
'https://127.0.0.1:8008/_synapse/admin/v1/purge_room'; done;

We also did not remove old history of large rooms.

===Step 1: Postgresql (for synapse) Primary configuration===

Create postgresql user for replication.

$ psql -c "CREATE USER replication REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'yourpassword';"
The password is in the access repo if you need it later.

Allow standby to connect to primary using the user just created.

$ cd /etc/postgresql/11/main

$ nano pg_hba.conf

Add below line to allow replication user to get access to the server

host replication replication 172.16.0.3/32 md5

Next , open the postgres configuration file

nano postgresql.conf

Set the following configuration options in the postgresql.conf file

listen_addresses = 'localhost,172.16.0.2'
port=5432
wal_level = replica
max_wal_senders = 1
wal_keep_segments = 64
archive_mode = on
archive_command = 'cd .'

You need to restart since postgresql.conf was edited and parameters changed,

# systemctl restart postgresql

===Step 2: Postgresql (for synapse) Standby configuration ===

Install postgresql

# apt install postgresql

Check postgresql server is running

# su postgres -c psql

Make sure en_US.UTF-8 locale is available

# dpkg-reconfigure locales

Stop postgresql before changing any configuration

#systemctl stop postgresql@11-main

Switch to postgres user

# su - postgres
$ cd /etc/postgresql/11/

Copy data from master and create recovery.conf

$ pg_basebackup -h git.fosscommunity.in -D /var/lib/postgresql/11/main/ -P -U rep --wal-method=fetch -R

Open the postgres configuration file

$ nano postgresql.conf

Set the following configuration options in the postgresql.conf file

max_connections = 500 // This option and the one below are set to be same as in postgresql.conf at primary or the service won't start.
max_worker_processes = 16
host_standby = on // The above pg_basebackup command should set it. If it's not manually turn it to on.

Start the stopped postgresql service

# systemctl start postgresql@11-main

===Postgresql (for synapse) Replication Status===

On Primary,

$ ps -ef | grep sender
$ psql -c "select * from pg_stat_activity where usename='rep';"

On Standby,

$ ps -ef | grep receiver

===Backup steps on 7th Jan 2025===
====Matrix-synapse====
For synapse, the following files were backed up:
- Dump of postgresql using `pg_dump`
- `/etc/matrix-synapse`
- `/var/lib/static/synapse/media`

In order to access the poddery server from the backup server (with your public ssh keys added to both the servers in `~/.ssh/authorized-keys`), run the following command in your local system:

`eval "$(ssh-agent -s)"`

followed by

`ssh user@server -o "ForwardAgent yes" -o "AddKeysToAgent yes"`
on the local system.

The dump was taken using the command from the [https://element-hq.github.io/synapse/latest/usage/administration/backups.html#quick-and-easy-database-backup-and-restore official docs]:

`ssh user@poddery-server 'sudo -u postgres pg_dump -Fc --exclude-table-data e2e_one_time_keys_json synapse' > poddery-backups/matrix-synapse/synapse-2025-01-07.dump`

====Prosody====
The following were backed up for prosody:
- Dump of the database using `mysqldump`
- `/var/lib/prosody` for media files
- `/etc/prosody` for config files

For taking the dump, the following was run from the backup-server
`ssh user@poddery-server 'mysqldump -u prosody --password="$(cat <path/to/password-file>)" prosody | gzip' > backups/prosody-backup-2024-10-27.sql.gz`

= Troubleshooting =
== Allow XMPP login even if diaspora account is closed ==
Diaspora has a [https://github.com/diaspora/diaspora/blob/develop/Changelog.md#new-maintenance-feature-to-automatically-expire-inactive-accounts default setting] to close accounts that have been inactive for 2 years. At the time of writing, there seems [https://github.com/diaspora/diaspora/issues/5358#issuecomment-371921462 no way] to reopen a closed account. This also means that if your account is closed, you will no longer be able to login to the associated XMPP service as well. Here we discuss a workaround to get access back to the XMPP account.

The prosody module [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua mod_auth_diaspora] is used for diaspora-based XMPP auth. It checks if <code>locked_at</code> value in the <code>users</code> table of diaspora db is <code>null</code> [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L89 here] and [https://gist.github.com/jhass/948e8e8d87b9143f97ad#file-mod_auth_diaspora-lua-L98 here]. If your account is locked, it will have the <code>datetime</code> value that represents the date and time at which your account is locked. Setting it back to <code>null</code> will let you use your XMPP account again.

-- Replace <username> with actual username of the locked account
UPDATE users SET locked_at=NULL WHERE username='<username>';

NOTE: Matrix account won't be affected even if the associated diaspora account is closed because it uses a [https://pypi.org/project/synapse-diaspora-auth/ custom auth module] which works differently.

= History =
* [[Poddery/Archive|See here]] for the archive of Poddery wiki page before the migration to Hetzner.

[[Category:Services]]

User:Ravi

2022-01-30T22:11:07Z

Ravi:

Name: Ravi

Webpage: https://ravidwivedi.in

User:Ravi

2022-01-30T22:10:46Z

Ravi: Created page with "Name: Ravi Webpage: https://ravidwivedi.in"

Name: Ravi
Webpage: https://ravidwivedi.in

Learn Debian Packaging

2022-01-30T22:06:58Z

Ravi: /* Level 0: Basics of release process and setup a development environment */

We are guiding some learners of [https://camp.fsf.org.in Free Software Camp] to learn Debian Packaging and this page will be used to track the progress of the tasks.

== Level 0: Basics of release process and setup a development environment ==
* [http://www.queryadmin.com/2203/how-to-install-a-deb-file-on-debian-linux-via-command-line/ How to Install a .Deb File via Command-Line]
* [https://debian-handbook.info/browse/stable/sect.release-lifecycle.html Lifecycle of a Release]
* [https://raphaelhertzog.com/2010/10/18/understanding-debians-release-process/ Understanding Debian’s release process]
* [https://backports.debian.org/Instructions/ How to install packages from stable-backports]
* [https://wiki.debian.org/Packaging/Pre-Requisites Different options for setting up a Debian Sid environment]
* [https://wiki.debian.org/BuildingTutorial Building existing packages from source]

By this time you should be familiar with
# apt source/dget,
# dpkg-source -x,
# dpkg-buildpackage/debuild,
# apt build-dep,
# apt-source -b
commands to rebuild an existing debian package from source.

== Level 1: Learn basics of Packaging ==

Understand the basic concepts using debmake/dh_make (getting source tarballs, creating source package, building the binary package, making it lintian clean)

* [https://wiki.abrahamraji.in/simple-packaging-tutorial/ Abraham Raji's simple packaging tutorial]
* [https://wiki.debian.org/SimplePackagingTutorial Simple Packaging Tutorial on Debian Wiki]

Once you understand the basic concepts, use npm2deb to automate some of those tasks like getting source tarball, a better debian directory template than the ones created by dh_make/debmake as npm2deb knows more details specific to node modules. You will still have to fix the remaining issues flagged by lintian.

* [https://wiki.debian.org/Javascript/Nodejs/Npm2Deb/Tutorial npm2deb Tutorial on Debian Wiki]

By this time you should know,
# creating lintian clean packages for simple modules and
# building it in a clean environment like sbuild.
# You should also know to import a dsc file to a git repo (gbp import-dsc --pristine-tar) and
# push your work to a public git hosting service like salsa.debian.org (git push -u --all --follow-tags)

== Level 2: Update existing packages to new upstream minor or patch versions ==
Once you get a clear picture of packaging a simple module, we can move to the next stage of updating existing packages

* [https://wiki.debian.org/UpdatingaPackagetoNewUpstreamVersion Update packages to new upstream version on Debian Wiki]
* [https://wiki.debian.org/UsingQuilt Using Quilt on Debian Wiki]

# How to send RFS mails
# Using Quilt to modify upstream source if required

== Level 3: Packaging more complicated modules ==

Next step is packaging more complicated modules that will involve things like, modifying some upstream files, removing some files from source tarball, generating some files from source, getting the source tarball from a git commit etc.

* [https://wiki.debian.org/Javascript/Nodejs/Npm2Deb/AdvancedTutorial Advanced tutorial for more complicated modules]

By this time you should know,
# Creating patches with quilt
# Repacking orig.tar and exclude specific files
# Use pkg-js-tools options to build from source files
# Build packages with typescript sources

== Level 4: Pick an unpackaged but useful module and upload to archive ==

* [https://git.fosscommunity.in/debian-ruby/TaskTracker/-/issues/175 List of node dependencies for gitlab]

By this time you should know,
# How to file ITP

Learn Debian Packaging

2022-01-30T14:55:47Z

Ravi: /* Level 0: Basics of release process and setup a development environment */

We are guiding some learners of [https://camp.fsf.org.in Free Software Camp] to learn Debian Packaging and this page will be used to track the progress of the tasks.

== Level 0: Basics of release process and setup a development environment ==
* [http://www.queryadmin.com/2203/how-to-install-a-deb-file-on-debian-linux-via-command-line/ How to Install a .Deb File via Command-Line]
* [https://debian-handbook.info/browse/stable/sect.release-lifecycle.html/ Lifecycle of a Release]
* [https://raphaelhertzog.com/2010/10/18/understanding-debians-release-process/ Understanding Debian’s release process]
* [https://backports.debian.org/Instructions/ How to install packages from stable-backports]
* [https://wiki.debian.org/Packaging/Pre-Requisites Different options for setting up a Debian Sid environment]
* [https://wiki.debian.org/BuildingTutorial Building existing packages from source]

By this time you should be familiar with
# apt source/dget,
# dpkg-source -x,
# dpkg-buildpackage/debuild,
# apt build-dep,
# apt-source -b
commands to rebuild an existing debian package from source.

== Level 1: Learn basics of Packaging ==

Understand the basic concepts using debmake/dh_make (getting source tarballs, creating source package, building the binary package, making it lintian clean)

* [https://wiki.abrahamraji.in/simple-packaging-tutorial/ Abraham Raji's simple packaging tutorial]
* [https://wiki.debian.org/SimplePackagingTutorial Simple Packaging Tutorial on Debian Wiki]

Once you understand the basic concepts, use npm2deb to automate some of those tasks like getting source tarball, a better debian directory template than the ones created by dh_make/debmake as npm2deb knows more details specific to node modules. You will still have to fix the remaining issues flagged by lintian.

* [https://wiki.debian.org/Javascript/Nodejs/Npm2Deb/Tutorial npm2deb Tutorial on Debian Wiki]

By this time you should know,
# creating lintian clean packages for simple modules and
# building it in a clean environment like sbuild.
# You should also know to import a dsc file to a git repo (gbp import-dsc --pristine-tar) and
# push your work to a public git hosting service like salsa.debian.org (git push -u --all --follow-tags)

== Level 2: Update existing packages to new upstream minor or patch versions ==
Once you get a clear picture of packaging a simple module, we can move to the next stage of updating existing packages

* [https://wiki.debian.org/UpdatingaPackagetoNewUpstreamVersion Update packages to new upstream version on Debian Wiki]
* [https://wiki.debian.org/UsingQuilt Using Quilt on Debian Wiki]

# How to send RFS mails
# Using Quilt to modify upstream source if required

== Level 3: Packaging more complicated modules ==

Next step is packaging more complicated modules that will involve things like, modifying some upstream files, removing some files from source tarball, generating some files from source, getting the source tarball from a git commit etc.

* [https://wiki.debian.org/Javascript/Nodejs/Npm2Deb/AdvancedTutorial Advanced tutorial for more complicated modules]

By this time you should know,
# Creating patches with quilt
# Repacking orig.tar and exclude specific files
# Use pkg-js-tools options to build from source files
# Build packages with typescript sources

== Level 4: Pick an unpackaged but useful module and upload to archive ==

* [https://git.fosscommunity.in/debian-ruby/TaskTracker/-/issues/175 List of node dependencies for gitlab]

By this time you should know,
# How to file ITP