Mailing list

Mailing lists are very useful communication and coordination tool. FSCI provides mailing lists to Free Libre Open Source (FLOSS) groups and non profit organizations. If you want to open a new public list, please drop a mail at postmaster(at)lists.fsci.org.in.

Environment

Machine Summary

We are on scaleway virtual cloud server.

* Cores	: 2 x86 cores
* Memory	: 2GB
* Disk	: 50GB
* OS	: Debian GNU/Linux
* Web Server	: Nginx
* List manager	: Mailman3
* Spam Filter	: Rspamd
* Host name	: lists.fsci.org.in

Coordination

• Hangout with us in our Matrix room #mailman:poddery.com
• issue tracker - we use this to track progress of tasks

Admins

1. Abhijith PA
2. Abhinav CK
3. Akhil Varkey

emeritus

1. Prinz Piuz
2. Balasankar C

Admin Documentation

Postfix Aliases

Apart from mailman's aliases. We have some system wide aliases for sending and receiving postfix related issues and warning. Those are kept in /etc/aliases .

Once done editing please run command `newaliases`

Generate DKIM for new domains

We use 'opendkim'[1] to implement DKIM. To generate DKIM keys, use opendkim-genkey -b 4096 -h rsa-sha256 -r -s <selector_name> -d <domain_name_of_this_list> -v`
We don't have any format on selector naming. Pick something related to this list(for eg: the selector for the domain lists.fsci.org.in is lists what we used). After generating the DKIM keys, we will get two files.

• <selector_name>.private - which contains the private signing key.
• <selector_name>.txt - which contains the DKIM TXT DNS record for that domain.

Now, create a folder under /etc/opendkim/keys/<domain_name_for_this_list>/ and move the above file to that location.
Create an entry for the newly created domain in /etc/opendkim/KeyTable as well as /etc/opendkim/SigningTable
On /etc/opendkim/KeyTable

<selector_name._domainkey.<domain_name_of_this_list> <domain_name_of_this_list>:<selector_name>:/etc/opendkim/keys/<domain_name_of_this_list>/<selector_name>.private
eg: lists._domainkey.lists.fsci.org.in lists.fsci.org.in:lists:/etc/opendkim/lists.private

On /etc/opendkim/SigningTable

<domain_name_of_this_list> <selector_name>._domainkey.<domain_name_of_this_list>
eg: lists.fsci.org.in lists._domainkey.lists.fsci.org.in

More on DKIM keys, formating etc: [DomainKeys Identified Mail], Setup DKIM with OpenDKIM on Debian: [How To Install and Configure DKIM with Postfix on Debian]

Verify DKIM and other settings for new domains

Create a text file, dkim-test.txt (replace To address with random address generated by https://dkimvalidator.com or https://www.mail-tester.com/)

From: "Test" <test-dkim@mm.gnu.org.in>
To: <random-address>@dkimvalidator.com
Subject: Testing DKIM and other settings for this domain
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

This email is to test email settings for a domain is correct

and use sendmail command to send it

/usr/sbin/sendmail -t < dkim-test.txt

Setup Rspamd

See [Rspamd quick start] for the latest installation and setup instructions. Install Rspamd. Redis will be installed automatically(check).

$ sudo apt install rspamd

Check service is running.

$ systemctl status rspamd.service
  ● rspamd.service - rapid spam filtering system
  Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
  Active: active (running) since Sat 2020-11-07 14:54:03 UTC; 1min 58s ago
    Docs: https://rspamd.com/doc/
    Main PID: 19339 (rspamd)
    Tasks: 5 (limit: 2336)
    Memory: 276.2M
    CGroup: /system.slice/rspamd.service
          ├─19339 rspamd: main process
          ├─19517 rspamd: rspamd_proxy process (localhost:11332)
          ├─19518 rspamd: controller process (localhost:11334)
          ├─19519 rspamd: normal process (localhost:11333)
          └─19520 rspamd: hs_helper process
    Nov 07 14:54:03 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.

Check if redis is working properly using redis-cli

$ redis-cli
 127.0.0.1:6379> ping
  PONG
 127.0.0.1:6379> ping Hello[qwe]
  "Hello[qwe]"
 127.0.0.1:6379> exit

Change some redis config to use it as a LRU cache[2]. On /etc/redis/redis.conf ~line 850

maxmemory 500mb
maxmemory-policy volatile-ttl
bind 127.0.0.1 ::1 //already exists

Configure rspamd

$ sudo rspamadm configwizard
 ____                                     _
|  _ \  ___  _ __    __ _  _ __ ___    __| |
| |_) |/ __|| '_ \  / _` || '_ ` _ \  / _` |
|  _ < \__ \| |_) || (_| || | | | | || (_| |
|_| \_\|___/| .__/  \__,_||_| |_| |_| \__,_|
            |_|
Welcome to the configuration tool
We use /etc/rspamd/rspamd.conf configuration file, writing results to /etc/rspamd
Modules enabled: hfilter, phishing, emails, asn, settings, chartable, arc, bayes_expiry, once_received, rbl, fuzzy_check, metadata_exporter, elastic, mid, multimap, spf, dkim_signing, dkim, mime_types, regexp, maillist, dmarc,   forged_recipients, milter_headers, whitelist, force_actions, trie
Modules disabled (explicitly): p0f, spamtrap, rspamd_update, mx_check, dcc
Modules disabled (unconfigured): spamassassin, maps_stats, metric_exporter, dynamic_conf, clustering, reputation, antivirus, fuzzy_collect, external_services, ip_score, clickhouse
Modules disabled (no Redis): greylist, url_redirector, replies, neural, ratelimit, history_redis
Modules disabled (experimental): 
Modules disabled (failed): 
Do you wish to continue?[Y/n]: 
Setup WebUI and controller worker:
Controller password is not set, do you want to set one?[Y/n]: Y
Enter passphrase: 
Set encrypted password to: <password_hash>
Redis servers are not set:
The following modules will be enabled if you add Redis servers:
   * greylist
   * url_redirector
   * replies
   * neural
   * ratelimit
   * history_redis
Do you wish to set Redis servers?[Y/n]: 
Input read only servers separated by `,` [default: localhost]: 
Input write only servers separated by `,` [default: localhost]: 
Do you have any password set for your Redis?[y/N]: 
Do you have any specific database for your Redis?[y/N]: 
Do you want to setup dkim signing feature?[y/N]: N
File: /etc/rspamd/local.d/redis.conf, changes list:
write_servers => localhost
read_servers => localhost

File: /etc/rspamd/local.d/worker-controller.inc, changes list:
password => <password_hash>

Apply changes?[Y/n]: Y
Create file /etc/rspamd/local.d/redis.conf
Create file /etc/rspamd/local.d/worker-controller.inc
2 changes applied, the wizard is finished now
*** Please reload the Rspamd configuration ***

Now, restart the rspamd service and check status

$ sudo systemctl restart rspamd
$ sudo systemctl status rspamd
  ● rspamd.service - rapid spam filtering system
    Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
    Active: active (running) since Sat 2020-11-07 15:28:14 UTC; 4s ago
      Docs: https://rspamd.com/doc/
    Main PID: 21899 (rspamd)
    Tasks: 5 (limit: 2336)
    Memory: 116.5M
    CGroup: /system.slice/rspamd.service
          ├─21899 rspamd: main process
          ├─21915 rspamd: rspamd_proxy process (localhost:11332)
          ├─21916 rspamd: controller process (localhost:11334)
          ├─21917 rspamd: normal process (localhost:11333)
          └─21918 rspamd: hs_helper process
    Nov 07 15:28:14 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.

Worker Proxy setup in /etc/rspamd/local.d/worker-proxy.inc

# local.d/worker-proxy.inc
milter = yes; # Enable milter mode
timeout = 120s; # Needed for Milter usually
upstream "local" {
  default = yes; # Self-scan upstreams are always default
  self_scan = yes; # Enable self-scan
}
count = 4; # Spawn more processes in self-scan mode
max_retries = 5; # How many times master is queried in case of failure
discard_on_reject = false; # Discard message instead of rejection
quarantine_on_reject = false; # Tell MTA to quarantine rejected messages
spam_header = "X-Spam"; # Use the specific spam header
reject_message = "Spam message rejected"; # Use custom rejection message

Disable DKIM Signing on rspamd

To disable DKIM signing (i.e. you use OpenDKIM, or signing is done elsewhere- ie, that's our case)
# local.d/dkim_signing.conf
enabled = false;

and restart rspamd service after that.

Setting up the WebUI
For using the location, <webroot>/rspamd,
Add the following on /etc/mailman3/nginx.conf

location /rspamd/ {
    proxy_pass       http://localhost:11334/;
    proxy_set_header Host      $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Now, check nginx syntax, restart the nginx service and check the status.
After restart, you can visit <webroot>/rspamd and login with the previously created password.

Setting up rspamd Worker proxy config on postfix
Rspamd Worker proxy config
On /etc/postfix/main.cf

Don't forget to disable existing  configs for the same below
##
## Spam filter and DKIM signatures via Rspamd
##
smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock
non_smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock
milter_protocol = 6
milter_mail_macros =  i {mail_addr} {client_addr} {client_name} {auth_authen}
milter_default_action = accept

and then restart postfix.

Finally, enable rspamd service if everything seems to work properly $ sudo systemctl enable rspamd.

Remove spam confirmation requests

Too many pending subscription requests can slow down[3] list index. Updating postorius is how to fix this. But in case there's no other way, and removing those requests through frontend [4] is impossible, this can be done (GNU Mailman 3.2.2 (La Villa Strangiato)):

$ sudo mailman shell -l listname.lists.fsci.org.in
Welcome to the GNU Mailman shell
The variable 'm' is the listname@lists.fsci.org.in mailing list
>>> from zope.component import getUtility
>>> from mailman.interfaces.pending import IPendings
>>> pendings = getUtility(IPendings)
>>> plist = []
>>> for p in pendings.find(m):
...   plist.append(p)
... 
>>> len(plist)
2047
>>> # Let us save the spammer email and when they signed up for analysis
>>> import csv
>>> with open('spam.csv', 'w', newline=) as csvfile:
...     spamwriter = csv.writer(csvfile, delimiter=',')
...     for p in plist:
...         spammer = p[1]
...         spamwriter.writerow([spammer.get('email'), spammer.get('when'), spammer.get('display_name')])
...
>>> # Now on to actually removing spam
>>> from mailman.interfaces.workflow import IWorkflowStateManager
>>> workflow = getUtility(IWorkflowStateManager)
>>> for p in plist:
...     token = p[0]
...     pendings.confirm(token)
...     workflow.discard(token)
...
>>> # Now, we have to commit this transaction to database
>>> from mailman.config import config
>>> config.db.commit()

---