Mailing lists are very useful communication and coordination tool. FSCI provides mailing lists to Free Libre Open Source (FLOSS) groups and non profit organizations. If you want to open a new public list, please drop a mail at postmaster(at)lists.fsci.org.in.
Environment
Machine Summary
We are on scaleway virtual cloud server.
| * Cores | : 2 x86 cores |
| * Memory | : 2GB |
| * Disk | : 50GB |
| * OS | : Debian GNU/Linux |
| * Web Server | : Nginx |
| * List manager | : Mailman3 |
| * Spam Filter | : Rspamd |
| * Host name | : lists.fsci.org.in |
Coordination
- Hangout with us in our Matrix room #mailman:poddery.com
- issue tracker - we use this to track progress of tasks
Admins
- Abhijith PA
- Abhinav CK
- Akhil Varkey
emeritus
- Prinz Piuz
- Balasankar C
Admin Documentation
Postfix Aliases
Apart from mailman's aliases. We have some system wide aliases for sending and receiving postfix related issues and warning. Those are kept in `/etc/aliases`.
Once done editing please run command `newaliases`
Generate DKIM for new domains
We use 'opendkim'[1] to implement DKIM.
To generate DKIM keys, use opendkim-genkey -b 4096 -h rsa-sha256 -r -s <selector_name> -d <domain_name_of_this_list> -v`
We don't have any format on selector naming. Pick something related to this list(for eg: the selector for the domain lists.fsci.org.in is lists what we used).
After generating the DKIM keys, we will get two files.
- <selector_name>.private - which contains the private signing key.
- <selector_name>.txt - which contains the DKIM TXT DNS record for that domain.
Now, create a folder under /etc/opendkim/keys/<domain_name_for_this_list>/ and move the above file to that location.
Create an entry for the newly created domain in /etc/opendkim/KeyTable as well as /etc/opendkim/SigningTable
On /etc/opendkim/KeyTable
<selector_name._domainkey.<domain_name_of_this_list> <domain_name_of_this_list>:<selector_name>:/etc/opendkim/keys/<domain_name_of_this_list>/<selector_name>.private eg: lists._domainkey.lists.fsci.org.in lists.fsci.org.in:lists:/etc/opendkim/lists.private
On /etc/opendkim/SigningTable
<domain_name_of_this_list> <selector_name>._domainkey.<domain_name_of_this_list> eg: lists.fsci.org.in lists._domainkey.lists.fsci.org.in
More on DKIM keys, formating etc: [DomainKeys Identified Mail], Setup DKIM with OpenDKIM on Debian: [How To Install and Configure DKIM with Postfix on Debian]
Verify DKIM and other settings for new domains
Create a text file, dkim-test.txt (replace To address with random address generated by https://dkimvalidator.com or https://www.mail-tester.com/)
From: "Test" <test-dkim@mm.gnu.org.in> To: <random-address>@dkimvalidator.com Subject: Testing DKIM and other settings for this domain Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit This email is to test email settings for a domain is correct
and use sendmail command to send it
/usr/sbin/sendmail -t < dkim-test.txt
Setup Rspamd
See [Rspamd quick start] for the latest installation and setup instructions. Install Rspamd. Redis will be installed automatically(check).
$ sudo apt install rspamd
Check service is running.
$ systemctl status rspamd.service
● rspamd.service - rapid spam filtering system
Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-11-07 14:54:03 UTC; 1min 58s ago
Docs: https://rspamd.com/doc/
Main PID: 19339 (rspamd)
Tasks: 5 (limit: 2336)
Memory: 276.2M
CGroup: /system.slice/rspamd.service
├─19339 rspamd: main process
├─19517 rspamd: rspamd_proxy process (localhost:11332)
├─19518 rspamd: controller process (localhost:11334)
├─19519 rspamd: normal process (localhost:11333)
└─19520 rspamd: hs_helper process
Nov 07 14:54:03 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.
Check if redis is working properly using redis-cli
$ redis-cli 127.0.0.1:6379> ping PONG 127.0.0.1:6379> ping Hello[qwe] "Hello[qwe]" 127.0.0.1:6379> exit
Change some redis config to use it as a LRU cache[2]. On /etc/redis/redis.conf ~line 850
maxmemory 500mb maxmemory-policy volatile-ttl bind 127.0.0.1 ::1 //already exists
Configure rspamd
$ sudo rspamadm configwizard
____ _
| _ \ ___ _ __ __ _ _ __ ___ __| |
| |_) |/ __|| '_ \ / _` || '_ ` _ \ / _` |
| _ < \__ \| |_) || (_| || | | | | || (_| |
|_| \_\|___/| .__/ \__,_||_| |_| |_| \__,_|
|_|
Welcome to the configuration tool
We use /etc/rspamd/rspamd.conf configuration file, writing results to /etc/rspamd
Modules enabled: hfilter, phishing, emails, asn, settings, chartable, arc, bayes_expiry, once_received, rbl, fuzzy_check, metadata_exporter, elastic, mid, multimap, spf, dkim_signing, dkim, mime_types, regexp, maillist, dmarc, forged_recipients, milter_headers, whitelist, force_actions, trie
Modules disabled (explicitly): p0f, spamtrap, rspamd_update, mx_check, dcc
Modules disabled (unconfigured): spamassassin, maps_stats, metric_exporter, dynamic_conf, clustering, reputation, antivirus, fuzzy_collect, external_services, ip_score, clickhouse
Modules disabled (no Redis): greylist, url_redirector, replies, neural, ratelimit, history_redis
Modules disabled (experimental):
Modules disabled (failed):
Do you wish to continue?[Y/n]:
Setup WebUI and controller worker:
Controller password is not set, do you want to set one?[Y/n]: Y
Enter passphrase:
Set encrypted password to: <password_hash>
Redis servers are not set:
The following modules will be enabled if you add Redis servers:
* greylist
* url_redirector
* replies
* neural
* ratelimit
* history_redis
Do you wish to set Redis servers?[Y/n]:
Input read only servers separated by `,` [default: localhost]:
Input write only servers separated by `,` [default: localhost]:
Do you have any password set for your Redis?[y/N]:
Do you have any specific database for your Redis?[y/N]:
Do you want to setup dkim signing feature?[y/N]: N
File: /etc/rspamd/local.d/redis.conf, changes list:
write_servers => localhost
read_servers => localhost
File: /etc/rspamd/local.d/worker-controller.inc, changes list: password => <password_hash>
Apply changes?[Y/n]: Y Create file /etc/rspamd/local.d/redis.conf Create file /etc/rspamd/local.d/worker-controller.inc 2 changes applied, the wizard is finished now *** Please reload the Rspamd configuration ***
Now, restart the rspamd service and check status
$ sudo systemctl restart rspamd
$ sudo systemctl status rspamd
● rspamd.service - rapid spam filtering system
Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-11-07 15:28:14 UTC; 4s ago
Docs: https://rspamd.com/doc/
Main PID: 21899 (rspamd)
Tasks: 5 (limit: 2336)
Memory: 116.5M
CGroup: /system.slice/rspamd.service
├─21899 rspamd: main process
├─21915 rspamd: rspamd_proxy process (localhost:11332)
├─21916 rspamd: controller process (localhost:11334)
├─21917 rspamd: normal process (localhost:11333)
└─21918 rspamd: hs_helper process
Nov 07 15:28:14 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.
Worker Proxy setup in /etc/rspamd/local.d/worker-proxy.inc
# local.d/worker-proxy.inc
milter = yes; # Enable milter mode
timeout = 120s; # Needed for Milter usually
upstream "local" {
default = yes; # Self-scan upstreams are always default
self_scan = yes; # Enable self-scan
}
count = 4; # Spawn more processes in self-scan mode
max_retries = 5; # How many times master is queried in case of failure
discard_on_reject = false; # Discard message instead of rejection
quarantine_on_reject = false; # Tell MTA to quarantine rejected messages
spam_header = "X-Spam"; # Use the specific spam header
reject_message = "Spam message rejected"; # Use custom rejection message
Disable DKIM Signing on rspamd
To disable DKIM signing (i.e. you use OpenDKIM, or signing is done elsewhere- ie, that's our case) # local.d/dkim_signing.conf enabled = false;
and restart rspamd service after that.
Setting up the WebUI
For using the location, <webroot>/rspamd,
Add the following on /etc/mailman3/nginx.conf
location /rspamd/ {
proxy_pass http://localhost:11334/;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
Now, check nginx syntax, restart the nginx service and check the status.
After restart, you can visit <webroot>/rspamd and login with the previously created password.
Setting up rspamd Worker proxy config on postfix
Rspamd Worker proxy config
On /etc/postfix/main.cf
Don't forget to disable existing configs for the same below
##
## Spam filter and DKIM signatures via Rspamd
##
smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock
non_smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock
milter_protocol = 6
milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen}
milter_default_action = accept
and then restart postfix.
Finally, enable rspamd service if everything seems to work properly
$ sudo systemctl enable rspamd.
Remove spam confirmation requests
Too many pending subscription requests can slow down[3] list index. Updating postorius is how to fix this. But in case there's no other way, and removing those requests through frontend [4] is impossible, this can be done (GNU Mailman 3.2.2 (La Villa Strangiato)):
$ sudo mailman shell -l listname.lists.fsci.org.in
Welcome to the GNU Mailman shell
The variable 'm' is the listname@lists.fsci.org.in mailing list
>>> from zope.component import getUtility
>>> from mailman.interfaces.pending import IPendings
>>> pendings = getUtility(IPendings)
>>> plist = []
>>> for p in pendings.find(m):
... plist.append(p)
...
>>> len(plist)
2047
>>> # Let us save the spammer email and when they signed up for analysis
>>> import csv
>>> with open('spam.csv', 'w', newline=) as csvfile:
... spamwriter = csv.writer(csvfile, delimiter=',')
... for p in plist:
... spammer = p[1]
... spamwriter.writerow([spammer.get('email'), spammer.get('when'), spammer.get('display_name')])
...
>>> # Now on to actually removing spam
>>> from mailman.interfaces.workflow import IWorkflowStateManager
>>> workflow = getUtility(IWorkflowStateManager)
>>> for p in plist:
... token = p[0]
... pendings.confirm(token)
... workflow.discard(token)
...
>>> # Now, we have to commit this transaction to database
>>> from mailman.config import config
>>> config.db.commit()