Mailing list

Revision as of 21:54, 12 April 2024 by Bhe (talk | contribs) (minor markup update)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Mailing lists are very useful communication and coordination tool. FSCI provides mailing lists to Free Libre Open Source (FLOSS) groups and non profit organizations. If you want to open a new public list, please drop a mail at postmaster(at)lists.fsci.org.in.

Environment

Machine Summary

We are on scaleway virtual cloud server.

* Cores : 2 x86 cores
* Memory : 2GB
* Disk : 50GB
* OS : Debian GNU/Linux
* Web Server : Nginx
* List manager : Mailman3
* Spam Filter : Rspamd
* Host name : lists.fsci.org.in

Coordination

Admins

  1. Abhijith PA
  2. Abhinav CK
  3. Akhil Varkey
emeritus
  1. Prinz Piuz
  2. Balasankar C

Admin Documentation

Postfix Aliases

Apart from mailman's aliases. We have some system wide aliases for sending and receiving postfix related issues and warning. Those are kept in /etc/aliases .

Once done editing please run command `newaliases`

Generate DKIM for new domains

We use 'opendkim'[1] to implement DKIM. To generate DKIM keys, use opendkim-genkey -b 4096 -h rsa-sha256 -r -s <selector_name> -d <domain_name_of_this_list> -v`
We don't have any format on selector naming. Pick something related to this list(for eg: the selector for the domain lists.fsci.org.in is lists what we used). After generating the DKIM keys, we will get two files.

  • <selector_name>.private - which contains the private signing key.
  • <selector_name>.txt - which contains the DKIM TXT DNS record for that domain.


Now, create a folder under /etc/opendkim/keys/<domain_name_for_this_list>/ and move the above file to that location.
Create an entry for the newly created domain in /etc/opendkim/KeyTable as well as /etc/opendkim/SigningTable
On /etc/opendkim/KeyTable

<selector_name._domainkey.<domain_name_of_this_list> <domain_name_of_this_list>:<selector_name>:/etc/opendkim/keys/<domain_name_of_this_list>/<selector_name>.private
eg: lists._domainkey.lists.fsci.org.in lists.fsci.org.in:lists:/etc/opendkim/lists.private

On /etc/opendkim/SigningTable

<domain_name_of_this_list> <selector_name>._domainkey.<domain_name_of_this_list>
eg: lists.fsci.org.in lists._domainkey.lists.fsci.org.in

More on DKIM keys, formating etc: [DomainKeys Identified Mail], Setup DKIM with OpenDKIM on Debian: [How To Install and Configure DKIM with Postfix on Debian]

Verify DKIM and other settings for new domains

Create a text file, dkim-test.txt (replace To address with random address generated by https://dkimvalidator.com or https://www.mail-tester.com/)

From: "Test" <test-dkim@mm.gnu.org.in>
To: <random-address>@dkimvalidator.com
Subject: Testing DKIM and other settings for this domain
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

This email is to test email settings for a domain is correct

and use sendmail command to send it

/usr/sbin/sendmail -t < dkim-test.txt

Setup Rspamd

See [Rspamd quick start] for the latest installation and setup instructions. Install Rspamd. Redis will be installed automatically(check).

$ sudo apt install rspamd

Check service is running.

$ systemctl status rspamd.service
  ● rspamd.service - rapid spam filtering system
  Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
  Active: active (running) since Sat 2020-11-07 14:54:03 UTC; 1min 58s ago
    Docs: https://rspamd.com/doc/
    Main PID: 19339 (rspamd)
    Tasks: 5 (limit: 2336)
    Memory: 276.2M
    CGroup: /system.slice/rspamd.service
          ├─19339 rspamd: main process
          ├─19517 rspamd: rspamd_proxy process (localhost:11332)
          ├─19518 rspamd: controller process (localhost:11334)
          ├─19519 rspamd: normal process (localhost:11333)
          └─19520 rspamd: hs_helper process
    Nov 07 14:54:03 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.

Check if redis is working properly using redis-cli

$ redis-cli
 127.0.0.1:6379> ping
  PONG
 127.0.0.1:6379> ping Hello[qwe]
  "Hello[qwe]"
 127.0.0.1:6379> exit

Change some redis config to use it as a LRU cache[2]. On /etc/redis/redis.conf ~line 850

maxmemory 500mb
maxmemory-policy volatile-ttl
bind 127.0.0.1 ::1 //already exists

Configure rspamd

$ sudo rspamadm configwizard
 ____                                     _
|  _ \  ___  _ __    __ _  _ __ ___    __| |
| |_) |/ __|| '_ \  / _` || '_ ` _ \  / _` |
|  _ < \__ \| |_) || (_| || | | | | || (_| |
|_| \_\|___/| .__/  \__,_||_| |_| |_| \__,_|
            |_|
Welcome to the configuration tool
We use /etc/rspamd/rspamd.conf configuration file, writing results to /etc/rspamd
Modules enabled: hfilter, phishing, emails, asn, settings, chartable, arc, bayes_expiry, once_received, rbl, fuzzy_check, metadata_exporter, elastic, mid, multimap, spf, dkim_signing, dkim, mime_types, regexp, maillist, dmarc,   forged_recipients, milter_headers, whitelist, force_actions, trie
Modules disabled (explicitly): p0f, spamtrap, rspamd_update, mx_check, dcc
Modules disabled (unconfigured): spamassassin, maps_stats, metric_exporter, dynamic_conf, clustering, reputation, antivirus, fuzzy_collect, external_services, ip_score, clickhouse
Modules disabled (no Redis): greylist, url_redirector, replies, neural, ratelimit, history_redis
Modules disabled (experimental): 
Modules disabled (failed): 
Do you wish to continue?[Y/n]: 
Setup WebUI and controller worker:
Controller password is not set, do you want to set one?[Y/n]: Y
Enter passphrase: 
Set encrypted password to: <password_hash>
Redis servers are not set:
The following modules will be enabled if you add Redis servers:
   * greylist
   * url_redirector
   * replies
   * neural
   * ratelimit
   * history_redis
Do you wish to set Redis servers?[Y/n]: 
Input read only servers separated by `,` [default: localhost]: 
Input write only servers separated by `,` [default: localhost]: 
Do you have any password set for your Redis?[y/N]: 
Do you have any specific database for your Redis?[y/N]: 
Do you want to setup dkim signing feature?[y/N]: N
File: /etc/rspamd/local.d/redis.conf, changes list:
write_servers => localhost
read_servers => localhost
File: /etc/rspamd/local.d/worker-controller.inc, changes list:
password => <password_hash>
Apply changes?[Y/n]: Y
Create file /etc/rspamd/local.d/redis.conf
Create file /etc/rspamd/local.d/worker-controller.inc
2 changes applied, the wizard is finished now
*** Please reload the Rspamd configuration ***

Now, restart the rspamd service and check status

$ sudo systemctl restart rspamd
$ sudo systemctl status rspamd
  ● rspamd.service - rapid spam filtering system
    Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
    Active: active (running) since Sat 2020-11-07 15:28:14 UTC; 4s ago
      Docs: https://rspamd.com/doc/
    Main PID: 21899 (rspamd)
    Tasks: 5 (limit: 2336)
    Memory: 116.5M
    CGroup: /system.slice/rspamd.service
          ├─21899 rspamd: main process
          ├─21915 rspamd: rspamd_proxy process (localhost:11332)
          ├─21916 rspamd: controller process (localhost:11334)
          ├─21917 rspamd: normal process (localhost:11333)
          └─21918 rspamd: hs_helper process
    Nov 07 15:28:14 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.

Worker Proxy setup in /etc/rspamd/local.d/worker-proxy.inc

# local.d/worker-proxy.inc
milter = yes; # Enable milter mode
timeout = 120s; # Needed for Milter usually
upstream "local" {
  default = yes; # Self-scan upstreams are always default
  self_scan = yes; # Enable self-scan
}
count = 4; # Spawn more processes in self-scan mode
max_retries = 5; # How many times master is queried in case of failure
discard_on_reject = false; # Discard message instead of rejection
quarantine_on_reject = false; # Tell MTA to quarantine rejected messages
spam_header = "X-Spam"; # Use the specific spam header
reject_message = "Spam message rejected"; # Use custom rejection message

Disable DKIM Signing on rspamd

To disable DKIM signing (i.e. you use OpenDKIM, or signing is done elsewhere- ie, that's our case)
# local.d/dkim_signing.conf
enabled = false;

and restart rspamd service after that.

Setting up the WebUI
For using the location, <webroot>/rspamd,
Add the following on /etc/mailman3/nginx.conf

location /rspamd/ {
    proxy_pass       http://localhost:11334/;
    proxy_set_header Host      $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Now, check nginx syntax, restart the nginx service and check the status.
After restart, you can visit <webroot>/rspamd and login with the previously created password.

Setting up rspamd Worker proxy config on postfix
Rspamd Worker proxy config
On /etc/postfix/main.cf

Don't forget to disable existing  configs for the same below
##
## Spam filter and DKIM signatures via Rspamd
##
smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock
non_smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock
milter_protocol = 6
milter_mail_macros =  i {mail_addr} {client_addr} {client_name} {auth_authen}
milter_default_action = accept

and then restart postfix.

Finally, enable rspamd service if everything seems to work properly $ sudo systemctl enable rspamd.

Remove spam confirmation requests

Too many pending subscription requests can slow down[3] list index. Updating postorius is how to fix this. But in case there's no other way, and removing those requests through frontend [4] is impossible, this can be done (GNU Mailman 3.2.2 (La Villa Strangiato)):

$ sudo mailman shell -l listname.lists.fsci.org.in
Welcome to the GNU Mailman shell
The variable 'm' is the listname@lists.fsci.org.in mailing list
>>> from zope.component import getUtility
>>> from mailman.interfaces.pending import IPendings
>>> pendings = getUtility(IPendings)
>>> plist = []
>>> for p in pendings.find(m):
...   plist.append(p)
... 
>>> len(plist)
2047
>>> # Let us save the spammer email and when they signed up for analysis
>>> import csv
>>> with open('spam.csv', 'w', newline=) as csvfile:
...     spamwriter = csv.writer(csvfile, delimiter=',')
...     for p in plist:
...         spammer = p[1]
...         spamwriter.writerow([spammer.get('email'), spammer.get('when'), spammer.get('display_name')])
...
>>> # Now on to actually removing spam
>>> from mailman.interfaces.workflow import IWorkflowStateManager
>>> workflow = getUtility(IWorkflowStateManager)
>>> for p in plist:
...     token = p[0]
...     pendings.confirm(token)
...     workflow.discard(token)
...
>>> # Now, we have to commit this transaction to database
>>> from mailman.config import config
>>> config.db.commit()