FSCI Wiki - User contributions [en]

Mailing list

2021-03-30T09:06:51Z

Abbyck: /* Admin Documentation */ add rspamd setup docs

Mailing lists are very useful communication and coordination tool. FSCI provides mailing lists to Free Libre Open Source (FLOSS) groups and non profit organizations. If you want to open a new public list, please drop a mail at [mailto:postmaster@lists.fsci.org.in postmaster(at)lists.fsci.org.in].

= Environment =
== Machine Summary ==
We are on [https://www.scaleway.com/virtual-cloud-servers scaleway] virtual cloud server.

{|
| * Cores||: 2 x86 cores
|-
| * Memory||: 2GB
|-
|* Disk||: 50GB
|-
|* OS||: Debian GNU/Linux
|-
|* Web Server||: Nginx
|-
|* List manager||: Mailman3
|-
|* Spam Filter||: Rspamd
|-
|* Host name||: [https://lists.fsci.org.in lists.fsci.org.in]
|}

= Coordination =

* Hangout with us in our Matrix room [https://matrix.to/#/#mailman:poddery.com #mailman:poddery.com]
* [https://git.fosscommunity.in/community/lists.fsci.org.in/issues issue tracker] - we use this to track progress of tasks

== Admins ==

Abhijith PA, Prinz Piuz, Balasankar C

= Admin Documentation =

== Generate DKIM for new domains ==

We use 'opendkim'[http://opendkim.org/] to implement DKIM.
To generate DKIM keys, use <code>opendkim-genkey -b 4096 -h rsa-sha256 -r -s <selector_name> -d <domain_name_of_this_list> -v`</code>
 
We don't have any format on selector naming. Pick something related to this list(for eg: the selector for the domain lists.fsci.org.in is <code>lists</code> what we used).
After generating the DKIM keys, we will get two files.
* <selector_name>.private - which contains the private signing key.
* <selector_name>.txt - which contains the DKIM TXT DNS record for that domain.
 
Now, create a folder under <code>/etc/opendkim/keys/<domain_name_for_this_list>/</code> and move the above file to that location.
 
Create an entry for the newly created domain in /etc/opendkim/KeyTable as well as /etc/opendkim/SigningTable
 
On /etc/opendkim/KeyTable
<selector_name._domainkey.<domain_name_of_this_list> <domain_name_of_this_list>:<selector_name>:/etc/opendkim/keys/<domain_name_of_this_list>/<selector_name>.private
eg: lists._domainkey.lists.fsci.org.in lists.fsci.org.in:lists:/etc/opendkim/lists.private
On /etc/opendkim/SigningTable
<domain_name_of_this_list> <selector_name>._domainkey.<domain_name_of_this_list>
eg: lists.fsci.org.in lists._domainkey.lists.fsci.org.in
More on DKIM keys, formating etc: [[https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail DomainKeys Identified Mail]],
Setup DKIM with OpenDKIM on Debian: [[https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-dkim-with-postfix-on-debian-wheezy How To Install and Configure DKIM with Postfix on Debian]]

== Verify DKIM and other settings for new domains ==

Create a text file, dkim-test.txt (replace To address with random address generated by https://dkimvalidator.com or https://www.mail-tester.com/)

From: "Test" <test-dkim@mm.gnu.org.in>
To: <random-address>@dkimvalidator.com
Subject: Testing DKIM and other settings for this domain
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

This email is to test email settings for a domain is correct

and use sendmail command to send it

/usr/sbin/sendmail -t < dkim-test.txt

== Setup Rspamd ==
See [[https://rspamd.com/doc/quickstart.html Rspamd quick start]] for the latest installation and setup instructions.
Install Rspamd. Redis will be installed automatically(check).
$ sudo apt install rspamd
Check service is running.
$ systemctl status rspamd.service
● rspamd.service - rapid spam filtering system
Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-11-07 14:54:03 UTC; 1min 58s ago
Docs: https://rspamd.com/doc/
Main PID: 19339 (rspamd)
Tasks: 5 (limit: 2336)
Memory: 276.2M
CGroup: /system.slice/rspamd.service
├─19339 rspamd: main process
├─19517 rspamd: rspamd_proxy process (localhost:11332)
├─19518 rspamd: controller process (localhost:11334)
├─19519 rspamd: normal process (localhost:11333)
└─19520 rspamd: hs_helper process
Nov 07 14:54:03 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.

Check if redis is working properly using redis-cli
$ redis-cli
127.0.0.1:6379> ping
PONG
127.0.0.1:6379> ping Hello[qwe]
"Hello[qwe]"
127.0.0.1:6379> exit

Change some redis config to use it as a LRU cache[https://redis.io/topics/lru-cache].
On /etc/redis/redis.conf
~line 850
maxmemory 500mb
maxmemory-policy volatile-ttl
bind 127.0.0.1 ::1 //already exists

'''Configure rspamd'''
$ sudo rspamadm configwizard
____ _
| _ \ ___ _ __ __ _ _ __ ___ __| |
| |_) |/ __|| '_ \ / _` || '_ ` _ \ / _` |
| _ < \__ \| |_) || (_| || | | | | || (_| |
|_| \_\|___/| .__/ \__,_||_| |_| |_| \__,_|
|_|
Welcome to the configuration tool
We use /etc/rspamd/rspamd.conf configuration file, writing results to /etc/rspamd
Modules enabled: hfilter, phishing, emails, asn, settings, chartable, arc, bayes_expiry, once_received, rbl, fuzzy_check, metadata_exporter, elastic, mid, multimap, spf, dkim_signing, dkim, mime_types, regexp, maillist, dmarc, forged_recipients, milter_headers, whitelist, force_actions, trie
Modules disabled (explicitly): p0f, spamtrap, rspamd_update, mx_check, dcc
Modules disabled (unconfigured): spamassassin, maps_stats, metric_exporter, dynamic_conf, clustering, reputation, antivirus, fuzzy_collect, external_services, ip_score, clickhouse
Modules disabled (no Redis): greylist, url_redirector, replies, neural, ratelimit, history_redis
Modules disabled (experimental):
Modules disabled (failed):
Do you wish to continue?[Y/n]:
Setup WebUI and controller worker:
Controller password is not set, do you want to set one?[Y/n]: Y
Enter passphrase:
Set encrypted password to: <password_hash>
Redis servers are not set:
The following modules will be enabled if you add Redis servers:
* greylist
* url_redirector
* replies
* neural
* ratelimit
* history_redis
Do you wish to set Redis servers?[Y/n]:
Input read only servers separated by `,` [default: localhost]:
Input write only servers separated by `,` [default: localhost]:
Do you have any password set for your Redis?[y/N]:
Do you have any specific database for your Redis?[y/N]:
Do you want to setup dkim signing feature?[y/N]: N
File: /etc/rspamd/local.d/redis.conf, changes list:
write_servers => localhost
read_servers => localhost

File: /etc/rspamd/local.d/worker-controller.inc, changes list:
password => <password_hash>

Apply changes?[Y/n]: Y
Create file /etc/rspamd/local.d/redis.conf
Create file /etc/rspamd/local.d/worker-controller.inc
2 changes applied, the wizard is finished now
*** Please reload the Rspamd configuration ***

Now, restart the rspamd service and check status
$ sudo systemctl restart rspamd
$ sudo systemctl status rspamd
● rspamd.service - rapid spam filtering system
Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-11-07 15:28:14 UTC; 4s ago
Docs: https://rspamd.com/doc/
Main PID: 21899 (rspamd)
Tasks: 5 (limit: 2336)
Memory: 116.5M
CGroup: /system.slice/rspamd.service
├─21899 rspamd: main process
├─21915 rspamd: rspamd_proxy process (localhost:11332)
├─21916 rspamd: controller process (localhost:11334)
├─21917 rspamd: normal process (localhost:11333)
└─21918 rspamd: hs_helper process
Nov 07 15:28:14 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.

'''Worker Proxy setup'''
in /etc/rspamd/local.d/worker-proxy.inc
# local.d/worker-proxy.inc
milter = yes; # Enable milter mode
timeout = 120s; # Needed for Milter usually
upstream "local" {
default = yes; # Self-scan upstreams are always default
self_scan = yes; # Enable self-scan
}
count = 4; # Spawn more processes in self-scan mode
max_retries = 5; # How many times master is queried in case of failure
discard_on_reject = false; # Discard message instead of rejection
quarantine_on_reject = false; # Tell MTA to quarantine rejected messages
spam_header = "X-Spam"; # Use the specific spam header
reject_message = "Spam message rejected"; # Use custom rejection message

'''Disable DKIM Signing on rspamd'''
To disable DKIM signing (i.e. you use OpenDKIM, or signing is done elsewhere- ie, that's our case)
# local.d/dkim_signing.conf
enabled = false;

and restart rspamd service after that.

'''Setting up the WebUI'''
 For using the location, <webroot>/rspamd,
 Add the following on /etc/mailman3/nginx.conf

location /rspamd/ {
proxy_pass http://localhost:11334/;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Now, check nginx syntax, restart the nginx service and check the status.
 After restart, you can visit <webroot>/rspamd and login with the previously created password.
 

'''Setting up rspamd Worker proxy config on postfix'''
 Rspamd Worker proxy config
 On /etc/postfix/main.cf
Don't forget to disable existing configs for the same below
##
## Spam filter and DKIM signatures via Rspamd
##
smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock
non_smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock
milter_protocol = 6
milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen}
milter_default_action = accept

and then restart postfix.

Finally, enable rspamd service if everything seems to work properly
<code>$ sudo systemctl enable rspamd</code>.

== Remove spam confirmation requests ==

Too many pending subscription requests can slow down[https://gitlab.com/mailman/postorius/-/issues/417] list index. Updating postorius is how to fix this. But in case there's no other way, and removing those requests through frontend [https://gitlab.com/mailman/mailman/-/issues/779#note_423132958] is impossible, this can be done (GNU Mailman 3.2.2 (La Villa Strangiato)):

$ sudo mailman shell -l listname.lists.fsci.org.in
Welcome to the GNU Mailman shell
The variable 'm' is the listname@lists.fsci.org.in mailing list
>>> from zope.component import getUtility
>>> from mailman.interfaces.pending import IPendings
>>> pendings = getUtility(IPendings)
>>> plist = []
>>> for p in pendings.find(m):
... plist.append(p)
...
>>> len(plist)
2047
>>> # Let us save the spammer email and when they signed up for analysis
>>> import csv
>>> with open('spam.csv', 'w', newline='') as csvfile:
... spamwriter = csv.writer(csvfile, delimiter=',')
... for p in plist:
... spammer = p[1]
... spamwriter.writerow([spammer.get('email'), spammer.get('when'), spammer.get('display_name')])
...
>>> # Now on to actually removing spam
>>> from mailman.interfaces.workflow import IWorkflowStateManager
>>> workflow = getUtility(IWorkflowStateManager)
>>> for p in plist:
... token = p[0]
... pendings.confirm(token)
... workflow.discard(token)
...
>>> # Now, we have to commit this transaction to database
>>> from mailman.config import config
>>> config.db.commit()

----

[[Category: Services]]

Mailing list

2021-03-30T08:21:15Z

Abbyck: /* Machine Summary */ add spam filter

Mailing list

2021-03-30T08:18:51Z

Abbyck: /* Generate DKIM for new domains */

Mailing list

2021-03-30T08:17:22Z

Abbyck: /* Generate DKIM for new domains */

Mailing list

2021-03-30T08:16:39Z

Abbyck: Add DKIM setup