Note: Currently new registrations are closed, if you want an account Contact us
Difference between revisions of "Mailing list"
m (minor markup update) |
|||
(4 intermediate revisions by 2 users not shown) | |||
Line 18: | Line 18: | ||
|- | |- | ||
|* List manager||: Mailman3 | |* List manager||: Mailman3 | ||
|- | |||
|* Spam Filter||: Rspamd | |||
|- | |- | ||
|* Host name||: [https://lists.fsci.org.in lists.fsci.org.in] | |* Host name||: [https://lists.fsci.org.in lists.fsci.org.in] | ||
|} | |} | ||
= Coordination = | = Coordination = | ||
Line 28: | Line 31: | ||
== Admins == | == Admins == | ||
Abhijith PA | # Abhijith PA | ||
# Abhinav CK | |||
# Akhil Varkey | |||
====== emeritus ====== | |||
# Prinz Piuz | |||
# Balasankar C | |||
= Admin Documentation = | = Admin Documentation = | ||
== Postfix Aliases == | |||
Apart from mailman's aliases. We have some system wide aliases for sending and receiving postfix related issues and warning. Those are kept in ''/etc/aliases'' . | |||
Once done editing please run command `''newaliases''` | |||
== Generate DKIM for new domains == | == Generate DKIM for new domains == | ||
Line 42: | Line 57: | ||
* <selector_name>.txt - which contains the DKIM TXT DNS record for that domain. | * <selector_name>.txt - which contains the DKIM TXT DNS record for that domain. | ||
<br /> | <br /> | ||
Now, create a folder under /etc/opendkim/keys/<domain_name_for_this_list>/ and move the above file to that location. | Now, create a folder under <code>/etc/opendkim/keys/<domain_name_for_this_list>/</code> and move the above file to that location. | ||
<br /> | <br /> | ||
Create an entry for the newly created domain in /etc/opendkim/KeyTable as well as /etc/opendkim/SigningTable | Create an entry for the newly created domain in /etc/opendkim/KeyTable as well as /etc/opendkim/SigningTable | ||
< | <br /> | ||
On /etc/opendkim/KeyTable | On /etc/opendkim/KeyTable | ||
<selector_name._domainkey.<domain_name_of_this_list> <domain_name_of_this_list>:<selector_name>:/etc/opendkim/keys/<domain_name_of_this_list>/<selector_name>.private | <selector_name._domainkey.<domain_name_of_this_list> <domain_name_of_this_list>:<selector_name>:/etc/opendkim/keys/<domain_name_of_this_list>/<selector_name>.private | ||
Line 70: | Line 85: | ||
/usr/sbin/sendmail -t < dkim-test.txt | /usr/sbin/sendmail -t < dkim-test.txt | ||
== Setup Rspamd == | |||
See [[https://rspamd.com/doc/quickstart.html Rspamd quick start]] for the latest installation and setup instructions. | |||
Install Rspamd. Redis will be installed automatically(check). | |||
$ sudo apt install rspamd | |||
Check service is running. | |||
$ systemctl status rspamd.service | |||
● rspamd.service - rapid spam filtering system | |||
Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled) | |||
Active: active (running) since Sat 2020-11-07 14:54:03 UTC; 1min 58s ago | |||
Docs: https://rspamd.com/doc/ | |||
Main PID: 19339 (rspamd) | |||
Tasks: 5 (limit: 2336) | |||
Memory: 276.2M | |||
CGroup: /system.slice/rspamd.service | |||
├─19339 rspamd: main process | |||
├─19517 rspamd: rspamd_proxy process (localhost:11332) | |||
├─19518 rspamd: controller process (localhost:11334) | |||
├─19519 rspamd: normal process (localhost:11333) | |||
└─19520 rspamd: hs_helper process | |||
Nov 07 14:54:03 lists.fsci.org.in systemd[1]: Started rapid spam filtering system. | |||
Check if redis is working properly using redis-cli | |||
$ redis-cli | |||
127.0.0.1:6379> ping | |||
PONG | |||
127.0.0.1:6379> ping Hello[qwe] | |||
"Hello[qwe]" | |||
127.0.0.1:6379> exit | |||
Change some redis config to use it as a LRU cache[https://redis.io/topics/lru-cache]. | |||
On /etc/redis/redis.conf | |||
~line 850 | |||
maxmemory 500mb | |||
maxmemory-policy volatile-ttl | |||
bind 127.0.0.1 ::1 //already exists | |||
'''Configure rspamd''' | |||
$ sudo rspamadm configwizard | |||
____ _ | |||
| _ \ ___ _ __ __ _ _ __ ___ __| | | |||
| |_) |/ __|| '_ \ / _` || '_ ` _ \ / _` | | |||
| _ < \__ \| |_) || (_| || | | | | || (_| | | |||
|_| \_\|___/| .__/ \__,_||_| |_| |_| \__,_| | |||
|_| | |||
Welcome to the configuration tool | |||
We use /etc/rspamd/rspamd.conf configuration file, writing results to /etc/rspamd | |||
Modules enabled: hfilter, phishing, emails, asn, settings, chartable, arc, bayes_expiry, once_received, rbl, fuzzy_check, metadata_exporter, elastic, mid, multimap, spf, dkim_signing, dkim, mime_types, regexp, maillist, dmarc, forged_recipients, milter_headers, whitelist, force_actions, trie | |||
Modules disabled (explicitly): p0f, spamtrap, rspamd_update, mx_check, dcc | |||
Modules disabled (unconfigured): spamassassin, maps_stats, metric_exporter, dynamic_conf, clustering, reputation, antivirus, fuzzy_collect, external_services, ip_score, clickhouse | |||
Modules disabled (no Redis): greylist, url_redirector, replies, neural, ratelimit, history_redis | |||
Modules disabled (experimental): | |||
Modules disabled (failed): | |||
Do you wish to continue?[Y/n]: | |||
Setup WebUI and controller worker: | |||
Controller password is not set, do you want to set one?[Y/n]: Y | |||
Enter passphrase: | |||
Set encrypted password to: <password_hash> | |||
Redis servers are not set: | |||
The following modules will be enabled if you add Redis servers: | |||
* greylist | |||
* url_redirector | |||
* replies | |||
* neural | |||
* ratelimit | |||
* history_redis | |||
Do you wish to set Redis servers?[Y/n]: | |||
Input read only servers separated by `,` [default: localhost]: | |||
Input write only servers separated by `,` [default: localhost]: | |||
Do you have any password set for your Redis?[y/N]: | |||
Do you have any specific database for your Redis?[y/N]: | |||
Do you want to setup dkim signing feature?[y/N]: N | |||
File: /etc/rspamd/local.d/redis.conf, changes list: | |||
write_servers => localhost | |||
read_servers => localhost | |||
File: /etc/rspamd/local.d/worker-controller.inc, changes list: | |||
password => <password_hash> | |||
Apply changes?[Y/n]: Y | |||
Create file /etc/rspamd/local.d/redis.conf | |||
Create file /etc/rspamd/local.d/worker-controller.inc | |||
2 changes applied, the wizard is finished now | |||
*** Please reload the Rspamd configuration *** | |||
Now, restart the rspamd service and check status | |||
$ sudo systemctl restart rspamd | |||
$ sudo systemctl status rspamd | |||
● rspamd.service - rapid spam filtering system | |||
Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled) | |||
Active: active (running) since Sat 2020-11-07 15:28:14 UTC; 4s ago | |||
Docs: https://rspamd.com/doc/ | |||
Main PID: 21899 (rspamd) | |||
Tasks: 5 (limit: 2336) | |||
Memory: 116.5M | |||
CGroup: /system.slice/rspamd.service | |||
├─21899 rspamd: main process | |||
├─21915 rspamd: rspamd_proxy process (localhost:11332) | |||
├─21916 rspamd: controller process (localhost:11334) | |||
├─21917 rspamd: normal process (localhost:11333) | |||
└─21918 rspamd: hs_helper process | |||
Nov 07 15:28:14 lists.fsci.org.in systemd[1]: Started rapid spam filtering system. | |||
'''Worker Proxy setup''' | |||
in /etc/rspamd/local.d/worker-proxy.inc | |||
# local.d/worker-proxy.inc | |||
milter = yes; # Enable milter mode | |||
timeout = 120s; # Needed for Milter usually | |||
upstream "local" { | |||
default = yes; # Self-scan upstreams are always default | |||
self_scan = yes; # Enable self-scan | |||
} | |||
count = 4; # Spawn more processes in self-scan mode | |||
max_retries = 5; # How many times master is queried in case of failure | |||
discard_on_reject = false; # Discard message instead of rejection | |||
quarantine_on_reject = false; # Tell MTA to quarantine rejected messages | |||
spam_header = "X-Spam"; # Use the specific spam header | |||
reject_message = "Spam message rejected"; # Use custom rejection message | |||
'''Disable DKIM Signing on rspamd''' | |||
To disable DKIM signing (i.e. you use OpenDKIM, or signing is done elsewhere- ie, that's our case) | |||
# local.d/dkim_signing.conf | |||
enabled = false; | |||
and restart rspamd service after that. | |||
'''Setting up the WebUI''' | |||
<br />For using the location, <webroot>/rspamd, | |||
<br />Add the following on /etc/mailman3/nginx.conf | |||
location /rspamd/ { | |||
proxy_pass http://localhost:11334/; | |||
proxy_set_header Host $host; | |||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | |||
} | |||
Now, check nginx syntax, restart the nginx service and check the status. | |||
<br/> After restart, you can visit <webroot>/rspamd and login with the previously created password. | |||
<br /> | |||
'''Setting up rspamd Worker proxy config on postfix''' | |||
<br />Rspamd Worker proxy config | |||
<br />On /etc/postfix/main.cf | |||
Don't forget to disable existing configs for the same below | |||
## | |||
## Spam filter and DKIM signatures via Rspamd | |||
## | |||
smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock | |||
non_smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock | |||
milter_protocol = 6 | |||
milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen} | |||
milter_default_action = accept | |||
and then restart postfix. | |||
Finally, enable rspamd service if everything seems to work properly | |||
<code>$ sudo systemctl enable rspamd</code>. | |||
== Remove spam confirmation requests == | == Remove spam confirmation requests == |
Latest revision as of 21:54, 12 April 2024
Mailing lists are very useful communication and coordination tool. FSCI provides mailing lists to Free Libre Open Source (FLOSS) groups and non profit organizations. If you want to open a new public list, please drop a mail at postmaster(at)lists.fsci.org.in.
Environment
Machine Summary
We are on scaleway virtual cloud server.
* Cores | : 2 x86 cores |
* Memory | : 2GB |
* Disk | : 50GB |
* OS | : Debian GNU/Linux |
* Web Server | : Nginx |
* List manager | : Mailman3 |
* Spam Filter | : Rspamd |
* Host name | : lists.fsci.org.in |
Coordination
- Hangout with us in our Matrix room #mailman:poddery.com
- issue tracker - we use this to track progress of tasks
Admins
- Abhijith PA
- Abhinav CK
- Akhil Varkey
emeritus
- Prinz Piuz
- Balasankar C
Admin Documentation
Postfix Aliases
Apart from mailman's aliases. We have some system wide aliases for sending and receiving postfix related issues and warning. Those are kept in /etc/aliases .
Once done editing please run command `newaliases`
Generate DKIM for new domains
We use 'opendkim'[1] to implement DKIM.
To generate DKIM keys, use opendkim-genkey -b 4096 -h rsa-sha256 -r -s <selector_name> -d <domain_name_of_this_list> -v`
We don't have any format on selector naming. Pick something related to this list(for eg: the selector for the domain lists.fsci.org.in is lists
what we used).
After generating the DKIM keys, we will get two files.
- <selector_name>.private - which contains the private signing key.
- <selector_name>.txt - which contains the DKIM TXT DNS record for that domain.
Now, create a folder under /etc/opendkim/keys/<domain_name_for_this_list>/
and move the above file to that location.
Create an entry for the newly created domain in /etc/opendkim/KeyTable as well as /etc/opendkim/SigningTable
On /etc/opendkim/KeyTable
<selector_name._domainkey.<domain_name_of_this_list> <domain_name_of_this_list>:<selector_name>:/etc/opendkim/keys/<domain_name_of_this_list>/<selector_name>.private eg: lists._domainkey.lists.fsci.org.in lists.fsci.org.in:lists:/etc/opendkim/lists.private
On /etc/opendkim/SigningTable
<domain_name_of_this_list> <selector_name>._domainkey.<domain_name_of_this_list> eg: lists.fsci.org.in lists._domainkey.lists.fsci.org.in
More on DKIM keys, formating etc: [DomainKeys Identified Mail], Setup DKIM with OpenDKIM on Debian: [How To Install and Configure DKIM with Postfix on Debian]
Verify DKIM and other settings for new domains
Create a text file, dkim-test.txt (replace To address with random address generated by https://dkimvalidator.com or https://www.mail-tester.com/)
From: "Test" <test-dkim@mm.gnu.org.in> To: <random-address>@dkimvalidator.com Subject: Testing DKIM and other settings for this domain Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit This email is to test email settings for a domain is correct
and use sendmail command to send it
/usr/sbin/sendmail -t < dkim-test.txt
Setup Rspamd
See [Rspamd quick start] for the latest installation and setup instructions. Install Rspamd. Redis will be installed automatically(check).
$ sudo apt install rspamd
Check service is running.
$ systemctl status rspamd.service ● rspamd.service - rapid spam filtering system Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2020-11-07 14:54:03 UTC; 1min 58s ago Docs: https://rspamd.com/doc/ Main PID: 19339 (rspamd) Tasks: 5 (limit: 2336) Memory: 276.2M CGroup: /system.slice/rspamd.service ├─19339 rspamd: main process ├─19517 rspamd: rspamd_proxy process (localhost:11332) ├─19518 rspamd: controller process (localhost:11334) ├─19519 rspamd: normal process (localhost:11333) └─19520 rspamd: hs_helper process Nov 07 14:54:03 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.
Check if redis is working properly using redis-cli
$ redis-cli 127.0.0.1:6379> ping PONG 127.0.0.1:6379> ping Hello[qwe] "Hello[qwe]" 127.0.0.1:6379> exit
Change some redis config to use it as a LRU cache[2]. On /etc/redis/redis.conf ~line 850
maxmemory 500mb maxmemory-policy volatile-ttl bind 127.0.0.1 ::1 //already exists
Configure rspamd
$ sudo rspamadm configwizard ____ _ | _ \ ___ _ __ __ _ _ __ ___ __| | | |_) |/ __|| '_ \ / _` || '_ ` _ \ / _` | | _ < \__ \| |_) || (_| || | | | | || (_| | |_| \_\|___/| .__/ \__,_||_| |_| |_| \__,_| |_| Welcome to the configuration tool We use /etc/rspamd/rspamd.conf configuration file, writing results to /etc/rspamd Modules enabled: hfilter, phishing, emails, asn, settings, chartable, arc, bayes_expiry, once_received, rbl, fuzzy_check, metadata_exporter, elastic, mid, multimap, spf, dkim_signing, dkim, mime_types, regexp, maillist, dmarc, forged_recipients, milter_headers, whitelist, force_actions, trie Modules disabled (explicitly): p0f, spamtrap, rspamd_update, mx_check, dcc Modules disabled (unconfigured): spamassassin, maps_stats, metric_exporter, dynamic_conf, clustering, reputation, antivirus, fuzzy_collect, external_services, ip_score, clickhouse Modules disabled (no Redis): greylist, url_redirector, replies, neural, ratelimit, history_redis Modules disabled (experimental): Modules disabled (failed): Do you wish to continue?[Y/n]: Setup WebUI and controller worker: Controller password is not set, do you want to set one?[Y/n]: Y Enter passphrase: Set encrypted password to: <password_hash> Redis servers are not set: The following modules will be enabled if you add Redis servers: * greylist * url_redirector * replies * neural * ratelimit * history_redis Do you wish to set Redis servers?[Y/n]: Input read only servers separated by `,` [default: localhost]: Input write only servers separated by `,` [default: localhost]: Do you have any password set for your Redis?[y/N]: Do you have any specific database for your Redis?[y/N]: Do you want to setup dkim signing feature?[y/N]: N File: /etc/rspamd/local.d/redis.conf, changes list: write_servers => localhost read_servers => localhost
File: /etc/rspamd/local.d/worker-controller.inc, changes list: password => <password_hash>
Apply changes?[Y/n]: Y Create file /etc/rspamd/local.d/redis.conf Create file /etc/rspamd/local.d/worker-controller.inc 2 changes applied, the wizard is finished now *** Please reload the Rspamd configuration ***
Now, restart the rspamd service and check status
$ sudo systemctl restart rspamd $ sudo systemctl status rspamd ● rspamd.service - rapid spam filtering system Loaded: loaded (/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2020-11-07 15:28:14 UTC; 4s ago Docs: https://rspamd.com/doc/ Main PID: 21899 (rspamd) Tasks: 5 (limit: 2336) Memory: 116.5M CGroup: /system.slice/rspamd.service ├─21899 rspamd: main process ├─21915 rspamd: rspamd_proxy process (localhost:11332) ├─21916 rspamd: controller process (localhost:11334) ├─21917 rspamd: normal process (localhost:11333) └─21918 rspamd: hs_helper process Nov 07 15:28:14 lists.fsci.org.in systemd[1]: Started rapid spam filtering system.
Worker Proxy setup in /etc/rspamd/local.d/worker-proxy.inc
# local.d/worker-proxy.inc milter = yes; # Enable milter mode timeout = 120s; # Needed for Milter usually upstream "local" { default = yes; # Self-scan upstreams are always default self_scan = yes; # Enable self-scan } count = 4; # Spawn more processes in self-scan mode max_retries = 5; # How many times master is queried in case of failure discard_on_reject = false; # Discard message instead of rejection quarantine_on_reject = false; # Tell MTA to quarantine rejected messages spam_header = "X-Spam"; # Use the specific spam header reject_message = "Spam message rejected"; # Use custom rejection message
Disable DKIM Signing on rspamd
To disable DKIM signing (i.e. you use OpenDKIM, or signing is done elsewhere- ie, that's our case) # local.d/dkim_signing.conf enabled = false;
and restart rspamd service after that.
Setting up the WebUI
For using the location, <webroot>/rspamd,
Add the following on /etc/mailman3/nginx.conf
location /rspamd/ { proxy_pass http://localhost:11334/; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }
Now, check nginx syntax, restart the nginx service and check the status.
After restart, you can visit <webroot>/rspamd and login with the previously created password.
Setting up rspamd Worker proxy config on postfix
Rspamd Worker proxy config
On /etc/postfix/main.cf
Don't forget to disable existing configs for the same below ## ## Spam filter and DKIM signatures via Rspamd ## smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock non_smtpd_milters = inet:localhost:11332, local:/opendkim/opendkim.sock milter_protocol = 6 milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen} milter_default_action = accept
and then restart postfix.
Finally, enable rspamd service if everything seems to work properly
$ sudo systemctl enable rspamd
.
Remove spam confirmation requests
Too many pending subscription requests can slow down[3] list index. Updating postorius is how to fix this. But in case there's no other way, and removing those requests through frontend [4] is impossible, this can be done (GNU Mailman 3.2.2 (La Villa Strangiato)):
$ sudo mailman shell -l listname.lists.fsci.org.in Welcome to the GNU Mailman shell The variable 'm' is the listname@lists.fsci.org.in mailing list >>> from zope.component import getUtility >>> from mailman.interfaces.pending import IPendings >>> pendings = getUtility(IPendings) >>> plist = [] >>> for p in pendings.find(m): ... plist.append(p) ... >>> len(plist) 2047 >>> # Let us save the spammer email and when they signed up for analysis >>> import csv >>> with open('spam.csv', 'w', newline=) as csvfile: ... spamwriter = csv.writer(csvfile, delimiter=',') ... for p in plist: ... spammer = p[1] ... spamwriter.writerow([spammer.get('email'), spammer.get('when'), spammer.get('display_name')]) ... >>> # Now on to actually removing spam >>> from mailman.interfaces.workflow import IWorkflowStateManager >>> workflow = getUtility(IWorkflowStateManager) >>> for p in plist: ... token = p[0] ... pendings.confirm(token) ... workflow.discard(token) ... >>> # Now, we have to commit this transaction to database >>> from mailman.config import config >>> config.db.commit()