Feature #16218

Migrate some of our Schleuder lists to lizard

Added by intrigeri 2018-12-11 09:10:53 . Updated 2019-01-10 07:50:15 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Tails_3.12
Start date:
2018-12-27
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

And then:

Subtasks

Bug #16253: Schleuder keys update cronjob fails for our lists on lizard Resolved

100
Bug #16263: /etc/cron.daily/spamassassin fails on mail.lizard Resolved

0
Bug #16270: mail.lizard lacks memory Resolved

100

Related issues

Related to Tails - Bug #16267: WhisperBack reports are not delivered Resolved 2019-01-04
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 2017-06-30

History

#1 Updated by intrigeri 2018-12-11 09:10:59

  • blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#3 Updated by intrigeri 2018-12-11 09:49:17

Starting point: at the moment we have no SMTPd listening to incoming email from the outside of lizard and no SMTP-related firewall rules there. For some reason the MX for tails.b.o points to lizard, but lizard’s Postfix is not aware of it and would not accept such email even if it was listening on the Internet.

To keep things simple, I’ll create a VM that will run Postfix + Schleuder; I’ll redirect lizard’s inbound SMTP ports to it; for now, as per Bug #16121#note-6, it’ll accept email for boum.org, relay to Schleuder messages sent to the lists we host on this VM, and “have a transport map that delivers everything else to wherever boum.org’s MX points”. And if needed, at some point this VM could also handle some of the incoming email sent to tails.b.o or to tails.TLD.

To manage Schleuder I’ll use the https://0xacab.org/schleuder/puppet-schleuder module. It depends on https://github.com/voxpupuli/puppet-healthcheck, which pretends it only supports Puppet 4.10+, but I could find no justification for this versioned dependency bump so I’ll try and see what happens.

#4 Updated by intrigeri 2018-12-11 14:42:17

  • blocked by Feature #15511: Switch to another Puppet module to manage Postfix added

#5 Updated by intrigeri 2018-12-11 14:42:50

I’d rather not write more Puppet code that I’ll have to migrate to a different postfix module a week later, so I’ll do Feature #15511 first.

#6 Updated by groente 2018-12-18 16:59:25

FYI, here are some snippets that might be useful:

in main.cf:

schleuder_destination_recipient_limit = 1
transport_maps = hash:/etc/postfix/transport.map
virtual_mailbox_domains = boum.org
virtual_mailbox_maps = regexp:/etc/postfix/vmm.regex

in master.cf

    schleuder  unix  -       n       n       -       -       pipe
            flags=DRhu user=schleuder argv=/usr/bin/schleuder work ${recipient}

in vmm.regex:

/^(.*)@boum.org$/ $1@boum.org

in transport.map:

list@boum.org                   schleuder:
list-request@boum.org           schleuder:
list-bounce@boum.org            schleuder:
list-sendkey@boum.org           schleuder:
list-owner@boum.org             schleuder:
boum.org                        smtp:boum.org:25

#7 Updated by intrigeri 2018-12-18 18:05:57

  • Priority changed from Normal to High

This needs to happen by Thursday night.

#8 Updated by intrigeri 2018-12-19 15:24:25

test!

#9 Updated by intrigeri 2018-12-19 16:27:55

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

On the Postfix front, the basics are in place:

  • we accept email sent from the outside of lizard to boum.org and forward them to boum.org's MX * email sent from mail.lizard to boum.org is also forwarded to boum.org’s MX

Next steps:

  • set up Schleuder
  • create a test list and ensure everything works fine
  • create lists and set up Postfix transport for them
  • import lists config, keyring & subscriber list
  • set admin email correctly in Hiera and add their pubkey to Git if needed
  • ask root@b.o to redirect email to lists we’ll host here
  • add some basic spam protection (smtpd_*_restrictions, possibly greylisting or anything cheap to set up and maintain that our local email expert will advise)

#10 Updated by intrigeri 2018-12-19 20:22:18

  • Assignee changed from intrigeri to groente
  • % Done changed from 10 to 50
  • QA Check set to Info Needed

intrigeri wrote:
> * add some basic spam protection (smtpd_*_restrictions, possibly greylisting or anything cheap to set up and maintain that our local email expert will advise)

Now that the part of the migration that was in the critical path is done, this can wait a bit more, unless we realize we suddenly receive way more spam. The last remaining bit of Feature #16217 has much higher prio as it’s still in the critical path.

I propose this to start with:

smtpd_helo_required = yes

smtpd_client_restrictions =
        permit_mynetworks,
        permit_sasl_authenticated

smtpd_data_restrictions = 
        permit_mynetworks,
        permit_sasl_authenticated,
        reject_unauth_pipelining

smtpd_helo_restrictions =
        permit_mynetworks,
        permit_sasl_authenticated,
        reject_invalid_helo_hostname,
        reject_non_fqdn_helo_hostname,
        reject_unknown_helo_hostname

smtpd_recipient_and_relay_restrictions =
        permit_mynetworks,
        permit_sasl_authenticated,
        reject_non_fqdn_hostname,
        reject_non_fqdn_sender,
        reject_non_fqdn_recipient,
        reject_unauth_destination,
        reject_unknown_sender_domain,
        reject_unknown_recipient_domain,
        reject_invalid_hostname

smtpd_recipient_restrictions = $smtpd_recipient_and_relay_restrictions
smtpd_relay_restrictions = $smtpd_recipient_and_relay_restrictions

#11 Updated by groente 2018-12-19 20:49:38

  • Assignee changed from groente to intrigeri

i’d propose to add the following:

smtpd_delay_reject = yes
disable_vrfy_command = yes
show_user_unknown_table_name = no
smtpd_etrn_restrictions = reject

and set up amavisd/spamassassin.

#12 Updated by intrigeri 2018-12-20 10:28:07

  • Assignee changed from intrigeri to groente

> i’d propose to add the following:

Applied the settings I’ve proposed + yours (except smtpd_delay_reject since it’s enabled by default). Please review :)

> and set up amavisd/spamassassin.

I agree we should run SA. Two questions wrt. integration:

  • If possible, I’d rather not bother with amavisd. I know there are a few other, somewhat simpler, options to integrate SA with Postfix and I’m tempted to give them a try. Now, if you tell me there are all bad and amavisd is the way to go, I’ll go that way.
  • I’m told SA’s default config tends to behave poorly wrt. encrypted email; any tips you can share to make it work better in this respect?

#13 Updated by groente 2018-12-20 11:11:29

intrigeri wrote:
> > i’d propose to add the following:
>
> Applied the settings I’ve proposed + yours (except smtpd_delay_reject since it’s enabled by default). Please review :)

Ok, those settings look good.
Now, rather than focussing on SA, i think we should get the SSL situation fixed asap.

> > and set up amavisd/spamassassin.
>
> I agree we should run SA. Two questions wrt. integration:
>
> * If possible, I’d rather not bother with amavisd. I know there are a few other, somewhat simpler, options to integrate SA with Postfix and I’m tempted to give them a try. Now, if you tell me there are all bad and amavisd is the way to go, I’ll go that way.

I do think amavisd is the way to go, it does a lot more than just SA, it’s a run of the mill setup with lots of documentation and community testing, and it will make it easier to add a lot of other cheap checks as well as move towards DMARC.

> * I’m told SA’s default config tends to behave poorly wrt. encrypted email; any tips you can share to make it work better in this respect?

Ah, I never had problems with this.

#14 Updated by groente 2018-12-20 11:11:52

  • Assignee changed from groente to intrigeri
  • QA Check changed from Info Needed to Dev Needed

#15 Updated by intrigeri 2018-12-20 11:29:30

> Now, rather than focussing on SA, i think we should get the SSL situation fixed asap.

Right! Forgot about that. I’ll need to learn how deal with Let’s Encrypt on a system that has no web server.

> I do think amavisd is the way to go, […]

Deal.

#16 Updated by intrigeri 2018-12-20 13:40:18

  • Assignee changed from intrigeri to groente
  • QA Check changed from Dev Needed to Ready for QA

>> Now, rather than focussing on SA, i think we should get the SSL situation fixed asap.

> Right! Forgot about that. I’ll need to learn how deal with Let’s Encrypt on a system that has no web server.

Done => please review git diff --submodule=diff 0f8389934c85eeacfb409e1ec7c57f0c9cd50e2d..a11971836735e748e33bee8e5fbc81374cab6170 in our manifests repo.

Next step here: amavisd + SA. But Bug #16232 is more urgent so not sure I’ll do this today.

#17 Updated by intrigeri 2018-12-22 10:52:04

I’ve set up amavis + SA: git diff --submodule=diff 66f10dada56cd2f8ae55d898cb55a9d1fd825f63..5a804612876d2c88c43d614ee1406e1bb8a0204e in our manifests repo. So I think we’re done here. Next step: review this + the commits I’ve pointed you above :)

#18 Updated by intrigeri 2018-12-22 13:50:19

  • Description updated

(TODO: monitoring)

#19 Updated by groente 2018-12-24 14:43:01

  • Assignee changed from groente to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

configs look fine for now, i’ll leave the monitoring to you.
please reasssign to me (or open a new ticket) if there’s trouble with spam, then we can do some tuning and extra checks.

#20 Updated by CyrilBrulebois 2018-12-28 04:39:28

Heya. After a few days I’ve finally received a DSN for the mail to -rm (sent 2018-12-23) that couldn’t be delivered from Riseup, due to a connection timeout on an onion address. I can share the details if needed (never entirely sure which part of the infra could be somewhat private, thus not doing so directly here).

#21 Updated by intrigeri 2018-12-28 09:39:06

> Heya. After a few days I’ve finally received a DSN for the mail to -rm (sent 2018-12-23) that couldn’t be delivered from Riseup, due to a connection timeout on an onion address.

Indeed, confirmed, I’ve received such DSNs for test email sent from Riseup to one of our lists too. This affects all email b.o, i.e. <del><a class='issue tracker-1 status-3 priority-4 priority-default closed child' href='/code/issues/16121' title='Migrate our Schleuder lists outside of boum.org'>Bug #16121</a></del> and more (e.g. Mailman lists): Riseup is still trying to deliver b.o email to a deprecated onion service that’s 1. down; 2. not on the current MX. I’ll notify them today.

#22 Updated by intrigeri 2018-12-29 10:41:09

  • Description updated

#23 Updated by intrigeri 2018-12-29 11:07:19

  • Description updated

#24 Updated by intrigeri 2018-12-29 14:47:27

  • Assignee changed from intrigeri to groente
  • QA Check changed from Dev Needed to Ready for QA

Monitoring added, see 70928f1ac3ff516c3ca9a3c3ab4a008fa55629df in puppet-tails.

#25 Updated by CyrilBrulebois 2018-12-29 17:32:33

Riseup → Tails mail looks OK now (at least for the mail I resent to tails-rm).

#26 Updated by intrigeri 2018-12-30 09:34:30

  • blocks deleted (Feature #15511: Switch to another Puppet module to manage Postfix)

#27 Updated by groente 2019-01-03 11:35:04

  • Status changed from In Progress to Resolved
  • Assignee deleted (groente)
  • QA Check deleted (Ready for QA)

#28 Updated by intrigeri 2019-01-04 13:08:55

  • related to Bug #16267: WhisperBack reports are not delivered added