we add various information while analysing an incoming message. like
dkim/spf/ip reputation. and content-based junk filter threshold/result and
ham/spam words used.
for issue #179 by Fell and #157 by mattfbacon
the members must currently all be addresses of local accounts.
a message sent to an alias is accepted if at least one of the members accepts
it. if no members accepts it (e.g. due to bad reputation of sender), the
message is rejected.
if a message is submitted to both an alias addresses and to recipients that are
members of the alias in an smtp transaction, the message will be delivered to
such members only once. the same applies if the address in the message
from-header is the address of a member: that member won't receive the message
(they sent it). this prevents duplicate messages.
aliases have three configuration options:
- PostPublic: whether anyone can send through the alias, or only members.
members-only lists can be useful inside organizations for internal
communication. public lists can be useful for support addresses.
- ListMembers: whether members can see the addresses of other members. this can
be seen in the account web interface. in the future, we could export this in
other ways, so clients can expand the list.
- AllowMsgFrom: whether messages can be sent through the alias with the alias
address used in the message from-header. the webmail knows it can use that
address, and will use it as from-address when replying to a message sent to
that address.
ideas for the future:
- allow external addresses as members. still with some restrictions, such as
requiring a valid dkim-signature so delivery has a chance to succeed. will
also need configuration of an admin that can receive any bounces.
- allow specifying specific members who can sent through the list (instead of
all members).
for github issue #57 by hmfaysal.
also relevant for #99 by naturalethic.
thanks to damir & marin from sartura for discussing requirements/features.
both when parsing our configs, and for incoming on smtp or in messages.
so we properly compare things like é and e+accent as equal, and accept the
different encodings of that same address.
all ui frontend code is now in typescript. we no longer need jshint, and we
build the frontend code during "make build".
this also changes tlsrpt types for a Report, not encoding field names with
dashes, but to keep them valid identifiers in javascript. this makes it more
conveniently to work with in the frontend, and works around a sherpats
limitation.
we don't want external software to include internal details like mlog.
slog.Logger is/will be the standard.
we still have mlog for its helper functions, and its handler that logs in
concise logfmt used by mox.
packages that are not meant for reuse still pass around mlog.Log for
convenience.
we use golang.org/x/exp/slog because we also support the previous Go toolchain
version. with the next Go release, we'll switch to the builtin slog.
both cases are quite typical for spammers, and not for legitimate senders.
this doesn't apply to known senders. and it only requires that the content look
more like ham instead of spam. so legitimate mail can still get through with
these properties.
- dmarc reports: add a cid to the log line about one run of sending reports, and log line for each report
- in smtpclient, also handle tls errors from the first read after a handshake. we appear to sometimes get tls alerts about bad certificates on the first read.
- for messages to dmarc/tls reporting addresses that we think should/can not be processed as reports, add an X-Mox- header explaining the reason.
- tls reports: send report messages with From address of postmaster at an actually configured domain for the mail host. and only send reports when dkim signing is configured for that domain. the domain is also the submitter domain. the rfc seems to require dkim-signing with an exact match with the message from and submitter.
- for incoming tls reports, in the smtp server, we do allow a dkim-signature domain that is higher-level (up to publicsuffix) of the message from domain. so we are stricter in what we send than what we receive.
we were already accepting, processing and displaying incoming tls reports. now
we start tracking TLS connection and security-policy-related errors for
outgoing message deliveries as well. we send reports once a day, to the
reporting addresses specified in TLSRPT records (rua) of a policy domain. these
reports are about MTA-STS policies and/or DANE policies, and about
STARTTLS-related failures.
sending reports is enabled by default, but can be disabled through setting
NoOutgoingTLSReports in mox.conf.
only at the end of the implementation process came the realization that the
TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy
for the recipient domain, and that MTA-STS and DANE TLS/policy results are
typically delivered in separate reports. so MX hosts need their own TLSRPT
policies.
config for the per-host TLSRPT policy should be added to mox.conf for existing
installs, in field HostTLSRPT. it is automatically configured by quickstart for
new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin
pages now suggest the per-host TLSRPT record. by creating that record, you're
requesting TLS reports about your MX host.
gathering all the TLS/policy results is somewhat tricky. the tentacles go
throughout the code. the positive result is that the TLS/policy-related code
had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect
reality better, with independent settings about whether PKIX and/or DANE
verification has to be done, and/or whether verification errors have to be
ignored (e.g. for tls-required: no header). also, cached mtasts policies of
mode "none" are now cleaned up once the MTA-STS DNS record goes away.
in smtpserver, we store dmarc evaluations (under the right conditions).
in dmarcdb, we periodically (hourly) send dmarc reports if there are
evaluations. for failed deliveries, we deliver the dsn quietly to a submailbox
of the postmaster mailbox.
this is on by default, but can be disabled in mox.conf.
we set the flag both for move in imap and in webmail.
this also ensures the "MailboxDestinedID", used for per-mailbox reputation
analysis, is set in more reject-situations. before this change, some rejects
(such as based on DMARC reject) wouldn't result in reputation being used after
having been moved the message out of the rejects mailbox.
in the future, we need more tests for scenario's like this...
for issue #63 reported by x8x
may also help with issue #64
this is based on @bobobo1618's PR #50. bobobo1618 had the right idea, i tried
including an "is forwarded email" configuration option but that indeed became
too tightly coupled. the "is forwarded" option is still planned, but it is
separate from the "accept rejects to mailbox" config option, because one could
still want to push back on forwarded spam messages.
we do an actual accept, delivering to a configured mailbox, instead of storing
to the rejects mailbox where messages can automatically be removed from. one
of the goals of mox is not pretend to accept email while actually junking it.
users can still configure delivery to a junk folder (as was already possible),
but aren't deleted automatically. there is still an X-Mox-Reason header in the
message, and a log line about accepting the reject, but otherwise it is
registered and treated as an (smtp) accept.
the ruleset mailbox is still required to keep that explicit. users can specify
Inbox again.
hope this is good enough for PR #50, otherwise we'll change it.
the idea of the rejects mailbox is to show messages that were rejected. you can
look there, and if you see a message that should have been delivered, you can
move it to your inbox or archive. next time a deliver attempt by that user is
attempted, they should be accepted, because you corrected the reject. but that
wasn't happening, because the reputation-calculation is per-delivery mailbox
(e.g. Inbox) and we look at MailboxOrigID when calculating the reputation. and
that was set to the Rejects mailbox id, so the message wasn't considered. the
same applies to moving messages from Rejects to Junk (to train your filter).
we now keep track of a MailboxDestinedID, that is set to the mailbox that we
would have delivered to if we would not have rejected the message. then, when a
message is moved out of the Rejects mailbox, we change MailboxOrigID to
MailboxDestinedID. this essentially makes the message look like it was
delivered normally.
