we already allowed it for (authenticated) SMTP submission. it turns out also
legitimate senders can use this invalid syntax to deliver messages.
for issue #101 by Fell, thanks for reporting & explaining!
the original size, with bare newlines, was stored in the database, not the
actual adjusted file size. this caused failures when reading the message.
users may want to run "mox fixmsgsize <account>" if they imported messages from
another account over IMAP.
reported by daftaupe, thanks!
i'm not sure this is good enough.
this is based on field MsgFromValidation, but it doesn't hold the full DMARC information.
we also don't know mailing list-status for all historic messages.
so the red underline can occur too often.
both cases are quite typical for spammers, and not for legitimate senders.
this doesn't apply to known senders. and it only requires that the content look
more like ham instead of spam. so legitimate mail can still get through with
these properties.
because that is what most of the code expects. we could work around having bare
lf, but it would complicate too much code.
currently, a message with bare lf is accepted (in smtpserver delivery,
imapserver append, etc), but when an imap session would try to fetch parsed
parts, that would fail because and even cause a imapserver panic (closing the
connection).
in message imports we would already convert bare lf to crlf (because it is
expected those messages are all lf-only-ending).
we store messages with crlf-ending instead of lf-ending so the imapserver has
all correct information at hand (line counts, byte counts).
found by using emclient with mox. it adds a message to the inbox that can have
mixed crlf and bare lf line endings in a few header fields (in some
localization, emclient authors explained how that happened, thanks!). we can
now convert those lines and read those messages over imap. emclient already
switched to all-crlf line endings in newer (development) versions.
based on discussion on uta mailing list. it seems the intention of the tlsrpt
is to only send reports to recipient domains. but i was able to interpret the
tlsrpt rfc as sending reports to mx hosts too ("policy domain", and because it
makes sense given how DANE works per MX host, not recipient domain). this
change makes the behaviour of outgoing reports to recipient domains work more
in line with expectations most folks may have about tls reporting (i.e. also
include per-mx host tlsa policies in the report). this also keeps reports to mx
hosts working, and makes them more useful by including the recipient domains of
affected deliveries.
anything that looks like it specifies a different host should not be loaded.
www.xmox.nl also has a CSP policy that should prevent resources from other
domains from being loaded.
- show commit hash, with a link to the commit
- highlight if this is the dev or released version page
- sort the rfc's, the list in rfc/index.txt has the major rfc's at the topic, but this nuance is lost in the html page
for reporting addresses that cause DSNs to be returned. that just adds noise.
the admin can add/remove/extend addresses through the webadmin.
in the future, we could send reports with a smtp mail from of
"postmaster+<signed-encoded-recipient>@...", and add the reporting recipient
on the suppression list automatically when a DSN comes in on that address, but
for now this will probably do.
so facilitate debugging. a remote client that logs details about failing
connections can give the cid to the mox operator to find the relevant logging.
instead of requiring policy domains to be configured recipient domains.
when accessing TLS reports, always do it under path #tlsrpt/reports, not under #domain/.../tlsrpt.
- accept incoming tls reports for the host, with policy-domain the host name.
instead of not storing the domain because it is not a configured (recipient)
domain.
- in tlsrpt summaries, rename domain to policy domain for clarity.
- in webadmin, fix html for table that lists tls reports in case of multiple
policies and/or multiple failure details.
- the rfc links back to the code now show any "todo" text that appears in the
code. helps when looking at an rfc to find any work that may need to be done.
- html pages can now be generated to view code and rfc's side by side. clicking
on links in one side opens the linked document in the other page, at the
correct line number.
i'll be publishing the "dev" html version (latest commit on main branch) on the
mox website, updated with each commit. the dev pages will also link to the
latest released version.
- dmarc reports: add a cid to the log line about one run of sending reports, and log line for each report
- in smtpclient, also handle tls errors from the first read after a handshake. we appear to sometimes get tls alerts about bad certificates on the first read.
- for messages to dmarc/tls reporting addresses that we think should/can not be processed as reports, add an X-Mox- header explaining the reason.
- tls reports: send report messages with From address of postmaster at an actually configured domain for the mail host. and only send reports when dkim signing is configured for that domain. the domain is also the submitter domain. the rfc seems to require dkim-signing with an exact match with the message from and submitter.
- for incoming tls reports, in the smtp server, we do allow a dkim-signature domain that is higher-level (up to publicsuffix) of the message from domain. so we are stricter in what we send than what we receive.
we were already accepting, processing and displaying incoming tls reports. now
we start tracking TLS connection and security-policy-related errors for
outgoing message deliveries as well. we send reports once a day, to the
reporting addresses specified in TLSRPT records (rua) of a policy domain. these
reports are about MTA-STS policies and/or DANE policies, and about
STARTTLS-related failures.
sending reports is enabled by default, but can be disabled through setting
NoOutgoingTLSReports in mox.conf.
only at the end of the implementation process came the realization that the
TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy
for the recipient domain, and that MTA-STS and DANE TLS/policy results are
typically delivered in separate reports. so MX hosts need their own TLSRPT
policies.
config for the per-host TLSRPT policy should be added to mox.conf for existing
installs, in field HostTLSRPT. it is automatically configured by quickstart for
new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin
pages now suggest the per-host TLSRPT record. by creating that record, you're
requesting TLS reports about your MX host.
gathering all the TLS/policy results is somewhat tricky. the tentacles go
throughout the code. the positive result is that the TLS/policy-related code
had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect
reality better, with independent settings about whether PKIX and/or DANE
verification has to be done, and/or whether verification errors have to be
ignored (e.g. for tls-required: no header). also, cached mtasts policies of
mode "none" are now cleaned up once the MTA-STS DNS record goes away.
this also has changes to make the dmarc report sending implementation more
similar to the tls reports implementation.
- factor out code to compose a dmarc report message to the message package
(from dmarcdb for reports), it will be shared soon.
- spread emails with dmarc reports over 45 minutes (it runs hourly), with at
most 5 mins in between reports. to prevent bursts of messages. properly abort
all sending attempts at mox shutdown.
- add use of missing error details in an error path.
- fix dmarc report message subject header by adding missing <>'s around report-id.
- fix dmarc report attachment filename syntax by leaving "unique-id" out.
firstly by using crypto/rand in those cases. and secondly by putting a lock
around the Read (though it isn't used at the moment).
found while working while implementing sending tls reports.
we were trying to offset the timezone, but that makes no sense: we already
created a date in the local timezone based on (milli)seconds passed. so we can
just use that date instead of calculating a wrong date.
before this change, a message in the rejects folder that was read and marked as
notjunk (e.g. automatically by webmail), could cause a dmarc report to be sent
for another junky message from the domain. we now require positive signals to
be for messages not in the rejects mailbox.
the text/plain body of a dmarc report contains the period, but it was in local
time while claiming to be in utc. make it utc, so we often get nicely rounded
whole 24h utc days.
by fixing a typo in the content-type...
and by recognizing the application/x-zip that is detected as content-type.
discovered when a dmarc report from aws ses wasn't processed.
it seems aws ses was sending a dmarc report because it received a dmarc report.
