Commit graph

786 commits

Author SHA1 Message Date
Mechiel Lukkien
c955fadb6d
fix parsing dmarc reports that come with content-type application/octet-stream
by fixing a typo in the content-type...
and by recognizing the application/x-zip that is detected as content-type.

discovered when a dmarc report from aws ses wasn't processed.

it seems aws ses was sending a dmarc report because it received a dmarc report.
2023-11-04 13:22:30 +01:00
Mechiel Lukkien
3a7ed9738a
update to latest go.etcd.io/bbolt v1.3.8 2023-11-03 08:31:30 +01:00
Mechiel Lukkien
4510e0ce78
webmail: add Delivered-To to example settingsPut call 2023-11-02 21:56:59 +01:00
Mechiel Lukkien
0200e539a9
when message is delivered, save whether it is from a mailing list; in webmail, show if message was a forward or mailing list, and don't enable requiretls when sending to a list. 2023-11-02 20:03:47 +01:00
Mechiel Lukkien
481a25f294
improvements to outgoing dmarc reports and displaying evaluations
- more eagerly report about overrides, so domain owners can better tell that
  switching from p=none to p=reject will not cause trouble for these messages.
- report multiple reasons, e.g. mailing list and sampled out
- in dmarc analysis for rejects from first-time senders (possibly spammers),
  fix the conditional check on nonjunk messages.
- in evaluations view in admin, show unaligned spf pass in yellow too and a few
  more small tweaks.
2023-11-02 17:54:24 +01:00
Mechiel Lukkien
79e522887e
change error value "fatal io error" to just "io error"
"fatal" was meant as "we need fatal for the connection, it will be dropped".
but it sounds more serious, as if something needs to be fixed.

hopefully enough for issue #39 by ArnoSen
2023-11-02 15:56:01 +01:00
Mechiel Lukkien
38694d3928
Merge remote-tracking branch 'github.com/mattfbacon/mox/message-is-text' 2023-11-02 14:41:43 +01:00
Mechiel Lukkien
81057ee685
add option -initonly to "mox localserve", to only create config files and then quit
for issue #89 by naturalethic
2023-11-02 14:10:41 +01:00
Mechiel Lukkien
9896639ff9
for incoming smtp deliveries, track whether tls and requiretls was used, and display this in the webmail
we store the tls version used, and cipher suite. we don't currently show that
in the webmail.
2023-11-02 09:12:47 +01:00
Mechiel Lukkien
186538fb57
when composing a dsn, try harder to dkim-sign it, also with higher-level domain than full mail hostname
e.g. typical setup is a hostname mail.<domain>. and dsns can be sent from
postmaster@mail.<domain>. so it helps to look for dkim keys for <domain>, and
use them when signing. instead of looking for dkim keys for mail.<domain>,
which won't typically exist.  similar to recent commit that added outgoing
dmarc aggregate reports.
2023-11-02 09:12:47 +01:00
Mechiel Lukkien
f7686b7db8
webmail: show email address instead of display name of "from" header in message listing if display name contains chars from "<@>"
it could be an attempt to confuse the reader with an email address. a classic.
2023-11-02 09:12:47 +01:00
Mechiel Lukkien
725f030d3c
webmail: add clear marker between message header and body, so if html message tries to fake ui elements, it'll be noticed (hopefully) 2023-11-02 09:12:47 +01:00
Mechiel Lukkien
ef50f4abf0
refactor common pattern of close & remove temporary file into calling the new store.CloseRemoveTempFile 2023-11-02 09:12:46 +01:00
Mechiel Lukkien
b6897d1837
add note about adns library 2023-11-02 09:12:46 +01:00
Mechiel Lukkien
e7699708ef
implement outgoing dmarc aggregate reporting
in smtpserver, we store dmarc evaluations (under the right conditions).
in dmarcdb, we periodically (hourly) send dmarc reports if there are
evaluations. for failed deliveries, we deliver the dsn quietly to a submailbox
of the postmaster mailbox.

this is on by default, but can be disabled in mox.conf.
2023-11-02 09:12:30 +01:00
Matt Fellenz
3b6e1851cb
Treat messages as text 2023-11-01 14:17:02 -07:00
Mechiel Lukkien
d1e93020d8
give delivering to mx targets with underscores in name a chance of succeeding
the underscores aren't valid, but have been seen in the wild, so we have a
workaround for them. there are limitations, it won't work with idna domains.
and if the domain has other policies, like mta-sts, the mx host won't pass
either.

after report from richard g about delivery issue, thanks!
2023-10-25 13:01:11 +02:00
Mechiel Lukkien
682f8a0904
dkim selectors shouldn't be interpreted as idna
given they are not part of the domain name (to which idna applies).
only the part after _domainkey may be idna.
found after going through code after report about mx targets with underscores
from richard g.
2023-10-25 12:49:39 +02:00
Mechiel Lukkien
34f7e04474
update roadmap 2023-10-25 12:33:22 +02:00
Mechiel Lukkien
8a866a60dc
when expunging a message, keep its threadid
we will need it for jmap, which needs history for threads
2023-10-24 13:16:00 +02:00
Mechiel Lukkien
7b047ed28d
no need for absolute path for prometheus endpoint pointing to metrics 2023-10-24 13:11:04 +02:00
Mechiel Lukkien
a6d55b7e76
add metric for number of times we fallback to plaintext delivery 2023-10-24 13:09:48 +02:00
Mechiel Lukkien
f9eb18b6a8
for mox localserve, only require being able to parse incoming messages over smtp as parsable with pedantic mode 2023-10-24 13:03:50 +02:00
Mechiel Lukkien
5b4de0523d
ignore mox.exe, since we can now build for windows 2023-10-24 13:02:06 +02:00
Mechiel Lukkien
2f5d6069bf
implement "requiretls", rfc 8689
with requiretls, the tls verification mode/rules for email deliveries can be
changed by the sender/submitter. in two ways:

1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or
dnssec+dane), along the entire delivery path until delivery into the final
destination mailbox (so entire transport is verified-tls-protected).

2. "tls-required: no" message header, to ignore any tls and tls verification
errors even if the recipient domain has a policy that requires tls verification
(mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in
case of misconfiguration/interoperability issues (at least useful for sending
tls reports).

we enable requiretls by default (only when tls is active), for smtp and
submission. it can be disabled through the config.

for each delivery attempt, we now store (per recipient domain, in the account
of the sender) whether the smtp server supports starttls and requiretls. this
support is shown (after having sent a first message) in the webmail when
sending a message (the previous 3 bars under the address input field are now 5
bars, the first for starttls support, the last for requiretls support). when
all recipient domains for a message are known to implement requiretls,
requiretls is automatically selected for sending (instead of "default" tls
behaviour). users can also select the "fallback to insecure" to add the
"tls-required: no" header.

new metrics are added for insight into requiretls errors and (some, not yet
all) cases where tls-required-no ignored a tls/verification error.

the admin can change the requiretls status for messages in the queue. so with
default delivery attempts, when verified tls is required by failing, an admin
could potentially change the field to "tls-required: no"-behaviour.

messages received (over smtp) with the requiretls option, get a comment added
to their Received header line, just before "id", after "with".
2023-10-24 10:10:46 +02:00
Moritz Poldrack
0e5e16b3d0
main: remove redundant equal function 2023-10-21 16:49:28 +02:00
Mechiel Lukkien
08995c7806
webmail: when composing a message, show security status in a bar below addressee input field
the bar is currently showing 3 properties:
1. mta-sts enforced;
2. mx lookup returned dnssec-signed response;
3. first delivery destination host has dane records

the colors are: red for not-implemented, green for implemented, gray for error,
nothing for unknown/irrelevant.

the plan is to implement "requiretls" soon and start caching per domain whether
delivery can be done with starttls and whether the domain supports requiretls.
and show that in two new parts of the bar.

thanks to damian poddebniak for pointing out that security indicators should
always be visible, not only for positive/negative result. otherwise users won't
notice their absence.
2023-10-15 15:40:13 +02:00
Mechiel Lukkien
4ab3e6bc9b
webmail: autoresize address input field in compose window
so full name/email address is visible.

using a hidden grid element that gets the same content as the input element.
from https://css-tricks.com/auto-growing-inputs-textareas/

a recent commit probably also make the compose window full-screen-width on
chrome, this restores to the intended behaviour of a less wide default size.

if you add multiple address fields, the compose window will still grow. not
great, in the future, we should make the compose window resizable by dragging.
2023-10-15 10:53:57 +02:00
Mechiel Lukkien
101c2703d2
do not lookup cname after looking up the txt for mta-sts, and follow cnames for mocks
because the txt would already follow cnames.
the additional cname lookup didn't hurt, it just didn't do anything.
i probably didn't realize that before looking deeper into dns.
2023-10-14 22:42:26 +02:00
Mechiel Lukkien
8ca198882e
security fix: use correct domain for mta-sts, that of the email address
the original next-hop domain. not anything after resolving cname's, because
then it takes just a single injected dns cname record to lead us to an
unrelated server (that we would verify, but it's the wrong server).

also don't fallback to just strict tls when something is wrong. we must use the
policy to check if an mx host is allowed. the whole idea is that unsigned dns
records cannot be trusted.

i noticed this while implementing dane.
2023-10-14 22:30:43 +02:00
Mechiel Lukkien
42d817ef3d
quick fix for making compose window resizable by expanding/shrinking when textarea is resized
the textarea is resizable (though it's not convenient to do in firefox which
only shows a dragcorner in the bottomright, usually located in the bottom
corner of the screen, so there is little space left to drag the corner; the
workaround is to move the window temporarily).
2023-10-14 21:02:54 +02:00
Mechiel Lukkien
56956c224b
webmail: when quoting text that switches unicode blocks (as highlighted), don't lose the switched text
by using a String object as the textarea child.  instead of a regular js string
that would be unicode-block-switch-highlighted, which would cause it to be
split into parts, with odd or even parts added as span elements, which the
textarea would then ignore.
2023-10-14 14:47:24 +02:00
Mechiel Lukkien
a40f5a5eb3
webmail: recognize q/b-word-encoded filenames in attachments in messages
according to the rfc's (2231, and 2047), non-ascii filenames in content-type
and content-disposition headers should be encoded like this:

	Content-Type: text/plain; name*=utf-8''hi%E2%98%BA.txt
	Content-Disposition: attachment; filename*=utf-8''hi%E2%98%BA.txt

and that is what the Go standard library mime.ParseMediaType and
mime.FormatMediaType parse and generate.

this is what thunderbird sends:

	Content-Type: text/plain; charset=UTF-8; name="=?UTF-8?B?aGnimLoudHh0?="
	Content-Disposition: attachment; filename*=UTF-8''%68%69%E2%98%BA%2E%74%78%74

(thunderbird will also correctly split long filenames over multiple parameters,
named "filename*0*", "filename*1*", etc.)

this is what gmail sends:

	Content-Type: text/plain; charset="US-ASCII"; name="=?UTF-8?B?aGnimLoudHh0?="
	Content-Disposition: attachment; filename="=?UTF-8?B?aGnimLoudHh0?="

i cannot find where the q/b-word encoded values in "name" and "filename" are
allowed. until that time, we try parsing them unless in pedantic mode.

we didn't generate correctly encoded filenames yet, this commit also fixes that.

for issue #82 by mattfbacon, thanks for reporting!
2023-10-14 14:14:13 +02:00
Mechiel Lukkien
3e53343d21
remove message during delivery when we encounter an error after having placed the message in the destination path
before, we would leave the file, but rollback the delivery. future deliveries
would attempt to deliver to the same path, but would fail because a file
already exists.

encountered during testing on windows, not during actual operation. though it
could in theory have happened.
2023-10-14 11:16:39 +02:00
Mechiel Lukkien
6e391c3be0
ensure there is a space between active requests mentioned in the status bar at the top 2023-10-14 11:13:26 +02:00
Mechiel Lukkien
28fae96a9b
make mox compile on windows, without "mox serve" but with working "mox localserve"
getting mox to compile required changing code in only a few places where
package "syscall" was used: for accessing file access times and for umask
handling. an open problem is how to start a process as an unprivileged user on
windows.  that's why "mox serve" isn't implemented yet. and just finding a way
to implement it now may not be good enough in the near future: we may want to
starting using a more complete privilege separation approach, with a process
handling sensitive tasks (handling private keys, authentication), where we may
want to pass file descriptors between processes. how would that work on
windows?

anyway, getting mox to compile for windows doesn't mean it works properly on
windows. the largest issue: mox would normally open a file, rename or remove
it, and finally close it. this happens during message delivery. that doesn't
work on windows, the rename/remove would fail because the file is still open.
so this commit swaps many "remove" and "close" calls. renames are a longer
story: message delivery had two ways to deliver: with "consuming" the
(temporary) message file (which would rename it to its final destination), and
without consuming (by hardlinking the file, falling back to copying). the last
delivery to a recipient of a message (and the only one in the common case of a
single recipient) would consume the message, and the earlier recipients would
not.  during delivery, the already open message file was used, to parse the
message.  we still want to use that open message file, and the caller now stays
responsible for closing it, but we no longer try to rename (consume) the file.
we always hardlink (or copy) during delivery (this works on windows), and the
caller is responsible for closing and removing (in that order) the original
temporary file. this does cost one syscall more. but it makes the delivery code
(responsibilities) a bit simpler.

there is one more obvious issue: the file system path separator. mox already
used the "filepath" package to join paths in many places, but not everywhere.
and it still used strings with slashes for local file access. with this commit,
the code now uses filepath.FromSlash for path strings with slashes, uses
"filepath" in a few more places where it previously didn't. also switches from
"filepath" to regular "path" package when handling mailbox names in a few
places, because those always use forward slashes, regardless of local file
system conventions.  windows can handle forward slashes when opening files, so
test code that passes path strings with forward slashes straight to go stdlib
file i/o functions are left unchanged to reduce code churn. the regular
non-test code, or test code that uses path strings in places other than
standard i/o functions, does have the paths converted for consistent paths
(otherwise we would end up with paths with mixed forward/backward slashes in
log messages).

windows cannot dup a listening socket. for "mox localserve", it isn't
important, and we can work around the issue. the current approach for "mox
serve" (forking a process and passing file descriptors of listening sockets on
"privileged" ports) won't work on windows. perhaps it isn't needed on windows,
and any user can listen on "privileged" ports? that would be welcome.

on windows, os.Open cannot open a directory, so we cannot call Sync on it after
message delivery. a cursory internet search indicates that directories cannot
be synced on windows. the story is probably much more nuanced than that, with
long deep technical details/discussions/disagreement/confusion, like on unix.
for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 10:54:07 +02:00
Mechiel Lukkien
96774de8d6
add workaround for windows mail authentication in smtpserver 2023-10-13 21:35:03 +02:00
Mechiel Lukkien
8640fd8cff
webmail: top-post with no text selected and add "on ... wrote"-line, keep bottom-quoting with text selected
top-posting causes "On $datetime, $sender wrote:" above the quoted text to be
added (unless there was no Date header or valid address in a From header).

in the near future we should create settings, and add a setting for adding the
"on ... wrote"-line, ideally including a template.

for issue #83 by mattfbacon, thanks!
2023-10-13 19:28:04 +02:00
Mechiel Lukkien
7d28d80191
if requesting a tls certificate through acme fails, put any validation error messages provided by the acme server in the error message
so users can understand what is going on. e.g. a CAA record that doesn't allow
a CA to sign a certificate. previously, the error message would just be "no
viable challenge type found", which doesn't help the user.
2023-10-13 09:28:01 +02:00
Mechiel Lukkien
14d09bb308
format long multi-string dkim txt records for rsa 2048 as a mult-line record, enclosed in ()'s
more easily readable, though still long
2023-10-13 09:14:42 +02:00
Mechiel Lukkien
40040542f6
for generated dkim keys, use clearer file names
with ".rsa2048.privatekey.pkcs8.pem", instead of "rsakey.pkcs8.pem". "rsakey"
doesn't say if it is a public or private key.
2023-10-13 08:59:35 +02:00
Mechiel Lukkien
4e26fd13e2
when api docs cannot be loaded, say which 2023-10-13 08:52:06 +02:00
Mechiel Lukkien
67fe88f431
change the autodiscover SRV record to point to the mail server hostname directly, not to a cname
srv targest shouldn't be cname's. bind was warning about it.
2023-10-13 08:51:02 +02:00
Mechiel Lukkien
850f4444d4
when suggesting DNS records, leave "IN" out
people will either paste the records in their zone file. in that case, the
records will inherit "IN" from earlier records, and there will always be one
record. if anyone uses a different class, their smart enough to know they need
to add IN manually.

plenty of people will add their records through some clunky web interface of
their dns operator. they probably won't even have the choice to set the class,
it'll always be IN.
2023-10-13 08:25:35 +02:00
Mechiel Lukkien
52e71167a9
rename rfc/index.md to txt, it isn't markdown 2023-10-12 23:15:54 +02:00
Mechiel Lukkien
a93dd348fe
webmail: ensure wrap of long header lines, instead of horizontal scrollbar in message header section 2023-10-12 22:08:13 +02:00
Mechiel Lukkien
8dacc31445
webmail: for high images (aspect ratio), don't let image extend beyond window height
apparently the flex parent and flex child with grow 1 is unbounded even with a parent height of 100%
2023-10-12 21:53:05 +02:00
Mechiel Lukkien
7dce883097
simplify dns.MockResolver, changing MockReq to just a string representing the request
similar to Authentic/Inauthentic
2023-10-12 16:07:53 +02:00
Mechiel Lukkien
c095f3f39c
in "mox import ..." help output, make it more clear what should be done to make mbox/maildir archives accessible to the mox process
for issue #79 reported by mattfbacon, thanks!
2023-10-12 15:50:43 +02:00
Mechiel Lukkien
daa908e9f4
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery
the vendored dns resolver code is a copy of the go stdlib dns resolver, with
awareness of the "authentic data" (i.e. dnssec secure) added, as well as support
for enhanced dns errors, and looking up tlsa records (for dane). ideally it
would be upstreamed, but the chances seem slim.

dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their
dnssec status is added to the Received message headers for incoming email.

but the main reason to add dnssec was for implementing dane. with dane, the
verification of tls certificates can be done through certificates/public keys
published in dns (in the tlsa records). this only makes sense (is trustworthy)
if those dns records can be verified to be authentic.

mox now applies dane to delivering messages over smtp. mox already implemented
mta-sts for webpki/pkix-verification of certificates against the (large) pool
of CA's, and still enforces those policies when present. but it now also checks
for dane records, and will verify those if present. if dane and mta-sts are
both absent, the regular opportunistic tls with starttls is still done. and the
fallback to plaintext is also still done.

mox also makes it easy to setup dane for incoming deliveries, so other servers
can deliver with dane tls certificate verification. the quickstart now
generates private keys that are used when requesting certificates with acme.
the private keys are pre-generated because they must be static and known during
setup, because their public keys must be published in tlsa records in dns.
autocert would generate private keys on its own, so had to be forked to add the
option to provide the private key when requesting a new certificate. hopefully
upstream will accept the change and we can drop the fork.

with this change, using the quickstart to setup a new mox instance, the checks
at internet.nl result in a 100% score, provided the domain is dnssec-signed and
the network doesn't have any issues.
2023-10-10 12:09:35 +02:00