renaming inbox is special. the mailbox isn't renamed, but its messages moved to
a new mailbox. we weren't updating the destination mailbox uidnext with the new
messages. the fix not only sets the uidnext correctly, but also renumbers the
uids, starting at 1.
this also adds a consistency check for message uids and mailbox uidnexts, and
for mailbox uidvalidity account nextuidvalidity in "mox verifydata".
this also adds command "mox fixuidmeta" (not listed) that fixes up mailbox uidnext
and account uidvalidity. and command "mox reassignuids" that will renumber the
uids for either one or all mailboxes in an account.
for a uid set, the syntax <num>:* must be interpreted as <num>:<maxuid>. a
wrong check turned the uid set into <maxuid>:<maxuid>. that check was meant for
the case where <num> is higher than <maxuid>, in which case num must be
replaced with maxuid.
this affected "uid expunge" with a uid set, possibly causing messages marked
for deletion not to be actually removed, and this affected "search" with the
uid parameter, possibly not returning all messages that were searched for.
found while writing tests for upcoming condstore/qresync extensions.
these could cause the parser to reject correct commands.
the first bug is about the allowed chars for an "atom", we were accepting too
many. this probably isn't easily triggered in practice.
the second bug is about how numbers (digits) are parsed. when gathering digits
to parse as number, we didn't consider only the directly upcoming digits that
make up the number, but continued looking for digits later on in the command.
then we tried to parse a string that was too long as a number, which would fail
because of additional characters. this could have been triggered with commands
containing two numbers. this is possible with e.g. "tag search or larger 123
smaller 123", the "or" takes two search keys again, each with a number. not too
common, but can happen.
found while writing tests for upcoming condstore/qresync implementation.
the mailbox select/examine responses now return all flags used in a mailbox in
the FLAGS response. and indicate in the PERMANENTFLAGS response that clients
can set new keywords. we store these values on the new Message.Keywords field.
system/well-known flags are still in Message.Flags, so we're recognizing those
and handling them separately.
the imap store command handles the new flags. as does the append command, and
the search command.
we store keywords in a mailbox when a message in that mailbox gets the keyword.
we don't automatically remove the keywords from a mailbox. there is currently
no way at all to remove a keyword from a mailbox.
the import commands now handle non-system/well-known keywords too, when
importing from mbox/maildir.
jmap requires keyword support, so best to get it out of the way now.
so external tools (like fail2ban) can monitor the logs and block ip's of bots.
for issue #30 by inigoserna, though i'm not sure i interpreted the suggestion correctly.
this is quite common on the internet. the other side may be trying some other
protocol, e.g. http, or some common vulnerability. we don't want to spam our
own logs with multiple invalid lines. if the first command is valid, but later
are not, we'll keep trying to process them. so this only affects protocol
sessions that are very likely not smtp/imap.
also remove a few more sleeps during tests, making imapserver and smtpserver tests a bit faster.
the idea of the rejects mailbox is to show messages that were rejected. you can
look there, and if you see a message that should have been delivered, you can
move it to your inbox or archive. next time a deliver attempt by that user is
attempted, they should be accepted, because you corrected the reject. but that
wasn't happening, because the reputation-calculation is per-delivery mailbox
(e.g. Inbox) and we look at MailboxOrigID when calculating the reputation. and
that was set to the Rejects mailbox id, so the message wasn't considered. the
same applies to moving messages from Rejects to Junk (to train your filter).
we now keep track of a MailboxDestinedID, that is set to the mailbox that we
would have delivered to if we would not have rejected the message. then, when a
message is moved out of the Rejects mailbox, we change MailboxOrigID to
MailboxDestinedID. this essentially makes the message look like it was
delivered normally.
makes it easier to run on bsd's, where you cannot (easily?) let non-root users
bind to ports <1024. starting as root also paves the way for future improvements
with privilege separation.
unfortunately, this requires changes to how you start mox. though mox will help
by automatically fix up dir/file permissions/ownership.
if you start mox from the systemd unit file, you should update it so it starts
as root and adds a few additional capabilities:
# first update the mox binary, then, as root:
./mox config printservice >mox.service
systemctl daemon-reload
systemctl restart mox
journalctl -f -u mox &
# you should see mox start up, with messages about fixing permissions on dirs/files.
if you used the recommended config/ and data/ directory, in a directory just for
mox, and with the mox user called "mox", this should be enough.
if you don't want mox to modify dir/file permissions, set "NoFixPermissions:
true" in mox.conf.
if you named the mox user something else than mox, e.g. "_mox", add "User: _mox"
to mox.conf.
if you created a shared service user as originally suggested, you may want to
get rid of that as it is no longer useful and may get in the way. e.g. if you
had /home/service/mox with a "service" user, that service user can no longer
access any files: only mox and root can.
this also adds scripts for building mox docker images for alpine-supported
platforms.
the "restart" subcommand has been removed. it wasn't all that useful and got in
the way.
and another change: when adding a domain while mtasts isn't enabled, don't add
the per-domain mtasts config, as it would cause failure to add the domain.
based on report from setting up mox on openbsd from mteege.
and based on issue #3. thanks for the feedback!
so users can easily take their email out of somewhere else, and import it into mox.
this goes a little way to give feedback as the import progresses: upload
progress is shown (surprisingly, browsers aren't doing this...), imported
mailboxes/messages are counted (batched) and import issues/warnings are
displayed, all sent over an SSE connection. an import token is stored in
sessionstorage. if you reload the page (e.g. after a connection error), the
browser will reconnect to the running import and show its progress again. and
you can just abort the import before it is finished and committed, and nothing
will have changed.
this also imports flags/keywords from mbox files.
ideally both account & admin web pages should be on non-public ips (e.g. a
wireguard tunnel). but during setup, users may not have that set up, and they
may want to configure the admin/account pages on their public ip's. the auth
rate limiting should make it less of issue.
users can now also only put the account web page publicly available. useful for
if you're the admin and you have a vpn connection, but your other/external
users do not have a vpn into your mail server. to make the account page more
easily findable, the http root serves the account page. the admin page is still
at /admin/, to prevent clash with potential account pages, but if no account
page is present, you are helpfully redirected from / to /admin/.
this also adds a prometheus metric counting how often auth attempts have been
rate limited.
before, we used heuristics to decide when to train/untrain a message as junk or
nonjunk: the message had to be seen, be in certain mailboxes. then if a message
was marked as junk, it was junk. and otherwise it was nonjunk. this wasn't good
enough: you may want to keep some messages around as neither junk or nonjunk.
and that wasn't possible.
ideally, we would just look at the imap $Junk and $NotJunk flags. the problem
is that mail clients don't set these flags, or don't make it easy. thunderbird
can set the flags based on its own bayesian filter. it has a shortcut for
marking Junk and moving it to the junk folder (good), but the counterpart of
notjunk only marks a message as notjunk without showing in the UI that it was
marked as notjunk. there is also no "move and mark as notjunk" mechanism. e.g.
"archive" does not mark a message as notjunk. ios mail and mutt don't appear to
have any way to see or change the $Junk and $NotJunk flags.
what email clients do have is the ability to move messages to other
mailboxes/folders. so mox now has a mechanism that allows you to configure
mailboxes that automatically set $Junk or $NotJunk (or clear both) when a
message is moved/copied/delivered to that folder. e.g. a mailbox called junk or
spam or rejects marks its messags as junk. inbox, postmaster, dmarc, tlsrpt,
neutral* mark their messages as neither junk or notjunk. other folders mark
their messages as notjunk. e.g. list/*, archive. this functionality is
optional, but enabled with the quickstart and for new accounts.
also, mox now keeps track of the previous training of a message and will only
untrain/train if needed. before, there probably have been duplicate or missing
(un)trainings.
this also includes a new subcommand "retrain" to recreate the junkfilter for an
account. you should run it after updating to this version. and you should
probably also modify your account config to include the AutomaticJunkFlags.
similar to greylisting, but not quite the same: with greylisting you would
always reject the first delivery attempt with a temporary failure. with the
hope that spammers won't retry their deliveries. the spams i've been receiving
seem to be quite consistent though. and we would keep rejecting them anyway.
we slow down the spammy connections to waste some of the resources of a
spammer. this may slow their campaigns down a bit, leaving a bit more time to
take measures.
we do the same with connections that have their 3rd authentication failure,
typically password guess attempts.
when we accept a message by a first-time sender, we sleep for 15 seconds before
actually delivering them. known-good senders don't have to wait. if the message
turns out to be a spammer, at least we've consumed one of their connections,
and they cannot deliver at too high a rate to us because of the max open
connection limit.
limiting is done based on remote ip's, with 3 ip mask variants to limit networks
of machines. often with two windows, enabling short bursts of activity, but not
sustained high activity. currently only for imap and smtp, not yet http.
limits are currently based on:
- number of open connections
- connection rate
- limits after authentication failures. too many failures, and new connections will be dropped.
- rate of delivery in total number of messages
- rate of delivery in total size of messages
the limits on connections and authentication failures are in-memory. the limits
on delivery of messages are based on stored messages.
the limits themselves are not yet configurable, let's use this first.
in the future, we may also want to have stricter limits for senders without any
reputation.
and change thunderbird autoconfiguration to use it.
unfortunately, for microsoft autodiscover, there appears to be no way to
request secure password negotiation. so it will default to plain text auth.
cram-md5 is less secure than scram-sha-*, but thunderbird does not yet support
scram auth. it currently chooses "plain", sending the literal password over the
connection (which is TLS-protected, but we don't want to receive clear text
passwords). in short, cram-md5 is better than nothing...
for cram-md5 to work, a new set of derived credentials need to be stored in the
database. so you need to save your password again to make it work. this was
also the case with the scram-sha-1 addition, but i forgot to mention it then.
the idea is that clients may not support SCRAM-SHA-256, but may support
SCRAM-SHA-1. if they do support the 256 variant, they'll use it.
unfortunately, thunderbird does not support scram-sha-1 either.
named "traceauth" and "tracedata".
with this, you can (almost) enable trace logging without fear of logging
sensitive data or ddos'ing your log server.
the caveat is that the imap login command has already printed the line as
regular trace before we can decide it should not be. can be fixed soon.
