the backup command will make consistent snapshots of all the database files. i
had been copying the db files before, and it usually works. but if the file is
modified during the backup, it is inconsistent and is likely to generate errors
when reading (can be at any moment in the future, when reading some db page).
"mox backup" opens the database file and writes out a copy in a transaction.
it also duplicates the message files.
before doing a restore, you could run "mox verifydata" on the to-be-restored
"data" directory. it check the database files, and compares the message files
with the database.
the new "gentestdata" subcommand generates a basic "data" directory, with a
queue and a few accounts. we will use it in the future along with "verifydata"
to test upgrades from old version to the latest version. both when going to the
next version, and when skipping several versions. the script test-upgrades.sh
executes these tests and doesn't do anything at the moment, because no releases
have this subcommand yet.
inspired by a failed upgrade attempt of a pre-release version.
by specifying a "destination" in an account that is just "@" followed by the
domain, e.g. "@example.org". messages are only delivered to the catchall
address when no regular destination matches (taking the per-domain
catchall-separator and case-sensisitivity into account).
for issue #18
localserve creates a config for listening on localhost for
imap/smtp/submission/http, on port numbers 1000 + the common service port
numbers. all incoming email is accepted (if checks pass), and a few pattern in
localparts are recognized and result in delivery errors.
the idea is to make it clear from the logging if non-ascii characters are used.
this is implemented by making mlog recognize if a field value that will be
logged has a LogString method. if so, that value is logged. dns.Domain,
smtp.Address, smtp.Localpart, smtp.Path now have a LogString method.
some explicit calls to String have been replaced to LogString, and some %q
formatting have been replaced with %s, because the escaped localpart would
already have double quotes, and double doublequotes aren't easy to read.
you can already get most http to https redirects through DontRedirectPlainHTTP
in WebHandler, but that needs handlers for all paths.
now you can just set up a redirect for a domain and all its path to baseurl
https://domain (leaving other webdirect fields empty). when the request comes
in with plain http, the redirect to https is done. that next request will also
evaluate the same redirect rule. but it will not cause a match because it would
redirect to the same scheme,host,path. so next webhandlers get a chance to
serve.
also clarify in webhandlers docs that also account & admin built-in handlers
run first.
related to issue #16
- make it easier to run with an existing webserver. the quickstart now has a new option for that, it generates a different mox.conf, and further instructions such as configuring the tls keys/certs and reverse proxy urls. and changes to make autoconfig work in that case too.
- when starting up, request a tls cert for the hostname and for the autoconfig endpoint. the first will be requested soon anyway, and the autoconfig cert is needed early so the first autoconfig request doesn't time out (without helpful message to the user by at least thunderbird). and don't request the certificate before the servers are online. the root process was now requesting the certs, before the child process was serving on the tls port.
- add examples of configs generated by the quickstart.
- enable debug logging in config from quickstart, to give user more info.
for issue #5
- make builtin http handlers serve on specific domains, such as for mta-sts, so
e.g. /.well-known/mta-sts.txt isn't served on all domains.
- add logging of a few more fields in access logging.
- small tweaks/bug fixes in webserver request handling.
- add config option for redirecting entire domains to another (common enough).
- split httpserver metric into two: one for duration until writing header (i.e.
performance of server), another for duration until full response is sent to
client (i.e. performance as perceived by users).
- add admin ui, a new page for managing the configs. after making changes
and hitting "save", the changes take effect immediately. the page itself
doesn't look very well-designed (many input fields, makes it look messy). i
have an idea to improve it (explained in admin.html as todo) by making the
layout look just like the config file. not urgent though.
i've already changed my websites/webapps over.
the idea of adding a webserver is to take away a (the) reason for folks to want
to complicate their mox setup by running an other webserver on the same machine.
i think the current webserver implementation can already serve most common use
cases. with a few more tweaks (feedback needed!) we should be able to get to 95%
of the use cases. the reverse proxy can take care of the remaining 5%.
nevertheless, a next step is still to change the quickstart to make it easier
for folks to run with an existing webserver, with existing tls certs/keys.
that's how this relates to issue #5.
- serve static files, serving index.html or optionally listings for directories
- redirects
- reverse-proxy, forwarding requests to a backend
these are configurable through the config file. a domain and path regexp have to
be configured. path prefixes can be stripped. configured domains are added to
the autotls allowlist, so acme automatically fetches certificates for them.
all webserver requests now have (access) logging, metrics, rate limiting.
on http errors, the error message prints an encrypted cid for relating with log files.
this also adds a new mechanism for example config files.
makes it easier to run on bsd's, where you cannot (easily?) let non-root users
bind to ports <1024. starting as root also paves the way for future improvements
with privilege separation.
unfortunately, this requires changes to how you start mox. though mox will help
by automatically fix up dir/file permissions/ownership.
if you start mox from the systemd unit file, you should update it so it starts
as root and adds a few additional capabilities:
# first update the mox binary, then, as root:
./mox config printservice >mox.service
systemctl daemon-reload
systemctl restart mox
journalctl -f -u mox &
# you should see mox start up, with messages about fixing permissions on dirs/files.
if you used the recommended config/ and data/ directory, in a directory just for
mox, and with the mox user called "mox", this should be enough.
if you don't want mox to modify dir/file permissions, set "NoFixPermissions:
true" in mox.conf.
if you named the mox user something else than mox, e.g. "_mox", add "User: _mox"
to mox.conf.
if you created a shared service user as originally suggested, you may want to
get rid of that as it is no longer useful and may get in the way. e.g. if you
had /home/service/mox with a "service" user, that service user can no longer
access any files: only mox and root can.
this also adds scripts for building mox docker images for alpine-supported
platforms.
the "restart" subcommand has been removed. it wasn't all that useful and got in
the way.
and another change: when adding a domain while mtasts isn't enabled, don't add
the per-domain mtasts config, as it would cause failure to add the domain.
based on report from setting up mox on openbsd from mteege.
and based on issue #3. thanks for the feedback!
this matches how the quickstart recommends setting up mox.
i've been running "./mox -config/mox.conf" from there for admin commands, but
"mox" is enough now.
before, we used heuristics to decide when to train/untrain a message as junk or
nonjunk: the message had to be seen, be in certain mailboxes. then if a message
was marked as junk, it was junk. and otherwise it was nonjunk. this wasn't good
enough: you may want to keep some messages around as neither junk or nonjunk.
and that wasn't possible.
ideally, we would just look at the imap $Junk and $NotJunk flags. the problem
is that mail clients don't set these flags, or don't make it easy. thunderbird
can set the flags based on its own bayesian filter. it has a shortcut for
marking Junk and moving it to the junk folder (good), but the counterpart of
notjunk only marks a message as notjunk without showing in the UI that it was
marked as notjunk. there is also no "move and mark as notjunk" mechanism. e.g.
"archive" does not mark a message as notjunk. ios mail and mutt don't appear to
have any way to see or change the $Junk and $NotJunk flags.
what email clients do have is the ability to move messages to other
mailboxes/folders. so mox now has a mechanism that allows you to configure
mailboxes that automatically set $Junk or $NotJunk (or clear both) when a
message is moved/copied/delivered to that folder. e.g. a mailbox called junk or
spam or rejects marks its messags as junk. inbox, postmaster, dmarc, tlsrpt,
neutral* mark their messages as neither junk or notjunk. other folders mark
their messages as notjunk. e.g. list/*, archive. this functionality is
optional, but enabled with the quickstart and for new accounts.
also, mox now keeps track of the previous training of a message and will only
untrain/train if needed. before, there probably have been duplicate or missing
(un)trainings.
this also includes a new subcommand "retrain" to recreate the junkfilter for an
account. you should run it after updating to this version. and you should
probably also modify your account config to include the AutomaticJunkFlags.
otherwise, setting a password will block if something has the account open,
e.g. any imap connection. so in practice it only worked when mox isn't running.
and change thunderbird autoconfiguration to use it.
unfortunately, for microsoft autodiscover, there appears to be no way to
request secure password negotiation. so it will default to plain text auth.
cram-md5 is less secure than scram-sha-*, but thunderbird does not yet support
scram auth. it currently chooses "plain", sending the literal password over the
connection (which is TLS-protected, but we don't want to receive clear text
passwords). in short, cram-md5 is better than nothing...
for cram-md5 to work, a new set of derived credentials need to be stored in the
database. so you need to save your password again to make it work. this was
also the case with the scram-sha-1 addition, but i forgot to mention it then.
named "traceauth" and "tracedata".
with this, you can (almost) enable trace logging without fear of logging
sensitive data or ddos'ing your log server.
the caveat is that the imap login command has already printed the line as
regular trace before we can decide it should not be. can be fixed soon.
- and don't have a global variable "d" in the big checkDomain function in http/admin.go.
- and set loglevel from command-line flag again after loading the config file, for all subcommands except "serve".
- when sendmail is invoked to sent to a local user, eg "root", send to a
configured defaultaddress instead. previously, we would fail printing usage.
- add a subcommand that prints the config file, /etc/moxsubmit.conf.
- follow the STARTTLS setting.
- fix running as just plain sendmail.
