this is based on @bobobo1618's PR #50. bobobo1618 had the right idea, i tried
including an "is forwarded email" configuration option but that indeed became
too tightly coupled. the "is forwarded" option is still planned, but it is
separate from the "accept rejects to mailbox" config option, because one could
still want to push back on forwarded spam messages.
we do an actual accept, delivering to a configured mailbox, instead of storing
to the rejects mailbox where messages can automatically be removed from. one
of the goals of mox is not pretend to accept email while actually junking it.
users can still configure delivery to a junk folder (as was already possible),
but aren't deleted automatically. there is still an X-Mox-Reason header in the
message, and a log line about accepting the reject, but otherwise it is
registered and treated as an (smtp) accept.
the ruleset mailbox is still required to keep that explicit. users can specify
Inbox again.
hope this is good enough for PR #50, otherwise we'll change it.
a few places still looked at the name "Sent". but since we have special-use
flags, we should always look at those. this also changes the config so admins
can specify different names for the special-use mailboxes to create for new
accounts, e.g. in a different language. the old config option is still
understood, just deprecated.
it was far down on the roadmap, but implemented earlier, because it's
interesting, and to help prepare for a jmap implementation. for jmap we need to
implement more client-like functionality than with just imap. internal data
structures need to change. jmap has lots of other requirements, so it's already
a big project. by implementing a webmail now, some of the required data
structure changes become clear and can be made now, so the later jmap
implementation can do things similarly to the webmail code. the webmail
frontend and webmail are written together, making their interface/api much
smaller and simpler than jmap.
one of the internal changes is that we now keep track of per-mailbox
total/unread/unseen/deleted message counts and mailbox sizes. keeping this
data consistent after any change to the stored messages (through the code base)
is tricky, so mox now has a consistency check that verifies the counts are
correct, which runs only during tests, each time an internal account reference
is closed. we have a few more internal "changes" that are propagated for the
webmail frontend (that imap doesn't have a way to propagate on a connection),
like changes to the special-use flags on mailboxes, and used keywords in a
mailbox. more changes that will be required have revealed themselves while
implementing the webmail, and will be implemented next.
the webmail user interface is modeled after the mail clients i use or have
used: thunderbird, macos mail, mutt; and webmails i normally only use for
testing: gmail, proton, yahoo, outlook. a somewhat technical user is assumed,
but still the goal is to make this webmail client easy to use for everyone. the
user interface looks like most other mail clients: a list of mailboxes, a
search bar, a message list view, and message details. there is a top/bottom and
a left/right layout for the list/message view, default is automatic based on
screen size. the panes can be resized by the user. buttons for actions are just
text, not icons. clicking a button briefly shows the shortcut for the action in
the bottom right, helping with learning to operate quickly. any text that is
underdotted has a title attribute that causes more information to be displayed,
e.g. what a button does or a field is about. to highlight potential phishing
attempts, any text (anywhere in the webclient) that switches unicode "blocks"
(a rough approximation to (language) scripts) within a word is underlined
orange. multiple messages can be selected with familiar ui interaction:
clicking while holding control and/or shift keys. keyboard navigation works
with arrows/page up/down and home/end keys, and also with a few basic vi-like
keys for list/message navigation. we prefer showing the text instead of
html (with inlined images only) version of a message. html messages are shown
in an iframe served from an endpoint with CSP headers to prevent dangerous
resources (scripts, external images) from being loaded. the html is also
sanitized, with javascript removed. a user can choose to load external
resources (e.g. images for tracking purposes).
the frontend is just (strict) typescript, no external frameworks. all
incoming/outgoing data is typechecked, both the api request parameters and
response types, and the data coming in over SSE. the types and checking code
are generated with sherpats, which uses the api definitions generated by
sherpadoc based on the Go code. so types from the backend are automatically
propagated to the frontend. since there is no framework to automatically
propagate properties and rerender components, changes coming in over the SSE
connection are propagated explicitly with regular function calls. the ui is
separated into "views", each with a "root" dom element that is added to the
visible document. these views have additional functions for getting changes
propagated, often resulting in the view updating its (internal) ui state (dom).
we keep the frontend compilation simple, it's just a few typescript files that
get compiled (combined and types stripped) into a single js file, no additional
runtime code needed or complicated build processes used. the webmail is served
is served from a compressed, cachable html file that includes style and the
javascript, currently just over 225kb uncompressed, under 60kb compressed (not
minified, including comments). we include the generated js files in the
repository, to keep Go's easily buildable self-contained binaries.
authentication is basic http, as with the account and admin pages. most data
comes in over one long-term SSE connection to the backend. api requests signal
which mailbox/search/messages are requested over the SSE connection. fetching
individual messages, and making changes, are done through api calls. the
operations are similar to imap, so some code has been moved from package
imapserver to package store. the future jmap implementation will benefit from
these changes too. more functionality will probably be moved to the store
package in the future.
the quickstart enables webmail on the internal listener by default (for new
installs). users can enable it on the public listener if they want to. mox
localserve enables it too. to enable webmail on existing installs, add settings
like the following to the listeners in mox.conf, similar to AccountHTTP(S):
WebmailHTTP:
Enabled: true
WebmailHTTPS:
Enabled: true
special thanks to liesbeth, gerben, andrii for early user feedback.
there is plenty still to do, see the list at the top of webmail/webmail.ts.
feedback welcome as always.
it's still not great. better to automatically change domains.conf. but that
would currently rewrite the whole file, which may not be what admins that
manually edit expect, it would remove their comments. we need better
config-update code.
for issue #40
this doesn't really test the output of the ctl commands, just that they succeed
without error. better than nothing...
testing found two small bugs, that are not an issue in practice:
1. we were ack'ing streamed data from the other side of the ctl connection
before having read it. when there is no buffer space on the connection (always
the case for net.Pipe) that would cause a deadlock. only actually happened
during the new tests.
2. the generated dkim keys are relatively to the directory of the dynamic
config file. mox looked it up relative to the directory of the _static_ config
file at startup. this directory is typicaly the same. users would have noticed
if they had triggered this.
the default transport is still just "direct delivery", where we connect to the
destination domain's MX servers.
other transports are:
- regular smtp without authentication, this is relaying to a smarthost.
- submission with authentication, e.g. to a third party email sending service.
- direct delivery, but with with connections going through a socks proxy. this
can be helpful if your ip is blocked, you need to get email out, and you have
another IP that isn't blocked.
keep in mind that for all of the above, appropriate SPF/DKIM settings have to
be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the
SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM
requirements cannot really be checked.
which transport is used can be configured through routes. routes can be set on
an account, a domain, or globally. the routes are evaluated in that order, with
the first match selecting the transport. these routes are evaluated for each
delivery attempt. common selection criteria are recipient domain and sender
domain, but also which delivery attempt this is. you could configured mox to
attempt sending through a 3rd party from the 4th attempt onwards.
routes and transports are optional. if no route matches, or an empty/zero
transport is selected, normal direct delivery is done.
we could already "submit" emails with 3rd party accounts with "sendmail". but
we now support more SASL authentication mechanisms with SMTP (not only PLAIN,
but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also
supports. sendmail will use the most secure mechanism supported by the server,
or the explicitly configured mechanism.
for issue #36 by dmikushin. also based on earlier discussion on hackernews.
makes it easier to use tls keys/certs managed by other tools, with or without
acme. the root process has access to open such files. the child process reads
the key from the file descriptor, then closes the file.
for issue #30 by inigoserna, thanks!
by specifying a "destination" in an account that is just "@" followed by the
domain, e.g. "@example.org". messages are only delivered to the catchall
address when no regular destination matches (taking the per-domain
catchall-separator and case-sensisitivity into account).
for issue #18
such configurations are certainly errors, but were silently accepted and highly
likely not doing what you may have hoped. i suspect no one has configured mox
this way.
by default 1000 messages per day, and to max 200 first-time receivers.
i don't think a person would reach those limits. a compromised account abused
by spammers could easily reach that limit. this prevents further damage.
the error message you will get is quite clear, pointing to the configuration
parameter that should be changed.
so you can use the host (domain) name of the mail server for serving other
resources too. the default is is still that account is served on /, and so
takes all incoming requests before giving webhandlers a chance.
mox localserve now serves the account pages on /account/
localserve creates a config for listening on localhost for
imap/smtp/submission/http, on port numbers 1000 + the common service port
numbers. all incoming email is accepted (if checks pass), and a few pattern in
localparts are recognized and result in delivery errors.
current behaviour isn't intuitive. it's not great to have to attempt parsing
the strings as both localpart and email address. so we deprecate the
localpart-only behaviour. when we load the config file, and it has
localpart-only Destinations keys, we'll change them to full addresses in
memory. when an admin causes a write of domains.conf, it'll automatically be
fixed. we log an error with a deprecated notice for each localpart-only
destinations key.
sometime in the future, we can remove the old localpart-only destination
support. will be in the release notes then.
also start keeping track of update notes that need to make it in the release
notes of the next release.
for issue #18
the idea is to make it clear from the logging if non-ascii characters are used.
this is implemented by making mlog recognize if a field value that will be
logged has a LogString method. if so, that value is logged. dns.Domain,
smtp.Address, smtp.Localpart, smtp.Path now have a LogString method.
some explicit calls to String have been replaced to LogString, and some %q
formatting have been replaced with %s, because the escaped localpart would
already have double quotes, and double doublequotes aren't easy to read.
- make it easier to run with an existing webserver. the quickstart now has a new option for that, it generates a different mox.conf, and further instructions such as configuring the tls keys/certs and reverse proxy urls. and changes to make autoconfig work in that case too.
- when starting up, request a tls cert for the hostname and for the autoconfig endpoint. the first will be requested soon anyway, and the autoconfig cert is needed early so the first autoconfig request doesn't time out (without helpful message to the user by at least thunderbird). and don't request the certificate before the servers are online. the root process was now requesting the certs, before the child process was serving on the tls port.
- add examples of configs generated by the quickstart.
- enable debug logging in config from quickstart, to give user more info.
for issue #5
- make builtin http handlers serve on specific domains, such as for mta-sts, so
e.g. /.well-known/mta-sts.txt isn't served on all domains.
- add logging of a few more fields in access logging.
- small tweaks/bug fixes in webserver request handling.
- add config option for redirecting entire domains to another (common enough).
- split httpserver metric into two: one for duration until writing header (i.e.
performance of server), another for duration until full response is sent to
client (i.e. performance as perceived by users).
- add admin ui, a new page for managing the configs. after making changes
and hitting "save", the changes take effect immediately. the page itself
doesn't look very well-designed (many input fields, makes it look messy). i
have an idea to improve it (explained in admin.html as todo) by making the
layout look just like the config file. not urgent though.
i've already changed my websites/webapps over.
the idea of adding a webserver is to take away a (the) reason for folks to want
to complicate their mox setup by running an other webserver on the same machine.
i think the current webserver implementation can already serve most common use
cases. with a few more tweaks (feedback needed!) we should be able to get to 95%
of the use cases. the reverse proxy can take care of the remaining 5%.
nevertheless, a next step is still to change the quickstart to make it easier
for folks to run with an existing webserver, with existing tls certs/keys.
that's how this relates to issue #5.
- serve static files, serving index.html or optionally listings for directories
- redirects
- reverse-proxy, forwarding requests to a backend
these are configurable through the config file. a domain and path regexp have to
be configured. path prefixes can be stripped. configured domains are added to
the autotls allowlist, so acme automatically fetches certificates for them.
all webserver requests now have (access) logging, metrics, rate limiting.
on http errors, the error message prints an encrypted cid for relating with log files.
this also adds a new mechanism for example config files.
makes it easier to run on bsd's, where you cannot (easily?) let non-root users
bind to ports <1024. starting as root also paves the way for future improvements
with privilege separation.
unfortunately, this requires changes to how you start mox. though mox will help
by automatically fix up dir/file permissions/ownership.
if you start mox from the systemd unit file, you should update it so it starts
as root and adds a few additional capabilities:
# first update the mox binary, then, as root:
./mox config printservice >mox.service
systemctl daemon-reload
systemctl restart mox
journalctl -f -u mox &
# you should see mox start up, with messages about fixing permissions on dirs/files.
if you used the recommended config/ and data/ directory, in a directory just for
mox, and with the mox user called "mox", this should be enough.
if you don't want mox to modify dir/file permissions, set "NoFixPermissions:
true" in mox.conf.
if you named the mox user something else than mox, e.g. "_mox", add "User: _mox"
to mox.conf.
if you created a shared service user as originally suggested, you may want to
get rid of that as it is no longer useful and may get in the way. e.g. if you
had /home/service/mox with a "service" user, that service user can no longer
access any files: only mox and root can.
this also adds scripts for building mox docker images for alpine-supported
platforms.
the "restart" subcommand has been removed. it wasn't all that useful and got in
the way.
and another change: when adding a domain while mtasts isn't enabled, don't add
the per-domain mtasts config, as it would cause failure to add the domain.
based on report from setting up mox on openbsd from mteege.
and based on issue #3. thanks for the feedback!
the error handling code attempts to collect all error messages it encounters,
to print them all at the end, so you can fix them all before trying again. so
we try to continue preparing the config after an error. in this case, we
continued as if acme.Manager was properly set and nil-dereferenced it.
for issue #2, from hismailbulut
so users can easily take their email out of somewhere else, and import it into mox.
this goes a little way to give feedback as the import progresses: upload
progress is shown (surprisingly, browsers aren't doing this...), imported
mailboxes/messages are counted (batched) and import issues/warnings are
displayed, all sent over an SSE connection. an import token is stored in
sessionstorage. if you reload the page (e.g. after a connection error), the
browser will reconnect to the running import and show its progress again. and
you can just abort the import before it is finished and committed, and nothing
will have changed.
this also imports flags/keywords from mbox files.
ideally both account & admin web pages should be on non-public ips (e.g. a
wireguard tunnel). but during setup, users may not have that set up, and they
may want to configure the admin/account pages on their public ip's. the auth
rate limiting should make it less of issue.
users can now also only put the account web page publicly available. useful for
if you're the admin and you have a vpn connection, but your other/external
users do not have a vpn into your mail server. to make the account page more
easily findable, the http root serves the account page. the admin page is still
at /admin/, to prevent clash with potential account pages, but if no account
page is present, you are helpfully redirected from / to /admin/.
this also adds a prometheus metric counting how often auth attempts have been
rate limited.
the previous default, marking the messages as junk had the interesting effect
of training the junk filter. rejecting could have been the result of the
sending IP being in the DNSBL. so the DNSBL helped to automatically train the
junk filter. perhaps we can keep that in the future and just not take messages
from the rejects mailbox into account when evaluating the reputation for
incoming deliveries.
before, we used heuristics to decide when to train/untrain a message as junk or
nonjunk: the message had to be seen, be in certain mailboxes. then if a message
was marked as junk, it was junk. and otherwise it was nonjunk. this wasn't good
enough: you may want to keep some messages around as neither junk or nonjunk.
and that wasn't possible.
ideally, we would just look at the imap $Junk and $NotJunk flags. the problem
is that mail clients don't set these flags, or don't make it easy. thunderbird
can set the flags based on its own bayesian filter. it has a shortcut for
marking Junk and moving it to the junk folder (good), but the counterpart of
notjunk only marks a message as notjunk without showing in the UI that it was
marked as notjunk. there is also no "move and mark as notjunk" mechanism. e.g.
"archive" does not mark a message as notjunk. ios mail and mutt don't appear to
have any way to see or change the $Junk and $NotJunk flags.
what email clients do have is the ability to move messages to other
mailboxes/folders. so mox now has a mechanism that allows you to configure
mailboxes that automatically set $Junk or $NotJunk (or clear both) when a
message is moved/copied/delivered to that folder. e.g. a mailbox called junk or
spam or rejects marks its messags as junk. inbox, postmaster, dmarc, tlsrpt,
neutral* mark their messages as neither junk or notjunk. other folders mark
their messages as notjunk. e.g. list/*, archive. this functionality is
optional, but enabled with the quickstart and for new accounts.
also, mox now keeps track of the previous training of a message and will only
untrain/train if needed. before, there probably have been duplicate or missing
(un)trainings.
this also includes a new subcommand "retrain" to recreate the junkfilter for an
account. you should run it after updating to this version. and you should
probably also modify your account config to include the AutomaticJunkFlags.
for example, by matching incoming messags on smtp mail from, verified domains
(spf/dkim), headers. then delivering to a configured mailbox. for mailing
lists, if a verified domain matches, regular spam checks can be skipped.
this was already possible by editing the configuration file, but only admins
can edit that file. now users can manage their own rulesets.
limiting is done based on remote ip's, with 3 ip mask variants to limit networks
of machines. often with two windows, enabling short bursts of activity, but not
sustained high activity. currently only for imap and smtp, not yet http.
limits are currently based on:
- number of open connections
- connection rate
- limits after authentication failures. too many failures, and new connections will be dropped.
- rate of delivery in total number of messages
- rate of delivery in total size of messages
the limits on connections and authentication failures are in-memory. the limits
on delivery of messages are based on stored messages.
the limits themselves are not yet configurable, let's use this first.
in the future, we may also want to have stricter limits for senders without any
reputation.
- make the example commands in the readme more likely to succeed, especially
for people who are not familiar with go and its toolchain.
- improve probability that the correct configuration is generated, especially
the hostname. previously, if the quickstart email address was "some.domain",
and the machine where you ran the quickstart was "myhost", the hostname used
for the configuration was assumed to be "myhost.some.domain". but this is often
not correct, especially when configuring mox to serve mail on a subdomain of an
existing domain. mox will now try to determine the host name by a reverse
lookup of the public ips it found. and it will warn if there are no/multiple
candidates.
based on feedback from erik dubbelboer, thanks!
and change thunderbird autoconfiguration to use it.
unfortunately, for microsoft autodiscover, there appears to be no way to
request secure password negotiation. so it will default to plain text auth.
cram-md5 is less secure than scram-sha-*, but thunderbird does not yet support
scram auth. it currently chooses "plain", sending the literal password over the
connection (which is TLS-protected, but we don't want to receive clear text
passwords). in short, cram-md5 is better than nothing...
for cram-md5 to work, a new set of derived credentials need to be stored in the
database. so you need to save your password again to make it work. this was
also the case with the scram-sha-1 addition, but i forgot to mention it then.
- and don't have a global variable "d" in the big checkDomain function in http/admin.go.
- and set loglevel from command-line flag again after loading the config file, for all subcommands except "serve".
