2023-01-30 16:43:50 +03:00
|
|
|
Mox is a modern full-featured open source secure mail server for low-maintenance self-hosted email.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
For more details, see the mox website, https://www.xmox.nl.
|
|
|
|
|
|
2023-01-30 16:27:06 +03:00
|
|
|
See Quickstart below to get started.
|
|
|
|
|
|
2023-02-25 15:38:07 +03:00
|
|
|
## Features
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-02-10 21:43:02 +03:00
|
|
|
- Quick and easy to start/maintain mail server, for your own domain(s).
|
new feature: when delivering messages from the queue, make it possible to use a "transport"
the default transport is still just "direct delivery", where we connect to the
destination domain's MX servers.
other transports are:
- regular smtp without authentication, this is relaying to a smarthost.
- submission with authentication, e.g. to a third party email sending service.
- direct delivery, but with with connections going through a socks proxy. this
can be helpful if your ip is blocked, you need to get email out, and you have
another IP that isn't blocked.
keep in mind that for all of the above, appropriate SPF/DKIM settings have to
be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the
SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM
requirements cannot really be checked.
which transport is used can be configured through routes. routes can be set on
an account, a domain, or globally. the routes are evaluated in that order, with
the first match selecting the transport. these routes are evaluated for each
delivery attempt. common selection criteria are recipient domain and sender
domain, but also which delivery attempt this is. you could configured mox to
attempt sending through a 3rd party from the 4th attempt onwards.
routes and transports are optional. if no route matches, or an empty/zero
transport is selected, normal direct delivery is done.
we could already "submit" emails with 3rd party accounts with "sendmail". but
we now support more SASL authentication mechanisms with SMTP (not only PLAIN,
but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also
supports. sendmail will use the most secure mechanism supported by the server,
or the explicitly configured mechanism.
for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
|
|
|
- SMTP (with extensions) for receiving, submitting and delivering email.
|
2023-02-16 12:09:33 +03:00
|
|
|
- IMAP4 (with extensions) for giving email clients access to email.
|
2023-09-24 13:34:46 +03:00
|
|
|
- Webmail for reading/sending email from the browser.
|
2023-11-01 19:55:40 +03:00
|
|
|
- SPF/DKIM/DMARC for authenticating messages/delivery, also DMARC aggregate
|
|
|
|
|
reports.
|
2023-09-24 13:34:46 +03:00
|
|
|
- Reputation tracking, learning (per user) host-, domain- and
|
|
|
|
|
sender address-based reputation from (Non-)Junk email classification.
|
2023-01-30 16:27:06 +03:00
|
|
|
- Bayesian spam filtering that learns (per user) from (Non-)Junk email.
|
2023-02-08 23:45:32 +03:00
|
|
|
- Slowing down senders with no/low reputation or questionable email content
|
|
|
|
|
(similar to greylisting). Rejected emails are stored in a mailbox called Rejects
|
2023-02-10 21:43:02 +03:00
|
|
|
for a short period, helping with misclassified legitimate synchronous
|
2023-01-30 16:27:06 +03:00
|
|
|
signup/login/transactional emails.
|
2023-09-24 13:34:46 +03:00
|
|
|
- Internationalized email, with unicode in email address usernames
|
|
|
|
|
("localparts"), and in domain names (IDNA).
|
|
|
|
|
- Automatic TLS with ACME, for use with Let's Encrypt and other CA's.
|
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery
the vendored dns resolver code is a copy of the go stdlib dns resolver, with
awareness of the "authentic data" (i.e. dnssec secure) added, as well as support
for enhanced dns errors, and looking up tlsa records (for dane). ideally it
would be upstreamed, but the chances seem slim.
dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their
dnssec status is added to the Received message headers for incoming email.
but the main reason to add dnssec was for implementing dane. with dane, the
verification of tls certificates can be done through certificates/public keys
published in dns (in the tlsa records). this only makes sense (is trustworthy)
if those dns records can be verified to be authentic.
mox now applies dane to delivering messages over smtp. mox already implemented
mta-sts for webpki/pkix-verification of certificates against the (large) pool
of CA's, and still enforces those policies when present. but it now also checks
for dane records, and will verify those if present. if dane and mta-sts are
both absent, the regular opportunistic tls with starttls is still done. and the
fallback to plaintext is also still done.
mox also makes it easy to setup dane for incoming deliveries, so other servers
can deliver with dane tls certificate verification. the quickstart now
generates private keys that are used when requesting certificates with acme.
the private keys are pre-generated because they must be static and known during
setup, because their public keys must be published in tlsa records in dns.
autocert would generate private keys on its own, so had to be forked to add the
option to provide the private key when requesting a new certificate. hopefully
upstream will accept the change and we can drop the fork.
with this change, using the quickstart to setup a new mox instance, the checks
at internet.nl result in a 100% score, provided the domain is dnssec-signed and
the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
|
|
|
- DANE and MTA-STS for inbound and outbound delivery over SMTP with STARTTLS,
|
implement outgoing tls reports
we were already accepting, processing and displaying incoming tls reports. now
we start tracking TLS connection and security-policy-related errors for
outgoing message deliveries as well. we send reports once a day, to the
reporting addresses specified in TLSRPT records (rua) of a policy domain. these
reports are about MTA-STS policies and/or DANE policies, and about
STARTTLS-related failures.
sending reports is enabled by default, but can be disabled through setting
NoOutgoingTLSReports in mox.conf.
only at the end of the implementation process came the realization that the
TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy
for the recipient domain, and that MTA-STS and DANE TLS/policy results are
typically delivered in separate reports. so MX hosts need their own TLSRPT
policies.
config for the per-host TLSRPT policy should be added to mox.conf for existing
installs, in field HostTLSRPT. it is automatically configured by quickstart for
new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin
pages now suggest the per-host TLSRPT record. by creating that record, you're
requesting TLS reports about your MX host.
gathering all the TLS/policy results is somewhat tricky. the tentacles go
throughout the code. the positive result is that the TLS/policy-related code
had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect
reality better, with independent settings about whether PKIX and/or DANE
verification has to be done, and/or whether verification errors have to be
ignored (e.g. for tls-required: no header). also, cached mtasts policies of
mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
|
|
|
including REQUIRETLS and with incoming/outgoing TLSRPT reporting.
|
2024-04-24 20:15:30 +03:00
|
|
|
- Web admin interface that helps you set up your domains, accounts and list
|
|
|
|
|
aliases (instructions to create DNS records, configure
|
|
|
|
|
SPF/DKIM/DMARC/TLSRPT/MTA-STS), for status information, and modifying the
|
|
|
|
|
configuration file.
|
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery
the vendored dns resolver code is a copy of the go stdlib dns resolver, with
awareness of the "authentic data" (i.e. dnssec secure) added, as well as support
for enhanced dns errors, and looking up tlsa records (for dane). ideally it
would be upstreamed, but the chances seem slim.
dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their
dnssec status is added to the Received message headers for incoming email.
but the main reason to add dnssec was for implementing dane. with dane, the
verification of tls certificates can be done through certificates/public keys
published in dns (in the tlsa records). this only makes sense (is trustworthy)
if those dns records can be verified to be authentic.
mox now applies dane to delivering messages over smtp. mox already implemented
mta-sts for webpki/pkix-verification of certificates against the (large) pool
of CA's, and still enforces those policies when present. but it now also checks
for dane records, and will verify those if present. if dane and mta-sts are
both absent, the regular opportunistic tls with starttls is still done. and the
fallback to plaintext is also still done.
mox also makes it easy to setup dane for incoming deliveries, so other servers
can deliver with dane tls certificate verification. the quickstart now
generates private keys that are used when requesting certificates with acme.
the private keys are pre-generated because they must be static and known during
setup, because their public keys must be published in tlsa records in dns.
autocert would generate private keys on its own, so had to be forked to add the
option to provide the private key when requesting a new certificate. hopefully
upstream will accept the change and we can drop the fork.
with this change, using the quickstart to setup a new mox instance, the checks
at internet.nl result in a 100% score, provided the domain is dnssec-signed and
the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
|
|
|
- Account autodiscovery (with SRV records, Microsoft-style, Thunderbird-style,
|
|
|
|
|
and Apple device management profiles) for easy account setup (though client
|
|
|
|
|
support is limited).
|
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config
- make builtin http handlers serve on specific domains, such as for mta-sts, so
e.g. /.well-known/mta-sts.txt isn't served on all domains.
- add logging of a few more fields in access logging.
- small tweaks/bug fixes in webserver request handling.
- add config option for redirecting entire domains to another (common enough).
- split httpserver metric into two: one for duration until writing header (i.e.
performance of server), another for duration until full response is sent to
client (i.e. performance as perceived by users).
- add admin ui, a new page for managing the configs. after making changes
and hitting "save", the changes take effect immediately. the page itself
doesn't look very well-designed (many input fields, makes it look messy). i
have an idea to improve it (explained in admin.html as todo) by making the
layout look just like the config file. not urgent though.
i've already changed my websites/webapps over.
the idea of adding a webserver is to take away a (the) reason for folks to want
to complicate their mox setup by running an other webserver on the same machine.
i think the current webserver implementation can already serve most common use
cases. with a few more tweaks (feedback needed!) we should be able to get to 95%
of the use cases. the reverse proxy can take care of the remaining 5%.
nevertheless, a next step is still to change the quickstart to make it easier
for folks to run with an existing webserver, with existing tls certs/keys.
that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
|
|
|
- Webserver with serving static files and forwarding requests (reverse
|
|
|
|
|
proxy), so port 443 can also be used to serve websites.
|
add a webapi and webhooks for a simple http/json-based api
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.
this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.
unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...
matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.
a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.
messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.
suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.
submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".
to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.
admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.
new config options have been introduced. they are editable through the admin
and/or account web interfaces.
the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.
gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.
for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
|
|
|
- Simple HTTP/JSON API for sending transaction email and receiving delivery
|
|
|
|
|
events and incoming messages (webapi and webhooks).
|
2023-01-30 16:27:06 +03:00
|
|
|
- Prometheus metrics and structured logging for operational insight.
|
2023-09-24 13:34:46 +03:00
|
|
|
- "mox localserve" subcommand for running mox locally for email-related
|
2023-03-12 17:16:01 +03:00
|
|
|
testing/developing, including pedantic mode.
|
2023-12-11 17:01:56 +03:00
|
|
|
- Most non-server Go packages mox consists of are written to be reusable.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-02-25 15:38:07 +03:00
|
|
|
Mox is available under the MIT-license and was created by Mechiel Lukkien,
|
2023-09-23 13:05:40 +03:00
|
|
|
mechiel@ueber.net. Mox includes BSD-3-claused code from the Go Authors, and the
|
|
|
|
|
Public Suffix List by Mozilla under Mozilla Public License, v2.0.
|
2023-02-11 15:35:07 +03:00
|
|
|
|
2023-09-24 13:42:19 +03:00
|
|
|
Mox has automated tests, including for interoperability with Postfix for SMTP.
|
|
|
|
|
Mox is manually tested with email clients: Mozilla Thunderbird, mutt, iOS Mail,
|
|
|
|
|
macOS Mail, Android Mail, Microsoft Outlook. Mox is also manually tested to
|
|
|
|
|
interoperate with popular cloud providers: gmail.com, outlook.com, yahoo.com,
|
|
|
|
|
proton.me.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-09-24 13:42:19 +03:00
|
|
|
The code is heavily cross-referenced with the RFCs for readability/maintainability.
|
2023-02-24 16:16:51 +03:00
|
|
|
|
2023-01-30 16:27:06 +03:00
|
|
|
# Quickstart
|
|
|
|
|
|
|
|
|
|
The easiest way to get started with serving email for your domain is to get a
|
2024-01-10 18:48:53 +03:00
|
|
|
(virtual) machine dedicated to serving email, name it `[host].[domain]` (e.g.
|
|
|
|
|
mail.example.com). Having a DNSSEC-verifying resolver installed, such as
|
|
|
|
|
unbound, is highly recommended. Run as root:
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-03-04 02:49:02 +03:00
|
|
|
# Create mox user and homedir (or pick another name or homedir):
|
|
|
|
|
useradd -m -d /home/mox mox
|
|
|
|
|
|
|
|
|
|
cd /home/mox
|
2023-09-24 13:42:19 +03:00
|
|
|
... compile or download mox to this directory, see below ...
|
2023-03-04 02:49:02 +03:00
|
|
|
|
|
|
|
|
# Generate config files for your address/domain:
|
2023-01-30 16:27:06 +03:00
|
|
|
./mox quickstart you@example.com
|
|
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
The quickstart:
|
|
|
|
|
|
|
|
|
|
- Creates configuration files mox.conf and domains.conf.
|
|
|
|
|
- Adds the domain and an account for the email address to domains.conf
|
|
|
|
|
- Generates an admin and account password.
|
|
|
|
|
- Prints the DNS records you need to add, for the machine and domain.
|
|
|
|
|
- Prints commands to start mox, and optionally install mox as a service.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-09-24 13:34:46 +03:00
|
|
|
A machine that doesn't already run a webserver is highly recommended because
|
2024-01-10 18:48:53 +03:00
|
|
|
modern email requires HTTPS, and mox currently needs to run a webserver for
|
|
|
|
|
automatic TLS with ACME. You could combine mox with an existing webserver, but
|
|
|
|
|
it requires a lot more configuration. If you want to serve websites on the same
|
|
|
|
|
machine, consider using the webserver built into mox. It's pretty good! If you
|
|
|
|
|
want to run an existing webserver on port 443/80, see `mox help quickstart`.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
|
|
|
|
After starting, you can access the admin web interface on internal IPs.
|
|
|
|
|
|
2023-09-24 13:42:19 +03:00
|
|
|
# Download
|
|
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
Download a mox binary from
|
|
|
|
|
https://beta.gobuilds.org/github.com/mjl-/mox@latest/linux-amd64-latest/.
|
2023-09-24 13:42:19 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
Symlink or rename it to "mox".
|
|
|
|
|
|
|
|
|
|
The URL above always resolves to the latest release for linux/amd64 built with
|
|
|
|
|
the latest Go toolchain. See the links at the bottom of that page for binaries
|
|
|
|
|
for other platforms.
|
2023-09-24 13:42:19 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
# Compiling
|
2023-09-24 13:42:19 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
You can easily (cross) compile mox yourself. You need a recent Go toolchain
|
2024-08-22 17:14:26 +03:00
|
|
|
installed. Run `go version`, it must be >= 1.22. Download the latest version
|
2024-01-10 18:48:53 +03:00
|
|
|
from https://go.dev/dl/ or see https://go.dev/doc/manage-install.
|
2023-09-24 13:42:19 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
To download the source code of the latest release, and compile it to binary "mox":
|
|
|
|
|
|
|
|
|
|
GOBIN=$PWD CGO_ENABLED=0 go install github.com/mjl-/mox@latest
|
2023-09-24 13:42:19 +03:00
|
|
|
|
make mox compile on windows, without "mox serve" but with working "mox localserve"
getting mox to compile required changing code in only a few places where
package "syscall" was used: for accessing file access times and for umask
handling. an open problem is how to start a process as an unprivileged user on
windows. that's why "mox serve" isn't implemented yet. and just finding a way
to implement it now may not be good enough in the near future: we may want to
starting using a more complete privilege separation approach, with a process
handling sensitive tasks (handling private keys, authentication), where we may
want to pass file descriptors between processes. how would that work on
windows?
anyway, getting mox to compile for windows doesn't mean it works properly on
windows. the largest issue: mox would normally open a file, rename or remove
it, and finally close it. this happens during message delivery. that doesn't
work on windows, the rename/remove would fail because the file is still open.
so this commit swaps many "remove" and "close" calls. renames are a longer
story: message delivery had two ways to deliver: with "consuming" the
(temporary) message file (which would rename it to its final destination), and
without consuming (by hardlinking the file, falling back to copying). the last
delivery to a recipient of a message (and the only one in the common case of a
single recipient) would consume the message, and the earlier recipients would
not. during delivery, the already open message file was used, to parse the
message. we still want to use that open message file, and the caller now stays
responsible for closing it, but we no longer try to rename (consume) the file.
we always hardlink (or copy) during delivery (this works on windows), and the
caller is responsible for closing and removing (in that order) the original
temporary file. this does cost one syscall more. but it makes the delivery code
(responsibilities) a bit simpler.
there is one more obvious issue: the file system path separator. mox already
used the "filepath" package to join paths in many places, but not everywhere.
and it still used strings with slashes for local file access. with this commit,
the code now uses filepath.FromSlash for path strings with slashes, uses
"filepath" in a few more places where it previously didn't. also switches from
"filepath" to regular "path" package when handling mailbox names in a few
places, because those always use forward slashes, regardless of local file
system conventions. windows can handle forward slashes when opening files, so
test code that passes path strings with forward slashes straight to go stdlib
file i/o functions are left unchanged to reduce code churn. the regular
non-test code, or test code that uses path strings in places other than
standard i/o functions, does have the paths converted for consistent paths
(otherwise we would end up with paths with mixed forward/backward slashes in
log messages).
windows cannot dup a listening socket. for "mox localserve", it isn't
important, and we can work around the issue. the current approach for "mox
serve" (forking a process and passing file descriptors of listening sockets on
"privileged" ports) won't work on windows. perhaps it isn't needed on windows,
and any user can listen on "privileged" ports? that would be welcome.
on windows, os.Open cannot open a directory, so we cannot call Sync on it after
message delivery. a cursory internet search indicates that directories cannot
be synced on windows. the story is probably much more nuanced than that, with
long deep technical details/discussions/disagreement/confusion, like on unix.
for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
|
|
|
Mox only compiles for and fully works on unix systems. Mox also compiles for
|
|
|
|
|
Windows, but "mox serve" does not yet work, though "mox localserve" (for a
|
|
|
|
|
local test instance) and most other subcommands do. Mox does not compile for
|
|
|
|
|
Plan 9.
|
2023-09-24 13:42:19 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
# Docker
|
|
|
|
|
|
|
|
|
|
Although not recommended, you can also run mox with docker image
|
|
|
|
|
`r.xmox.nl/mox`, with tags like `v0.0.1` and `v0.0.1-go1.20.1-alpine3.17.2`, see
|
|
|
|
|
https://r.xmox.nl/r/mox/. See
|
|
|
|
|
https://github.com/mjl-/mox/blob/main/docker-compose.yml to get started.
|
|
|
|
|
|
|
|
|
|
New docker images aren't (automatically) generated for new Go runtime/compile
|
|
|
|
|
releases.
|
|
|
|
|
|
|
|
|
|
It is important to run with docker host networking, so mox can use the public
|
|
|
|
|
IPs and has correct remote IP information for incoming connections (important
|
|
|
|
|
for junk filtering and rate-limiting).
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-02-25 15:38:07 +03:00
|
|
|
# Future/development
|
|
|
|
|
|
2024-03-26 11:16:33 +03:00
|
|
|
See develop.txt for instructions/tips for developing on mox.
|
|
|
|
|
|
2023-09-21 17:08:43 +03:00
|
|
|
Mox will receive funding for essentially full-time continued work from August
|
|
|
|
|
2023 to August 2024 through NLnet/EU's NGI0 Entrust, see
|
|
|
|
|
https://nlnet.nl/project/Mox/.
|
|
|
|
|
|
2023-02-25 15:38:07 +03:00
|
|
|
## Roadmap
|
|
|
|
|
|
2023-09-24 13:34:46 +03:00
|
|
|
- Calendaring with CalDAV/iCal
|
|
|
|
|
- More IMAP extensions (PREVIEW, WITHIN, IMPORTANT, COMPRESS=DEFLATE,
|
|
|
|
|
CREATE-SPECIAL-USE, SAVEDATE, UNAUTHENTICATE, REPLACE, QUOTA, NOTIFY,
|
queue: deliver to multiple recipients in a single smtp transaction
transferring the data only once. we only do this when the recipient domains
are the same. when queuing, we now take care to set the same NextAttempt
timestamp, so queued messages are actually eligable for combined delivery.
this adds a DeliverMultiple to the smtp client. for pipelined requests, it will
send all RCPT TO (and MAIL and DATA) in one go, and handles the various
responses and error conditions, returning either an overal error, or per
recipient smtp responses. the results of the smtp LIMITS extension are also
available in the smtp client now.
this also takes the "LIMITS RCPTMAX" smtp extension into account: if the server
only accepts a single recipient, we won't send multiple.
if a server doesn't announce a RCPTMAX limit, but still has one (like mox does
for non-spf-verified transactions), we'll recognize code 452 and 552 (for
historic reasons) as temporary error, and try again in a separate transaction
immediately after. we don't yet implement "LIMITS MAILMAX", doesn't seem likely
in practice.
2024-03-07 12:07:53 +03:00
|
|
|
MULTIAPPEND, OBJECTID, MULTISEARCH, THREAD, SORT)
|
add a webapi and webhooks for a simple http/json-based api
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.
this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.
unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...
matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.
a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.
messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.
suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.
submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".
to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.
admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.
new config options have been introduced. they are editable through the admin
and/or account web interfaces.
the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.
gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.
for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
|
|
|
- SMTP DSN extension
|
2024-05-09 11:51:11 +03:00
|
|
|
- "mox setup" command, with webapp for interactive setup
|
|
|
|
|
- Introbox, to which first-time senders are delivered
|
2023-09-24 13:34:46 +03:00
|
|
|
- ARC, with forwarded email from trusted source
|
|
|
|
|
- Forwarding (to an external address)
|
2023-02-25 15:38:07 +03:00
|
|
|
- Add special IMAP mailbox ("Queue?") that contains queued but
|
2023-12-11 17:01:56 +03:00
|
|
|
undelivered messages, updated with IMAP flags/keywords/tags and message headers.
|
2024-04-24 20:15:30 +03:00
|
|
|
- External addresses in aliases/lists.
|
2023-09-24 13:34:46 +03:00
|
|
|
- Autoresponder (out of office/vacation)
|
|
|
|
|
- OAUTH2 support, for single sign on
|
2024-04-29 22:10:25 +03:00
|
|
|
- IMAP extensions for "online"/non-syncing/webmail clients (SORT (including
|
|
|
|
|
DISPLAYFROM, DISPLAYTO), THREAD, PARTIAL, CONTEXT=SEARCH CONTEXT=SORT ESORT,
|
|
|
|
|
FILTERS)
|
|
|
|
|
- Improve support for mobile clients with extensions: IMAP URLAUTH, SMTP
|
|
|
|
|
CHUNKING and BINARYMIME, IMAP CATENATE
|
|
|
|
|
- Mailing list manager
|
2023-07-24 12:03:53 +03:00
|
|
|
- Privilege separation, isolating parts of the application to more restricted
|
|
|
|
|
sandbox (e.g. new unauthenticated connections)
|
|
|
|
|
- Using mox as backup MX
|
2023-02-27 17:04:53 +03:00
|
|
|
- JMAP
|
2024-04-29 22:10:25 +03:00
|
|
|
- Sieve for filtering (for now see Rulesets in the account config)
|
2023-07-24 12:03:53 +03:00
|
|
|
- Milter support, for integration with external tools
|
2023-09-24 13:34:46 +03:00
|
|
|
- IMAP Sieve extension, to run Sieve scripts after message changes (not only
|
|
|
|
|
new deliveries)
|
2023-02-25 15:38:07 +03:00
|
|
|
|
|
|
|
|
There are many smaller improvements to make as well, search for "todo" in the code.
|
|
|
|
|
|
2023-09-24 13:34:46 +03:00
|
|
|
## Not supported/planned
|
2023-02-25 15:38:07 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
There is currently no plan to implement the following. Though this may
|
|
|
|
|
change in the future.
|
2023-02-25 15:38:07 +03:00
|
|
|
|
|
|
|
|
- Functioning as SMTP relay
|
|
|
|
|
- POP3
|
|
|
|
|
- Delivery to (unix) OS system users
|
|
|
|
|
- Support for pluggable delivery mechanisms
|
2023-06-22 22:27:52 +03:00
|
|
|
- iOS Mail push notifications (with XAPPLEPUSHSERVICE undocumented imap
|
2023-05-31 11:24:48 +03:00
|
|
|
extension and hard to get APNS certificate)
|
2023-02-25 15:38:07 +03:00
|
|
|
|
|
|
|
|
|
2023-01-30 16:27:06 +03:00
|
|
|
# FAQ - Frequently Asked Questions
|
|
|
|
|
|
2023-02-11 15:35:07 +03:00
|
|
|
## Why a new mail server implementation?
|
2023-01-30 16:27:06 +03:00
|
|
|
|
|
|
|
|
Mox aims to make "running a mail server" easy and nearly effortless. Excellent
|
2023-09-24 13:34:46 +03:00
|
|
|
quality (open source) mail server software exists, but getting a working setup
|
|
|
|
|
typically requires you configure half a dozen services (SMTP, IMAP,
|
|
|
|
|
SPF/DKIM/DMARC, spam filtering), which are often written in C (where small bugs
|
|
|
|
|
often have large consequences). That seems to lead to people no longer running
|
|
|
|
|
their own mail servers, instead switching to one of the few centralized email
|
|
|
|
|
providers. Email with SMTP is a long-time decentralized messaging protocol. To
|
|
|
|
|
keep it decentralized, people need to run their own mail server. Mox aims to
|
|
|
|
|
make that easy.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-02-11 15:35:07 +03:00
|
|
|
## Where is the documentation?
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
To keep mox as a project maintainable, documentation is integrated into, and
|
|
|
|
|
generated from the code.
|
|
|
|
|
|
|
|
|
|
A list of mox commands, and their help output, are at
|
|
|
|
|
https://www.xmox.nl/commands/.
|
2023-09-24 13:34:46 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
Mox is configured through configuration files, and each field comes with
|
|
|
|
|
documentation. See https://www.xmox.nl/config/ for config files containing all
|
|
|
|
|
fields and their documentation.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-01-30 16:43:50 +03:00
|
|
|
You can get the same information by running "mox" without arguments to list its
|
2023-02-10 21:43:02 +03:00
|
|
|
subcommands and usage, and "mox help [subcommand]" for more details.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-01-30 16:43:50 +03:00
|
|
|
The example config files are printed by "mox config describe-static" and "mox
|
|
|
|
|
config describe-dynamic".
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2024-01-10 18:48:53 +03:00
|
|
|
If you're missing some documentation, please create an issue describing what is
|
|
|
|
|
unclear or confusing, and we'll try to improve the documentation.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2024-01-01 20:30:31 +03:00
|
|
|
## Is Mox affected by SMTP smuggling?
|
|
|
|
|
|
|
|
|
|
Mox itself is not affected: it only treats "\r\n.\r\n" as SMTP end-of-message.
|
|
|
|
|
But read on for caveats.
|
|
|
|
|
|
|
|
|
|
SMTP smuggling exploits differences in handling by SMTP servers of: carriage
|
|
|
|
|
returns (CR, or "\r"), newlines (line feeds, LF, "\n") in the context of "dot
|
|
|
|
|
stuffing". SMTP is a text-based protocol. An SMTP transaction to send a
|
|
|
|
|
message is finalized with a "\r\n.\r\n" sequence. This sequence could occur in
|
|
|
|
|
the message being transferred, so any verbatim "." at the start of a line in a
|
|
|
|
|
message is "escaped" with another dot ("dot stuffing"), to not trigger the SMTP
|
|
|
|
|
end-of-message. SMTP smuggling takes advantage of bugs in some mail servers
|
|
|
|
|
that interpret other sequences than "\r\n.\r\n" as SMTP end-of-message. For
|
|
|
|
|
example "\n.\n" or even "\r.\r", and perhaps even other magic character
|
|
|
|
|
combinations.
|
|
|
|
|
|
|
|
|
|
Before v0.0.9, mox accepted SMTP transactions with bare carriage returns
|
|
|
|
|
(without newline) for compatibility with real-world email messages, considering
|
|
|
|
|
them meaningless and therefore innocuous.
|
|
|
|
|
|
|
|
|
|
Since v0.0.9, SMTP transactions with bare carriage returns are rejected.
|
|
|
|
|
Sending messages with bare carriage returns to buggy mail servers can cause
|
|
|
|
|
those mail servers to materialize non-existent messages. Now that mox rejects
|
|
|
|
|
messages with bare carriage returns, sending a message through mox can no
|
|
|
|
|
longer be used to trigger those bugs.
|
|
|
|
|
|
|
|
|
|
Mox can still handle bare carriage returns in email messages, e.g. those
|
|
|
|
|
imported from mbox files or Maildirs, or from messages added over IMAP. Mox
|
|
|
|
|
still fixes up messages with bare newlines by adding the missing carriage
|
|
|
|
|
returns.
|
|
|
|
|
|
|
|
|
|
Before v0.0.9, an SMTP transaction for a message containing "\n.\n" would
|
|
|
|
|
result in a non-specific error message, and "\r\n.\n" would result in the dot
|
|
|
|
|
being dropped. Since v0.0.9, these sequences are rejected with a message
|
|
|
|
|
mentioning SMTP smuggling.
|
|
|
|
|
|
2023-02-11 15:35:07 +03:00
|
|
|
## How do I import/export email?
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-02-16 11:57:27 +03:00
|
|
|
Use the import functionality on the accounts web page to import a zip/tgz with
|
|
|
|
|
maildirs/mbox files, or use the "mox import maildir" or "mox import mbox"
|
|
|
|
|
subcommands. You could also use your IMAP email client, add your mox account,
|
|
|
|
|
and copy or move messages from one account to the other.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-02-16 11:57:27 +03:00
|
|
|
Similarly, see the export functionality on the accounts web page and the "mox
|
|
|
|
|
export maildir" and "mox export mbox" subcommands to export email.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-07-24 12:00:11 +03:00
|
|
|
Importing large mailboxes may require a lot of memory (a limitation of the
|
2023-09-24 13:34:46 +03:00
|
|
|
current database). Splitting up mailboxes in smaller parts (e.g. 100k messages)
|
|
|
|
|
would help.
|
2023-07-24 12:00:11 +03:00
|
|
|
|
2023-02-11 15:35:07 +03:00
|
|
|
## How can I help?
|
2023-01-30 16:27:06 +03:00
|
|
|
|
|
|
|
|
Mox needs users and testing in real-life setups! So just give it a try, send
|
|
|
|
|
and receive emails through it with your favourite email clients, and file an
|
|
|
|
|
issue if you encounter a problem or would like to see a feature/functionality
|
|
|
|
|
implemented.
|
|
|
|
|
|
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config
- make builtin http handlers serve on specific domains, such as for mta-sts, so
e.g. /.well-known/mta-sts.txt isn't served on all domains.
- add logging of a few more fields in access logging.
- small tweaks/bug fixes in webserver request handling.
- add config option for redirecting entire domains to another (common enough).
- split httpserver metric into two: one for duration until writing header (i.e.
performance of server), another for duration until full response is sent to
client (i.e. performance as perceived by users).
- add admin ui, a new page for managing the configs. after making changes
and hitting "save", the changes take effect immediately. the page itself
doesn't look very well-designed (many input fields, makes it look messy). i
have an idea to improve it (explained in admin.html as todo) by making the
layout look just like the config file. not urgent though.
i've already changed my websites/webapps over.
the idea of adding a webserver is to take away a (the) reason for folks to want
to complicate their mox setup by running an other webserver on the same machine.
i think the current webserver implementation can already serve most common use
cases. with a few more tweaks (feedback needed!) we should be able to get to 95%
of the use cases. the reverse proxy can take care of the remaining 5%.
nevertheless, a next step is still to change the quickstart to make it easier
for folks to run with an existing webserver, with existing tls certs/keys.
that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
|
|
|
Instead of switching email for your domain over to mox, you could simply
|
2023-02-10 21:43:02 +03:00
|
|
|
configure mox for a subdomain, e.g. [you]@moxtest.[yourdomain].
|
2023-01-30 16:27:06 +03:00
|
|
|
|
|
|
|
|
If you have experience with how the email protocols are used in the wild, e.g.
|
|
|
|
|
compatibility issues, limitations, anti-spam measures, specification
|
|
|
|
|
violations, that would be interesting to hear about.
|
|
|
|
|
|
|
|
|
|
Pull requests for bug fixes and new code are welcome too. If the changes are
|
2023-09-24 13:34:46 +03:00
|
|
|
large, it helps to start a discussion (create an "issue") before doing all the
|
|
|
|
|
work. In practice, starting with a small contribution and growing from there has
|
|
|
|
|
the highest chance of success.
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-03-02 22:22:26 +03:00
|
|
|
By contributing (e.g. code), you agree your contributions are licensed under the
|
|
|
|
|
MIT license (like mox), and have the rights to do so.
|
|
|
|
|
|
2023-02-25 15:38:07 +03:00
|
|
|
## Where can I discuss mox?
|
|
|
|
|
|
2024-10-18 18:05:59 +03:00
|
|
|
Join #mox on irc.oftc.net, or #mox:matrix.org (https://matrix.to/#/#mox:matrix.org),
|
|
|
|
|
or #mox on the "Gopher slack".
|
2023-02-25 15:38:07 +03:00
|
|
|
|
|
|
|
|
For bug reports, please file an issue at https://github.com/mjl-/mox/issues/new.
|
|
|
|
|
|
2023-02-11 15:35:07 +03:00
|
|
|
## How do I change my password?
|
2023-01-30 16:27:06 +03:00
|
|
|
|
|
|
|
|
Regular users (doing IMAP/SMTP with authentication) can change their password
|
2024-01-10 18:48:53 +03:00
|
|
|
at the account page, e.g. `http://localhost/`. Or you can set a password with "mox
|
2023-02-16 11:57:27 +03:00
|
|
|
setaccountpassword".
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-03-20 14:49:40 +03:00
|
|
|
The admin can change the password of any account through the admin page, at
|
2024-01-10 18:48:53 +03:00
|
|
|
`http://localhost/admin/` by default (leave username empty when logging in).
|
2023-03-20 14:49:40 +03:00
|
|
|
|
2023-09-24 13:34:46 +03:00
|
|
|
The account and admin pages are served on localhost for configs created with
|
|
|
|
|
the quickstart. To access these from your browser, run
|
2023-03-20 14:49:40 +03:00
|
|
|
`ssh -L 8080:localhost:80 you@yourmachine` locally and open
|
2024-01-10 18:48:53 +03:00
|
|
|
`http://localhost:8080/[...]`.
|
2023-03-20 14:49:40 +03:00
|
|
|
|
2023-01-30 16:27:06 +03:00
|
|
|
The admin password can be changed with "mox setadminpassword".
|
|
|
|
|
|
2023-02-11 15:35:07 +03:00
|
|
|
## How do I configure a second mox instance as a backup MX?
|
2023-01-30 16:27:06 +03:00
|
|
|
|
|
|
|
|
Unfortunately, mox does not yet provide an option for that. Mox does spam
|
|
|
|
|
filtering based on reputation of received messages. It will take a good amount
|
|
|
|
|
of work to share that information with a backup MX. Without that information,
|
2023-09-24 13:34:46 +03:00
|
|
|
spammers could use a backup MX to get their spam accepted.
|
|
|
|
|
|
|
|
|
|
Until mox has a proper solution, you can simply run a single SMTP server. The
|
|
|
|
|
author has run a single mail server for over a decade without issues. Machines
|
|
|
|
|
and network connectivity are stable nowadays, and email delivery will be
|
|
|
|
|
retried for many hours during temporary errors (e.g. when rebooting a machine
|
|
|
|
|
after updates).
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-02-27 17:04:53 +03:00
|
|
|
## How do I stay up to date?
|
|
|
|
|
|
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config
- make builtin http handlers serve on specific domains, such as for mta-sts, so
e.g. /.well-known/mta-sts.txt isn't served on all domains.
- add logging of a few more fields in access logging.
- small tweaks/bug fixes in webserver request handling.
- add config option for redirecting entire domains to another (common enough).
- split httpserver metric into two: one for duration until writing header (i.e.
performance of server), another for duration until full response is sent to
client (i.e. performance as perceived by users).
- add admin ui, a new page for managing the configs. after making changes
and hitting "save", the changes take effect immediately. the page itself
doesn't look very well-designed (many input fields, makes it look messy). i
have an idea to improve it (explained in admin.html as todo) by making the
layout look just like the config file. not urgent though.
i've already changed my websites/webapps over.
the idea of adding a webserver is to take away a (the) reason for folks to want
to complicate their mox setup by running an other webserver on the same machine.
i think the current webserver implementation can already serve most common use
cases. with a few more tweaks (feedback needed!) we should be able to get to 95%
of the use cases. the reverse proxy can take care of the remaining 5%.
nevertheless, a next step is still to change the quickstart to make it easier
for folks to run with an existing webserver, with existing tls certs/keys.
that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
|
|
|
Please set "CheckUpdates: true" in mox.conf. Mox will check for a new version
|
|
|
|
|
through a DNS TXT request for `_updates.xmox.nl` once per 24h. Only if a new
|
|
|
|
|
version is published will the changelog be fetched and delivered to the
|
2023-02-27 17:04:53 +03:00
|
|
|
postmaster mailbox.
|
|
|
|
|
|
2023-03-09 18:50:24 +03:00
|
|
|
The changelog, including latest update instructions, is at
|
|
|
|
|
https://updates.xmox.nl/changelog.
|
2023-02-27 17:04:53 +03:00
|
|
|
|
2023-03-09 18:50:24 +03:00
|
|
|
You can also monitor newly added releases on this repository with the github
|
|
|
|
|
"watch" feature, or use the github RSS feed for tags
|
|
|
|
|
(https://github.com/mjl-/mox/tags.atom) or releases
|
|
|
|
|
(https://github.com/mjl-/mox/releases.atom), or monitor the docker images.
|
2023-02-27 17:04:53 +03:00
|
|
|
|
2023-10-04 08:39:44 +03:00
|
|
|
Keep in mind you have a responsibility to keep the internet-connected software
|
2023-02-27 17:04:53 +03:00
|
|
|
you run up to date and secure.
|
|
|
|
|
|
2023-05-31 11:30:34 +03:00
|
|
|
## How do I upgrade my mox installation?
|
|
|
|
|
|
|
|
|
|
We try to make upgrades effortless and you can typically just put a new binary
|
|
|
|
|
in place and restart. If manual actions are required, the release notes mention
|
|
|
|
|
them. Check the release notes of all version between your current installation
|
|
|
|
|
and the release you're upgrading to.
|
|
|
|
|
|
|
|
|
|
Before upgrading, make a backup of the data directory with `mox backup
|
|
|
|
|
<destdir>`. This writes consistent snapshots of the database files, and
|
2023-09-24 13:34:46 +03:00
|
|
|
duplicates message files from the outgoing queue and accounts. Using the new
|
|
|
|
|
mox binary, run `mox verifydata <backupdir>` (do NOT use the "live" data
|
2023-05-31 11:30:34 +03:00
|
|
|
directory!) for a dry run. If this fails, an upgrade will probably fail too.
|
2023-09-24 13:34:46 +03:00
|
|
|
Important: verifydata with the new mox binary can modify the database files (due
|
|
|
|
|
to automatic schema upgrades). So make a fresh backup again before the actual
|
|
|
|
|
upgrade. See the help output of the "backup" and "verifydata" commands for more
|
|
|
|
|
details.
|
2023-05-31 11:30:34 +03:00
|
|
|
|
2023-09-24 13:34:46 +03:00
|
|
|
During backup, message files are hardlinked if possible, and copied otherwise.
|
|
|
|
|
Using a destination directory like `data/tmp/backup` increases the odds
|
|
|
|
|
hardlinking succeeds: the default mox systemd service file mounts
|
|
|
|
|
the data directory separately, so hardlinks to outside the data directory are
|
|
|
|
|
cross-device and will fail.
|
2023-05-31 11:30:34 +03:00
|
|
|
|
|
|
|
|
If an upgrade fails and you have to restore (parts) of the data directory, you
|
|
|
|
|
should run `mox verifydata <datadir>` (with the original binary) on the
|
|
|
|
|
restored directory before starting mox again. If problematic files are found,
|
|
|
|
|
for example queue or account message files that are not in the database, run
|
|
|
|
|
`mox verifydata -fix <datadir>` to move away those files. After a restore, you may
|
|
|
|
|
also want to run `mox bumpuidvalidity <account>` for each account for which
|
|
|
|
|
messages in a mailbox changed, to force IMAP clients to synchronize mailbox
|
|
|
|
|
state.
|
|
|
|
|
|
2023-02-11 15:35:07 +03:00
|
|
|
## How secure is mox?
|
2023-01-30 16:27:06 +03:00
|
|
|
|
2023-02-10 21:43:02 +03:00
|
|
|
Security is high on the priority list for mox. Mox is young, so don't expect no
|
2023-01-30 16:27:06 +03:00
|
|
|
bugs at all. Mox does have automated tests for some security aspects, e.g. for
|
|
|
|
|
login, and uses fuzzing. Mox is written in Go, so some classes of bugs such as
|
|
|
|
|
buffer mishandling do not typically result in privilege escalation. Of course
|
|
|
|
|
logic bugs will still exist. If you find any security issues, please email them
|
|
|
|
|
to mechiel@ueber.net.
|
2023-02-27 17:04:53 +03:00
|
|
|
|
|
|
|
|
## I'm now running an email server, but how does email work?
|
|
|
|
|
|
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config
- make builtin http handlers serve on specific domains, such as for mta-sts, so
e.g. /.well-known/mta-sts.txt isn't served on all domains.
- add logging of a few more fields in access logging.
- small tweaks/bug fixes in webserver request handling.
- add config option for redirecting entire domains to another (common enough).
- split httpserver metric into two: one for duration until writing header (i.e.
performance of server), another for duration until full response is sent to
client (i.e. performance as perceived by users).
- add admin ui, a new page for managing the configs. after making changes
and hitting "save", the changes take effect immediately. the page itself
doesn't look very well-designed (many input fields, makes it look messy). i
have an idea to improve it (explained in admin.html as todo) by making the
layout look just like the config file. not urgent though.
i've already changed my websites/webapps over.
the idea of adding a webserver is to take away a (the) reason for folks to want
to complicate their mox setup by running an other webserver on the same machine.
i think the current webserver implementation can already serve most common use
cases. with a few more tweaks (feedback needed!) we should be able to get to 95%
of the use cases. the reverse proxy can take care of the remaining 5%.
nevertheless, a next step is still to change the quickstart to make it easier
for folks to run with an existing webserver, with existing tls certs/keys.
that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
|
|
|
Congrats and welcome to the club! Running an email server on the internet comes
|
|
|
|
|
with some responsibilities so you should understand how it works. See
|
2023-02-27 17:04:53 +03:00
|
|
|
https://explained-from-first-principles.com/email/ for a thorough explanation.
|
2023-03-04 03:04:54 +03:00
|
|
|
|
|
|
|
|
## What are the minimum requirements to run mox?
|
|
|
|
|
|
|
|
|
|
Mox does not need much. Nowadays most machines are larger than mox needs. You
|
|
|
|
|
can start with a machine with 512MB RAM, any CPU will do. For storage you
|
|
|
|
|
should account for the size of the email messages (no compression currently),
|
|
|
|
|
an additional 15% overhead for the meta data, and add some more headroom.
|
|
|
|
|
Expand as necessary.
|
2023-04-20 17:44:07 +03:00
|
|
|
|
2023-06-12 17:25:35 +03:00
|
|
|
## Won't the big email providers block my email?
|
|
|
|
|
|
|
|
|
|
It is a common misconception that it is impossible to run your own email server
|
|
|
|
|
nowadays. The claim is that the handful big email providers will simply block
|
|
|
|
|
your email. However, you can run your own email server just fine, and your
|
|
|
|
|
email will be accepted, provided you are doing it right.
|
|
|
|
|
|
|
|
|
|
If your email is rejected, it is often because your IP address has a bad email
|
|
|
|
|
sending reputation. Email servers often use IP blocklists to reject email
|
|
|
|
|
networks with a bad email sending reputation. These blocklists often work at
|
|
|
|
|
the level of whole network ranges. So if you try to run an email server from a
|
|
|
|
|
hosting provider with a bad reputation (which happens if they don't monitor
|
|
|
|
|
their network or don't act on abuse/spam reports), your IP too will have a bad
|
|
|
|
|
reputation and other mail servers (both large and small) may reject messages
|
|
|
|
|
coming from you. During the quickstart, mox checks if your IPs are on a few
|
|
|
|
|
often-used blocklists. It's typically not a good idea to host an email server
|
|
|
|
|
on the cheapest or largest cloud providers: They often don't spend the
|
|
|
|
|
resources necessary for a good reputation, or they simply block all outgoing
|
|
|
|
|
SMTP traffic. It's better to look for a technically-focused local provider.
|
2023-06-12 17:35:03 +03:00
|
|
|
They too may initially block outgoing SMTP connections on new machines to
|
|
|
|
|
prevent spam from their networks. But they will either automatically open up
|
|
|
|
|
outgoing SMTP traffic after a cool down period (e.g. 24 hours), or after you've
|
|
|
|
|
contacted their support.
|
2023-06-12 17:25:35 +03:00
|
|
|
|
|
|
|
|
After you get past the IP blocklist checks, email servers use many more signals
|
|
|
|
|
to determine if your email message could be spam and should be rejected. Mox
|
|
|
|
|
helps you set up a system that doesn't trigger most of the technical signals
|
|
|
|
|
(e.g. with SPF/DKIM/DMARC). But there are more signals, for example: Sending to
|
|
|
|
|
a mail server or address for the first time. Sending from a newly registered
|
2024-03-05 11:40:22 +03:00
|
|
|
domain (especially if you're sending automated messages, and if you send more
|
|
|
|
|
messages after previous messages were rejected), domains that existed for a few
|
|
|
|
|
weeks to a month are treated more friendly. Sending messages with content that
|
|
|
|
|
resembles known spam messages.
|
2023-06-12 17:25:35 +03:00
|
|
|
|
2023-09-24 13:34:46 +03:00
|
|
|
Should your email be rejected, you will typically get an error message during
|
|
|
|
|
the SMTP transaction that explains why. In the case of big email providers the
|
2024-01-10 18:48:53 +03:00
|
|
|
error message often has instructions on how to prove to them you are a
|
|
|
|
|
legitimate sender.
|
2023-07-02 16:05:55 +03:00
|
|
|
|
2024-03-27 12:23:37 +03:00
|
|
|
## Can mox deliver through a smarthost?
|
|
|
|
|
|
|
|
|
|
Yes, you can configure a "Transport" in mox.conf and configure "Routes" in
|
|
|
|
|
domains.conf to send some or all messages through the transport. A transport
|
|
|
|
|
can be an SMTP relay or authenticated submission, or making mox make outgoing
|
|
|
|
|
connections through a SOCKS proxy.
|
|
|
|
|
|
|
|
|
|
For an example, see https://www.xmox.nl/config/#hdr-example-transport. For
|
|
|
|
|
details about Transports and Routes, see
|
|
|
|
|
https://www.xmox.nl/config/#cfg-mox-conf-Transports and
|
|
|
|
|
https://www.xmox.nl/config/#cfg-domains-conf-Routes.
|
|
|
|
|
|
|
|
|
|
Remember to add the IP addresses of the transport to the SPF records of your
|
|
|
|
|
domains. Keep in mind some 3rd party submission servers may mishandle your
|
|
|
|
|
messages, for example by replacing your Message-Id header and thereby
|
|
|
|
|
invalidating your DKIM-signatures, or rejecting messages with more than one
|
|
|
|
|
DKIM-signature.
|
|
|
|
|
|
add a webapi and webhooks for a simple http/json-based api
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.
this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.
unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...
matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.
a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.
messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.
suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.
submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".
to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.
admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.
new config options have been introduced. they are editable through the admin
and/or account web interfaces.
the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.
gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.
for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
|
|
|
## Can I use mox to send transactional email?
|
|
|
|
|
|
|
|
|
|
Yes. While you can use SMTP submission to send messages you've composed
|
|
|
|
|
yourself, and monitor a mailbox for DSNs, a more convenient option is to use
|
|
|
|
|
the mox HTTP/JSON-based webapi and webhooks.
|
|
|
|
|
|
|
|
|
|
The mox webapi can be used to send outgoing messages that mox composes. The web
|
|
|
|
|
api can also be used to deal with messages stored in an account, like changing
|
|
|
|
|
message flags, retrieving messages in parsed form or individual parts of
|
|
|
|
|
multipart messages, or moving messages to another mailbox or deleting messages
|
|
|
|
|
altogether.
|
|
|
|
|
|
|
|
|
|
Mox webhooks can be used to receive updates about incoming and outgoing
|
|
|
|
|
deliveries. Mox can automatically manage per account suppression lists.
|
|
|
|
|
|
|
|
|
|
See https://www.xmox.nl/features/#hdr-webapi-and-webhooks for details.
|
|
|
|
|
|
2023-07-02 16:05:55 +03:00
|
|
|
## Can I use existing TLS certificates/keys?
|
|
|
|
|
|
|
|
|
|
Yes. The quickstart command creates a config that uses ACME with Let's Encrypt,
|
|
|
|
|
but you can change the config file to use existing certificate and key files.
|
|
|
|
|
|
|
|
|
|
You'll see "ACME: letsencrypt" in the "TLS" section of the "public" Listener.
|
2024-01-10 18:48:53 +03:00
|
|
|
Remove or comment out the ACME-line, and add a "KeyCerts" section, see
|
|
|
|
|
https://www.xmox.nl/config/#cfg-mox-conf-Listeners-x-TLS-KeyCerts
|
|
|
|
|
|
|
|
|
|
You can have multiple certificates and keys: The line with the "-" (dash) is
|
|
|
|
|
the start of a list item. Duplicate that line up to and including the line with
|
|
|
|
|
KeyFile for each certificate/key you have. Mox makes a TLS config that holds
|
|
|
|
|
all specified certificates/keys, and uses it for all services for that Listener
|
|
|
|
|
(including a webserver), choosing the correct certificate for incoming
|
|
|
|
|
requests.
|
2023-07-02 16:05:55 +03:00
|
|
|
|
|
|
|
|
Keep in mind that for each email domain you host, you will need a certificate
|
2024-01-10 18:48:53 +03:00
|
|
|
for `mta-sts.<domain>`, `autoconfig.<domain>` and `mail.<domain>`, unless you
|
|
|
|
|
disable MTA-STS, autoconfig and the client-settings-domain for that domain.
|
2023-07-02 16:05:55 +03:00
|
|
|
|
|
|
|
|
Mox opens the key and certificate files during initial startup, as root (and
|
|
|
|
|
passes file descriptors to the unprivileged process). No special permissions
|
|
|
|
|
are needed on the key and certificate files.
|
2024-02-10 13:39:31 +03:00
|
|
|
|
|
|
|
|
## Can I directly access mailboxes through the file system?
|
|
|
|
|
|
|
|
|
|
No, mox only provides access to email through protocols like IMAP.
|
|
|
|
|
|
|
|
|
|
While it can be convenient for users/email clients to access email through
|
|
|
|
|
conventions like Maildir, providing such access puts quite a burden on the
|
|
|
|
|
server: The server has to continuously watch for changes made to the mail store
|
|
|
|
|
by external programs, and sync its internal state. By only providing access to
|
|
|
|
|
emails through mox, the storage/state management is simpler and easier to
|
|
|
|
|
implement reliably.
|
|
|
|
|
|
|
|
|
|
Not providing direct file system access also allows future improvements in the
|
|
|
|
|
storage mechanism. Such as encryption of all stored messages. Programs won't be
|
|
|
|
|
able to access such messages directly.
|
|
|
|
|
|
|
|
|
|
Mox stores metadata about delivered messages in its per-account message index
|
|
|
|
|
database, more than fits in a simple (filename-based) format like Maildir. The
|
|
|
|
|
IP address of the remote SMTP server during delivery, SPF/DKIM/DMARC domains
|
|
|
|
|
and validation status, and more...
|
|
|
|
|
|
|
|
|
|
For efficiency, mox doesn't prepend message headers generated during delivery
|
|
|
|
|
(e.g. Authentication-Results) to the on-disk message file, but only stores it
|
|
|
|
|
in the database. This prevents a rewrite of the entire message file. When
|
|
|
|
|
reading a message, mox combines the prepended headers from the database with
|
|
|
|
|
the message file.
|
|
|
|
|
|
|
|
|
|
Mox user accounts have no relation to operating system user accounts. Multiple
|
|
|
|
|
system users reading their email on a single machine is not very common
|
|
|
|
|
anymore. All data (for all accounts) stored by mox is accessible only by the
|
|
|
|
|
mox process. Messages are currently stored as individual files in standard
|
|
|
|
|
Internet Message Format (IMF), at `data/accounts/<account>/msg/<dir>/<msgid>`:
|
|
|
|
|
`msgid` is a consecutive unique integer id assigned by the per-account message
|
|
|
|
|
index database; `dir` groups 8k consecutive message ids into a directory,
|
|
|
|
|
ensuring they don't become too large. The message index database file for an
|
|
|
|
|
account is at `data/accounts/<account>/index.db`, accessed with the bstore
|
|
|
|
|
database library, which uses bbolt (formerly BoltDB) for storage, a
|
|
|
|
|
transactional key/value library/file format inspired by LMDB.
|