mox/quickstart.go

1103 lines
37 KiB
Go
Raw Permalink Normal View History

2023-01-30 16:27:06 +03:00
package main
import (
"bytes"
"context"
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
"crypto"
"crypto/ecdsa"
"crypto/elliptic"
cryptorand "crypto/rand"
"crypto/rsa"
"crypto/x509"
"encoding/pem"
"errors"
2023-01-30 16:27:06 +03:00
"fmt"
"io"
2023-01-30 16:27:06 +03:00
"log"
"net"
"net/url"
2023-01-30 16:27:06 +03:00
"os"
"path/filepath"
"runtime"
"sort"
2023-01-30 16:27:06 +03:00
"strings"
"time"
_ "embed"
"golang.org/x/crypto/bcrypt"
"github.com/mjl-/sconf"
"github.com/mjl-/mox/admin"
2023-01-30 16:27:06 +03:00
"github.com/mjl-/mox/config"
"github.com/mjl-/mox/dns"
"github.com/mjl-/mox/dnsbl"
"github.com/mjl-/mox/mlog"
2023-01-30 16:27:06 +03:00
"github.com/mjl-/mox/mox-"
"github.com/mjl-/mox/smtp"
"github.com/mjl-/mox/store"
)
//go:embed mox.service
var moxService string
func pwgen() string {
chars := "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*-_;:,<.>/"
s := ""
buf := make([]byte, 1)
2023-01-30 16:27:06 +03:00
for i := 0; i < 12; i++ {
for {
cryptorand.Read(buf)
i := int(buf[0])
if i+len(chars) > 255 {
continue // Prevent bias.
}
s += string(chars[i%len(chars)])
break
}
2023-01-30 16:27:06 +03:00
}
return s
}
func cmdQuickstart(c *cmd) {
c.params = "[-skipdial] [-existing-webserver] [-hostname host] user@domain [user | uid]"
2023-01-30 16:27:06 +03:00
c.help = `Quickstart generates configuration files and prints instructions to quickly set up a mox instance.
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
Quickstart writes configuration files, prints initial admin and account
passwords, DNS records you should create. If you run it on Linux it writes a
systemd service file and prints commands to enable and start mox as service.
All output is written to quickstart.log for later reference.
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
The user or uid is optional, defaults to "mox", and is the user or uid/gid mox
will run as after initialization.
Quickstart assumes mox will run on the machine you run quickstart on and uses
its host name and public IPs. On many systems the hostname is not a fully
qualified domain name, but only the first dns "label", e.g. "mail" in case of
"mail.example.org". If so, quickstart does a reverse DNS lookup to find the
hostname, and as fallback uses the label plus the domain of the email address
you specified. Use flag -hostname to explicitly specify the hostname mox will
run on.
Mox is by far easiest to operate if you let it listen on port 443 (HTTPS) and
80 (HTTP). TLS will be fully automatic with ACME with Let's Encrypt.
You can run mox along with an existing webserver, but because of MTA-STS and
autoconfig, you'll need to forward HTTPS traffic for two domains to mox. Run
"mox quickstart -existing-webserver ..." to generate configuration files and
instructions for configuring mox along with an existing webserver.
But please first consider configuring mox on port 443. It can itself serve
domains with HTTP/HTTPS, including with automatic TLS with ACME, is easily
configured through both configuration files and admin web interface, and can act
as a reverse proxy (and static file server for that matter), so you can forward
traffic to your existing backend applications. Look for "WebHandlers:" in the
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
output of "mox config describe-domains" and see the output of
"mox config example webhandlers".
2023-01-30 16:27:06 +03:00
`
var existingWebserver bool
var hostname string
var skipDial bool
c.flag.BoolVar(&existingWebserver, "existing-webserver", false, "use if a webserver is already running, so mox won't listen on port 80 and 443; you'll have to provide tls certificates/keys, and configure the existing webserver as reverse proxy, forwarding requests to mox.")
c.flag.StringVar(&hostname, "hostname", "", "hostname mox will run on, by default the hostname of the machine quickstart runs on; if specified, the IPs for the hostname are configured for the public listener")
c.flag.BoolVar(&skipDial, "skipdial", false, "skip check for outgoing smtp (port 25) connectivity")
2023-01-30 16:27:06 +03:00
args := c.Parse()
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
if len(args) != 1 && len(args) != 2 {
2023-01-30 16:27:06 +03:00
c.Usage()
}
// Write all output to quickstart.log.
logfile, err := os.Create("quickstart.log")
xcheckf(err, "creating quickstart.log")
origStdout := os.Stdout
origStderr := os.Stderr
piper, pipew, err := os.Pipe()
xcheckf(err, "creating pipe for logging to logfile")
pipec := make(chan struct{})
go func() {
io.Copy(io.MultiWriter(origStdout, logfile), piper)
close(pipec)
}()
// A single pipe, so writes to stdout and stderr don't get interleaved.
os.Stdout = pipew
os.Stderr = pipew
logClose := func() {
pipew.Close()
<-pipec
os.Stdout = origStdout
os.Stderr = origStderr
err := logfile.Close()
xcheckf(err, "closing quickstart.log")
}
defer logClose()
log.SetOutput(os.Stdout)
fmt.Printf("(output is also written to quickstart.log)\n\n")
defer fmt.Printf("\n(output is also written to quickstart.log)\n")
2023-01-30 16:27:06 +03:00
// We take care to cleanup created files when we error out.
// We don't want to get a new user into trouble with half of the files
// after encountering an error.
// We use fatalf instead of log.Fatal* to cleanup files.
var cleanupPaths []string
fatalf := func(format string, args ...any) {
// We remove in reverse order because dirs would have been created first and must
// be removed last, after their files have been removed.
for i := len(cleanupPaths) - 1; i >= 0; i-- {
p := cleanupPaths[i]
if err := os.Remove(p); err != nil {
log.Printf("cleaning up %q: %s", p, err)
}
}
log.Printf(format, args...)
logClose()
os.Exit(1)
2023-01-30 16:27:06 +03:00
}
xwritefile := func(path string, data []byte, perm os.FileMode) {
os.MkdirAll(filepath.Dir(path), 0770)
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, perm)
if err != nil {
fatalf("creating file %q: %s", path, err)
}
cleanupPaths = append(cleanupPaths, path)
_, err = f.Write(data)
if err == nil {
err = f.Close()
}
if err != nil {
fatalf("writing file %q: %s", path, err)
}
}
addr, err := smtp.ParseAddress(args[0])
if err != nil {
fatalf("parsing email address: %s", err)
}
accountName := addr.Localpart.String()
2023-01-30 16:27:06 +03:00
domain := addr.Domain
for _, c := range accountName {
2023-01-30 16:27:06 +03:00
if c > 0x7f {
fmt.Printf(`NOTE: Username %q is not ASCII-only. It is recommended you also configure an
ASCII-only alias. Both for delivery of email from other systems, and for
logging in with IMAP.
`, accountName)
2023-01-30 16:27:06 +03:00
break
}
}
resolver := dns.StrictResolver{}
// We don't want to spend too much total time on the DNS lookups. Because DNS may
// not work during quickstart, and we don't want to loop doing requests and having
// to wait for a timeout each time.
resolveCtx, resolveCancel := context.WithTimeout(context.Background(), 10*time.Second)
defer resolveCancel()
// Some DNSSEC-verifying resolvers return unauthentic data for ".", so we check "com".
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
fmt.Printf("Checking if DNS resolvers are DNSSEC-verifying...")
_, resolverDNSSECResult, err := resolver.LookupNS(resolveCtx, "com.")
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
if err != nil {
fmt.Println("")
fatalf("checking dnssec support in resolver: %v", err)
} else if !resolverDNSSECResult.Authentic {
fmt.Printf(`
WARNING: It looks like the DNS resolvers configured on your system do not
verify DNSSEC, or aren't trusted (by having loopback IPs or through "options
trust-ad" in /etc/resolv.conf). Without DNSSEC, outbound delivery with SMTP
used unprotected MX records, and SMTP STARTTLS connections cannot verify the TLS
certificate with DANE (based on a public key in DNS), and will fall back to
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
either MTA-STS for verification, or use "opportunistic TLS" with no certificate
verification.
Recommended action: Install unbound, a DNSSEC-verifying recursive DNS resolver,
ensure it has DNSSEC root keys (see unbound-anchor), and enable support for
"extended dns errors" (EDE, available since unbound v1.16.0, see below; not
required, but it gives helpful error messages about DNSSEC failures instead of
generic DNS SERVFAIL errors). Test with "dig com. ns" and look for "ad"
(authentic data) in response "flags".
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
cat <<EOF >/etc/unbound/unbound.conf.d/ede.conf
server:
ede: yes
val-log-level: 2
EOF
Troubleshooting hints:
- Ensure /etc/resolv.conf has "nameserver 127.0.0.1". If the IP is 127.0.0.53,
DNS resolving is done by systemd-resolved. Make sure "resolvconf" isn't
overwriting /etc/resolv.conf (Debian has a package "openresolv" that makes this
easier). "dig" also shows to which IP the DNS request was sent.
- Ensure unbound has DNSSEC root keys available. See unbound config option
"auto-trust-anchor-file" and the unbound-anchor command. Ensure the file exists.
- Run "./mox dns lookup ns com." to simulate the DNSSEC check done by mox. The
output should say "with dnssec".
- The "delv" command can check whether a domain is DNSSEC-signed, but it does
its own DNSSEC verification instead of relying on the resolver, so you cannot
use it to check whether unbound is verifying DNSSEC correctly.
- Increase logging in unbound, see options "verbosity" and "log-queries".
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
`)
} else {
fmt.Println(" OK")
}
// We are going to find the (public) IPs to listen on and possibly the host name.
// Start with reasonable defaults. We'll replace them specific IPs, if we can find them.
privateListenerIPs := []string{"127.0.0.1", "::1"}
publicListenerIPs := []string{"0.0.0.0", "::"}
var publicNATIPs []string // Actual public IP, but when it is NATed and machine doesn't have direct access.
defaultPublicListenerIPs := true
// If we find IPs based on network interfaces, {public,private}ListenerIPs are set
// based on these values.
var loopbackIPs, privateIPs, publicIPs []string
// Gather IP addresses for public and private listeners.
// We look at each network interface. If an interface has a private address, we
// conservatively assume all addresses on that interface are private.
ifaces, err := net.Interfaces()
if err != nil {
fatalf("listing network interfaces: %s", err)
}
parseAddrIP := func(s string) net.IP {
if strings.HasPrefix(s, "[") && strings.HasSuffix(s, "]") {
s = s[1 : len(s)-1]
}
ip, _, _ := net.ParseCIDR(s)
return ip
}
for _, iface := range ifaces {
if iface.Flags&net.FlagUp == 0 {
continue
}
addrs, err := iface.Addrs()
if err != nil {
fatalf("listing address for network interface: %s", err)
}
if len(addrs) == 0 {
continue
}
// todo: should we detect temporary/ephemeral ipv6 addresses and not add them?
var nonpublic bool
for _, addr := range addrs {
ip := parseAddrIP(addr.String())
if ip.IsInterfaceLocalMulticast() || ip.IsLinkLocalMulticast() || ip.IsLinkLocalUnicast() || ip.IsMulticast() {
continue
}
if ip.IsLoopback() || ip.IsPrivate() {
nonpublic = true
break
}
}
for _, addr := range addrs {
ip := parseAddrIP(addr.String())
if ip == nil {
continue
}
if ip.IsInterfaceLocalMulticast() || ip.IsLinkLocalMulticast() || ip.IsLinkLocalUnicast() || ip.IsMulticast() {
continue
}
if nonpublic {
if ip.IsLoopback() {
loopbackIPs = append(loopbackIPs, ip.String())
} else {
privateIPs = append(privateIPs, ip.String())
}
} else {
publicIPs = append(publicIPs, ip.String())
}
}
}
var dnshostname dns.Domain
if hostname == "" {
hostnameStr, err := os.Hostname()
if err != nil {
fatalf("hostname: %s", err)
}
if strings.Contains(hostnameStr, ".") {
dnshostname, err = dns.ParseDomain(hostnameStr)
if err != nil {
fatalf("parsing hostname: %v", err)
}
} else {
// It seems Linux machines don't have a single FQDN configured. E.g. /etc/hostname
// is just the name without domain. We'll look up the names for all IPs, and hope
// to find a single FQDN name (with at least 1 dot).
names := map[string]struct{}{}
if len(publicIPs) > 0 {
fmt.Printf("Trying to find hostname by reverse lookup of public IPs %s...", strings.Join(publicIPs, ", "))
}
var warned bool
warnf := func(format string, args ...any) {
warned = true
fmt.Printf("\n%s", fmt.Sprintf(format, args...))
}
for _, ip := range publicIPs {
revctx, revcancel := context.WithTimeout(resolveCtx, 5*time.Second)
defer revcancel()
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
l, _, err := resolver.LookupAddr(revctx, ip)
if err != nil {
warnf("WARNING: looking up reverse name(s) for %s: %v", ip, err)
}
for _, name := range l {
if strings.Contains(name, ".") {
names[name] = struct{}{}
}
}
}
var nameList []string
for k := range names {
nameList = append(nameList, strings.TrimRight(k, "."))
}
sort.Slice(nameList, func(i, j int) bool {
return nameList[i] < nameList[j]
})
if len(nameList) == 0 {
dnshostname, err = dns.ParseDomain(hostnameStr + "." + domain.Name())
if err != nil {
fmt.Println()
fatalf("parsing hostname: %v", err)
}
warnf(`WARNING: cannot determine hostname because the system name is not an FQDN and
no public IPs resolving to an FQDN were found. Quickstart guessed the host name
below. If it is not correct, please remove the generated config files and run
quickstart again with the -hostname flag.
%s
`, dnshostname)
} else {
if len(nameList) > 1 {
warnf(`WARNING: multiple hostnames found for the public IPs, using the first of: %s
If this is not correct, remove the generated config files and run quickstart
again with the -hostname flag.
`, strings.Join(nameList, ", "))
}
dnshostname, err = dns.ParseDomain(nameList[0])
if err != nil {
fmt.Println()
fatalf("parsing hostname %s: %v", nameList[0], err)
}
}
if warned {
fmt.Printf("\n\n")
} else {
fmt.Printf(" found %s\n", dnshostname)
}
}
} else {
// Host name was explicitly configured on command-line. We'll try to use its public
// IPs below.
var err error
dnshostname, err = dns.ParseDomain(hostname)
if err != nil {
fatalf("parsing hostname: %v", err)
2023-01-30 16:27:06 +03:00
}
}
fmt.Printf("Looking up IPs for hostname %s...", dnshostname)
ipctx, ipcancel := context.WithTimeout(resolveCtx, 5*time.Second)
defer ipcancel()
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
ips, domainDNSSECResult, err := resolver.LookupIPAddr(ipctx, dnshostname.ASCII+".")
ipcancel()
var xips []net.IPAddr
var hostIPs []string
var dnswarned bool
hostPrivate := len(ips) > 0
for _, ip := range ips {
if !ip.IP.IsPrivate() {
hostPrivate = false
}
// During linux install, you may get an alias for you full hostname in /etc/hosts
// resolving to 127.0.1.1, which would result in a false positive about the
// hostname having a record. Filter it out. It is a bit surprising that hosts don't
// otherwise know their FQDN.
if ip.IP.IsLoopback() {
dnswarned = true
fmt.Printf("\n\nWARNING: Your hostname is resolving to a loopback IP address %s. This likely breaks email delivery to local accounts. /etc/hosts likely contains a line like %q. Either replace it with your actual IP(s), or remove the line.\n", ip.IP, fmt.Sprintf("%s %s", ip.IP, dnshostname.ASCII))
continue
}
xips = append(xips, ip)
hostIPs = append(hostIPs, ip.String())
}
if err == nil && len(xips) == 0 {
// todo: possibly check this by trying to resolve without using /etc/hosts?
err = errors.New("hostname not in dns, probably only in /etc/hosts")
}
ips = xips
// We may have found private and public IPs on the machine, and IPs for the host
// name we think we should use. They may not match with each other. E.g. the public
// IPs on interfaces could be different from the IPs for the host. We don't try to
// detect all possible configs, but just generate what makes sense given whether we
// found public/private/hostname IPs. If the user is doing sensible things, it
// should be correct. But they should be checking the generated config file anyway.
// And we do log which host name we are using, and whether we detected a NAT setup.
// In the future, we may do an interactive setup that can guide the user better.
if !hostPrivate && len(publicIPs) == 0 && len(privateIPs) > 0 {
// We only have private IPs, assume we are behind a NAT and put the IPs of the host in NATIPs.
publicListenerIPs = privateIPs
publicNATIPs = hostIPs
defaultPublicListenerIPs = false
if len(loopbackIPs) > 0 {
privateListenerIPs = loopbackIPs
}
} else {
if len(hostIPs) > 0 {
publicListenerIPs = hostIPs
defaultPublicListenerIPs = false
// Only keep private IPs that are not in host-based publicListenerIPs. For
// internal-only setups, including integration tests.
m := map[string]bool{}
for _, ip := range hostIPs {
m[ip] = true
}
var npriv []string
for _, ip := range privateIPs {
if !m[ip] {
npriv = append(npriv, ip)
}
}
sort.Strings(npriv)
privateIPs = npriv
} else if len(publicIPs) > 0 {
publicListenerIPs = publicIPs
defaultPublicListenerIPs = false
hostIPs = publicIPs // For DNSBL check below.
}
if len(privateIPs) > 0 {
privateListenerIPs = append(privateIPs, loopbackIPs...)
} else if len(loopbackIPs) > 0 {
privateListenerIPs = loopbackIPs
}
}
2023-01-30 16:27:06 +03:00
if err != nil {
if !dnswarned {
fmt.Printf("\n")
}
dnswarned = true
fmt.Printf(`
WARNING: Quickstart assumed the hostname of this machine is %s and generates a
config for that host, but could not retrieve that name from DNS:
%s
This likely means one of two things:
1. You don't have any DNS records for this machine at all. You should add them
before continuing.
2. The hostname mentioned is not the correct host name of this machine. You will
have to replace the hostname in the suggested DNS records and generated
config/mox.conf file. Make sure your hostname resolves to your public IPs, and
your public IPs resolve back (reverse) to your hostname.
`, dnshostname, err)
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
} else if !domainDNSSECResult.Authentic {
if !dnswarned {
fmt.Printf("\n")
}
dnswarned = true
fmt.Printf(`
NOTE: It looks like the DNS records of your domain (zone) are not DNSSEC-signed.
Mail servers that send email to your domain, or receive email from your domain,
cannot verify that the MX/SPF/DKIM/DMARC/MTA-STS records they receive are
authentic. DANE, for authenticated delivery without relying on a pool of
certificate authorities, requires DNSSEC, so will not be configured at this
time.
Recommended action: Continue now, but consider enabling DNSSEC for your domain
later at your DNS operator, and adding DANE records for protecting incoming
messages over SMTP.
`)
}
if !dnswarned {
2023-01-30 16:27:06 +03:00
fmt.Printf(" OK\n")
var l []string
type result struct {
IP string
Addrs []string
Err error
}
results := make(chan result)
for _, ip := range ips {
s := ip.String()
l = append(l, s)
go func() {
revctx, revcancel := context.WithTimeout(resolveCtx, 5*time.Second)
defer revcancel()
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
addrs, _, err := resolver.LookupAddr(revctx, s)
results <- result{s, addrs, err}
}()
}
fmt.Printf("Looking up reverse names for IP(s) %s...", strings.Join(l, ", "))
var warned bool
warnf := func(format string, args ...any) {
fmt.Printf("\nWARNING: %s", fmt.Sprintf(format, args...))
warned = true
}
for i := 0; i < len(ips); i++ {
r := <-results
if r.Err != nil {
warnf("looking up reverse name for %s: %v", r.IP, r.Err)
continue
}
if len(r.Addrs) != 1 {
warnf("expected exactly 1 name for %s, got %d (%v)", r.IP, len(r.Addrs), r.Addrs)
}
var match bool
for i, a := range r.Addrs {
a = strings.TrimRight(a, ".")
r.Addrs[i] = a // For potential error message below.
d, err := dns.ParseDomain(a)
if err != nil {
warnf("parsing reverse name %q for %s: %v", a, r.IP, err)
}
if d == dnshostname {
match = true
}
}
if !match {
warnf("reverse name(s) %s for ip %s do not match hostname %s, which will cause other mail servers to reject incoming messages from this IP", strings.Join(r.Addrs, ","), r.IP, dnshostname)
}
}
if warned {
fmt.Printf("\n\n")
} else {
fmt.Printf(" OK\n")
}
}
// Check outgoing SMTP connectivity.
if !skipDial {
fmt.Printf("Checking if outgoing smtp connections can be made by connecting to gmail.com mx on port 25...")
mxctx, mxcancel := context.WithTimeout(context.Background(), 5*time.Second)
mx, _, err := resolver.LookupMX(mxctx, "gmail.com.")
mxcancel()
if err == nil && len(mx) == 0 {
err = errors.New("no mx records")
}
var ok bool
if err != nil {
fmt.Printf("\n\nERROR: looking up gmail.com mx record: %s\n", err)
} else {
dialctx, dialcancel := context.WithTimeout(context.Background(), 10*time.Second)
d := net.Dialer{}
addr := net.JoinHostPort(mx[0].Host, "25")
conn, err := d.DialContext(dialctx, "tcp", addr)
dialcancel()
if err != nil {
fmt.Printf("\n\nERROR: connecting to %s: %s\n", addr, err)
} else {
conn.Close()
fmt.Printf(" OK\n")
ok = true
}
}
if !ok {
fmt.Printf(`
WARNING: Could not verify outgoing smtp connections can be made, outgoing
delivery may not be working. Many providers block outgoing smtp connections by
default, requiring an explicit request or a cooldown period before allowing
outgoing smtp connections. To send through a smarthost, configure a "Transport"
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
in mox.conf and use it in "Routes" in domains.conf. See
"mox config example transport".
`)
}
}
zones := []dns.Domain{
{ASCII: "sbl.spamhaus.org"},
{ASCII: "bl.spamcop.net"},
}
if len(hostIPs) > 0 {
fmt.Printf("Checking whether host name IPs are listed in popular DNS block lists...")
var listed bool
for _, zone := range zones {
for _, ip := range hostIPs {
dnsblctx, dnsblcancel := context.WithTimeout(context.Background(), 5*time.Second)
status, expl, err := dnsbl.Lookup(dnsblctx, c.log.Logger, resolver, zone, net.ParseIP(ip))
dnsblcancel()
if status == dnsbl.StatusPass {
continue
}
errstr := ""
if err != nil {
errstr = fmt.Sprintf(" (%s)", err)
}
fmt.Printf("\nWARNING: checking your public IP %s in DNS block list %s: %v %s%s", ip, zone.Name(), status, expl, errstr)
listed = true
}
}
if listed {
log.Printf(`
Other mail servers are likely to reject email from IPs that are in a blocklist.
If all your IPs are in block lists, you will encounter problems delivering
email. Your IP may be in block lists only temporarily. To see if your IPs are
listed in more DNS block lists, visit:
`)
for _, ip := range hostIPs {
fmt.Printf("- https://multirbl.valli.org/lookup/%s.html\n", url.PathEscape(ip))
}
fmt.Printf("\n")
} else {
fmt.Printf(" OK\n")
}
2023-01-30 16:27:06 +03:00
}
if defaultPublicListenerIPs {
log.Printf(`
WARNING: Could not find your public IP address(es). The "public" listener is
configured to listen on 0.0.0.0 (IPv4) and :: (IPv6). If you don't change these
to your actual public IP addresses, you will likely get "address in use" errors
when starting mox because the "internal" listener binds to a specific IP
address on the same port(s). If you are behind a NAT, instead configure the
actual public IPs in the listener's "NATIPs" option.
`)
}
if len(publicNATIPs) > 0 {
log.Printf(`
NOTE: Quickstart used the IPs of the host name of the mail server, but only
found private IPs on the machine. This indicates this machine is behind a NAT,
so the host IPs were configured in the NATIPs field of the public listeners. If
you are behind a NAT that does not preserve the remote IPs of connections, you
will likely experience problems accepting email due to IP-based policies. For
example, SPF is a mechanism that checks if an IP address is allowed to send
email for a domain, and mox uses IP-based (non)junk classification, and IP-based
rate-limiting both for accepting email and blocking bad actors (such as with too
many authentication failures).
`)
}
fmt.Printf("\n")
2023-01-30 16:27:06 +03:00
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
user := "mox"
if len(args) == 2 {
user = args[1]
}
2023-01-30 16:27:06 +03:00
dc := config.Dynamic{}
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
sc := config.Static{
make mox compile on windows, without "mox serve" but with working "mox localserve" getting mox to compile required changing code in only a few places where package "syscall" was used: for accessing file access times and for umask handling. an open problem is how to start a process as an unprivileged user on windows. that's why "mox serve" isn't implemented yet. and just finding a way to implement it now may not be good enough in the near future: we may want to starting using a more complete privilege separation approach, with a process handling sensitive tasks (handling private keys, authentication), where we may want to pass file descriptors between processes. how would that work on windows? anyway, getting mox to compile for windows doesn't mean it works properly on windows. the largest issue: mox would normally open a file, rename or remove it, and finally close it. this happens during message delivery. that doesn't work on windows, the rename/remove would fail because the file is still open. so this commit swaps many "remove" and "close" calls. renames are a longer story: message delivery had two ways to deliver: with "consuming" the (temporary) message file (which would rename it to its final destination), and without consuming (by hardlinking the file, falling back to copying). the last delivery to a recipient of a message (and the only one in the common case of a single recipient) would consume the message, and the earlier recipients would not. during delivery, the already open message file was used, to parse the message. we still want to use that open message file, and the caller now stays responsible for closing it, but we no longer try to rename (consume) the file. we always hardlink (or copy) during delivery (this works on windows), and the caller is responsible for closing and removing (in that order) the original temporary file. this does cost one syscall more. but it makes the delivery code (responsibilities) a bit simpler. there is one more obvious issue: the file system path separator. mox already used the "filepath" package to join paths in many places, but not everywhere. and it still used strings with slashes for local file access. with this commit, the code now uses filepath.FromSlash for path strings with slashes, uses "filepath" in a few more places where it previously didn't. also switches from "filepath" to regular "path" package when handling mailbox names in a few places, because those always use forward slashes, regardless of local file system conventions. windows can handle forward slashes when opening files, so test code that passes path strings with forward slashes straight to go stdlib file i/o functions are left unchanged to reduce code churn. the regular non-test code, or test code that uses path strings in places other than standard i/o functions, does have the paths converted for consistent paths (otherwise we would end up with paths with mixed forward/backward slashes in log messages). windows cannot dup a listening socket. for "mox localserve", it isn't important, and we can work around the issue. the current approach for "mox serve" (forking a process and passing file descriptors of listening sockets on "privileged" ports) won't work on windows. perhaps it isn't needed on windows, and any user can listen on "privileged" ports? that would be welcome. on windows, os.Open cannot open a directory, so we cannot call Sync on it after message delivery. a cursory internet search indicates that directories cannot be synced on windows. the story is probably much more nuanced than that, with long deep technical details/discussions/disagreement/confusion, like on unix. for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
DataDir: filepath.FromSlash("../data"),
User: user,
LogLevel: "debug", // Help new users, they'll bring it back to info when it all works.
Hostname: dnshostname.Name(),
AdminPasswordFile: "adminpasswd",
}
// todo: let user specify an alternative fallback address?
// Don't attempt to use a non-ascii localpart with Let's Encrypt, it won't work.
// Messages to postmaster will get to the account too.
var contactEmail string
if addr.Localpart.IsInternational() {
contactEmail = smtp.NewAddress("postmaster", addr.Domain).Pack(false)
} else {
contactEmail = addr.Pack(false)
}
if !existingWebserver {
sc.ACME = map[string]config.ACME{
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
"letsencrypt": {
DirectoryURL: "https://acme-v02.api.letsencrypt.org/directory",
ContactEmail: contactEmail,
IssuerDomainName: "letsencrypt.org",
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
},
}
2023-01-30 16:27:06 +03:00
}
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
dataDir := "data" // ../data is relative to config/
os.MkdirAll(dataDir, 0770)
2023-01-30 16:27:06 +03:00
adminpw := pwgen()
adminpwhash, err := bcrypt.GenerateFromPassword([]byte(adminpw), bcrypt.DefaultCost)
if err != nil {
fatalf("generating hash for generated admin password: %s", err)
}
xwritefile(filepath.Join("config", sc.AdminPasswordFile), adminpwhash, 0660)
fmt.Printf("Admin password: %s\n", adminpw)
public := config.Listener{
IPs: publicListenerIPs,
NATIPs: publicNATIPs,
2023-01-30 16:27:06 +03:00
}
public.SMTP.Enabled = true
public.Submissions.Enabled = true
public.IMAPS.Enabled = true
if existingWebserver {
make mox compile on windows, without "mox serve" but with working "mox localserve" getting mox to compile required changing code in only a few places where package "syscall" was used: for accessing file access times and for umask handling. an open problem is how to start a process as an unprivileged user on windows. that's why "mox serve" isn't implemented yet. and just finding a way to implement it now may not be good enough in the near future: we may want to starting using a more complete privilege separation approach, with a process handling sensitive tasks (handling private keys, authentication), where we may want to pass file descriptors between processes. how would that work on windows? anyway, getting mox to compile for windows doesn't mean it works properly on windows. the largest issue: mox would normally open a file, rename or remove it, and finally close it. this happens during message delivery. that doesn't work on windows, the rename/remove would fail because the file is still open. so this commit swaps many "remove" and "close" calls. renames are a longer story: message delivery had two ways to deliver: with "consuming" the (temporary) message file (which would rename it to its final destination), and without consuming (by hardlinking the file, falling back to copying). the last delivery to a recipient of a message (and the only one in the common case of a single recipient) would consume the message, and the earlier recipients would not. during delivery, the already open message file was used, to parse the message. we still want to use that open message file, and the caller now stays responsible for closing it, but we no longer try to rename (consume) the file. we always hardlink (or copy) during delivery (this works on windows), and the caller is responsible for closing and removing (in that order) the original temporary file. this does cost one syscall more. but it makes the delivery code (responsibilities) a bit simpler. there is one more obvious issue: the file system path separator. mox already used the "filepath" package to join paths in many places, but not everywhere. and it still used strings with slashes for local file access. with this commit, the code now uses filepath.FromSlash for path strings with slashes, uses "filepath" in a few more places where it previously didn't. also switches from "filepath" to regular "path" package when handling mailbox names in a few places, because those always use forward slashes, regardless of local file system conventions. windows can handle forward slashes when opening files, so test code that passes path strings with forward slashes straight to go stdlib file i/o functions are left unchanged to reduce code churn. the regular non-test code, or test code that uses path strings in places other than standard i/o functions, does have the paths converted for consistent paths (otherwise we would end up with paths with mixed forward/backward slashes in log messages). windows cannot dup a listening socket. for "mox localserve", it isn't important, and we can work around the issue. the current approach for "mox serve" (forking a process and passing file descriptors of listening sockets on "privileged" ports) won't work on windows. perhaps it isn't needed on windows, and any user can listen on "privileged" ports? that would be welcome. on windows, os.Open cannot open a directory, so we cannot call Sync on it after message delivery. a cursory internet search indicates that directories cannot be synced on windows. the story is probably much more nuanced than that, with long deep technical details/discussions/disagreement/confusion, like on unix. for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
hostbase := filepath.FromSlash("path/to/" + dnshostname.Name())
mtastsbase := filepath.FromSlash("path/to/mta-sts." + domain.Name())
autoconfigbase := filepath.FromSlash("path/to/autoconfig." + domain.Name())
mailbase := filepath.FromSlash("path/to/mail." + domain.Name())
public.TLS = &config.TLS{
KeyCerts: []config.KeyCert{
{CertFile: hostbase + "-chain.crt.pem", KeyFile: hostbase + ".key.pem"},
{CertFile: mtastsbase + "-chain.crt.pem", KeyFile: mtastsbase + ".key.pem"},
{CertFile: autoconfigbase + "-chain.crt.pem", KeyFile: autoconfigbase + ".key.pem"},
},
}
if mailbase != hostbase {
public.TLS.KeyCerts = append(public.TLS.KeyCerts, config.KeyCert{CertFile: mailbase + "-chain.crt.pem", KeyFile: mailbase + ".key.pem"})
}
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
fmt.Println(
`Placeholder paths to TLS certificates to be provided by the existing webserver
have been placed in config/mox.conf and need to be edited.
No private keys for the public listener have been generated for use with DANE.
To configure DANE (which requires DNSSEC), set config field HostPrivateKeyFiles
in the "public" Listener to both RSA 2048-bit and ECDSA P-256 private key files
and check the admin page for the needed DNS records.`)
} else {
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
// todo: we may want to generate a second set of keys, make the user already add it to the DNS, but keep the private key offline. would require config option to specify a public key only, so the dane records can be generated.
hostRSAPrivateKey, err := rsa.GenerateKey(cryptorand.Reader, 2048)
if err != nil {
fatalf("generating rsa private key for host: %s", err)
}
hostECDSAPrivateKey, err := ecdsa.GenerateKey(elliptic.P256(), cryptorand.Reader)
if err != nil {
fatalf("generating ecsa private key for host: %s", err)
}
now := time.Now()
timestamp := now.Format("20060102T150405")
make mox compile on windows, without "mox serve" but with working "mox localserve" getting mox to compile required changing code in only a few places where package "syscall" was used: for accessing file access times and for umask handling. an open problem is how to start a process as an unprivileged user on windows. that's why "mox serve" isn't implemented yet. and just finding a way to implement it now may not be good enough in the near future: we may want to starting using a more complete privilege separation approach, with a process handling sensitive tasks (handling private keys, authentication), where we may want to pass file descriptors between processes. how would that work on windows? anyway, getting mox to compile for windows doesn't mean it works properly on windows. the largest issue: mox would normally open a file, rename or remove it, and finally close it. this happens during message delivery. that doesn't work on windows, the rename/remove would fail because the file is still open. so this commit swaps many "remove" and "close" calls. renames are a longer story: message delivery had two ways to deliver: with "consuming" the (temporary) message file (which would rename it to its final destination), and without consuming (by hardlinking the file, falling back to copying). the last delivery to a recipient of a message (and the only one in the common case of a single recipient) would consume the message, and the earlier recipients would not. during delivery, the already open message file was used, to parse the message. we still want to use that open message file, and the caller now stays responsible for closing it, but we no longer try to rename (consume) the file. we always hardlink (or copy) during delivery (this works on windows), and the caller is responsible for closing and removing (in that order) the original temporary file. this does cost one syscall more. but it makes the delivery code (responsibilities) a bit simpler. there is one more obvious issue: the file system path separator. mox already used the "filepath" package to join paths in many places, but not everywhere. and it still used strings with slashes for local file access. with this commit, the code now uses filepath.FromSlash for path strings with slashes, uses "filepath" in a few more places where it previously didn't. also switches from "filepath" to regular "path" package when handling mailbox names in a few places, because those always use forward slashes, regardless of local file system conventions. windows can handle forward slashes when opening files, so test code that passes path strings with forward slashes straight to go stdlib file i/o functions are left unchanged to reduce code churn. the regular non-test code, or test code that uses path strings in places other than standard i/o functions, does have the paths converted for consistent paths (otherwise we would end up with paths with mixed forward/backward slashes in log messages). windows cannot dup a listening socket. for "mox localserve", it isn't important, and we can work around the issue. the current approach for "mox serve" (forking a process and passing file descriptors of listening sockets on "privileged" ports) won't work on windows. perhaps it isn't needed on windows, and any user can listen on "privileged" ports? that would be welcome. on windows, os.Open cannot open a directory, so we cannot call Sync on it after message delivery. a cursory internet search indicates that directories cannot be synced on windows. the story is probably much more nuanced than that, with long deep technical details/discussions/disagreement/confusion, like on unix. for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
hostRSAPrivateKeyFile := filepath.Join("hostkeys", fmt.Sprintf("%s.%s.%s.privatekey.pkcs8.pem", dnshostname.Name(), timestamp, "rsa2048"))
hostECDSAPrivateKeyFile := filepath.Join("hostkeys", fmt.Sprintf("%s.%s.%s.privatekey.pkcs8.pem", dnshostname.Name(), timestamp, "ecdsap256"))
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
xwritehostkeyfile := func(path string, key crypto.Signer) {
buf, err := x509.MarshalPKCS8PrivateKey(key)
if err != nil {
fatalf("marshaling host private key to pkcs8 for %s: %s", path, err)
}
var b bytes.Buffer
block := pem.Block{
Type: "PRIVATE KEY",
Bytes: buf,
}
err = pem.Encode(&b, &block)
if err != nil {
fatalf("pem-encoding host private key file for %s: %s", path, err)
}
xwritefile(path, b.Bytes(), 0600)
}
xwritehostkeyfile(filepath.Join("config", hostRSAPrivateKeyFile), hostRSAPrivateKey)
xwritehostkeyfile(filepath.Join("config", hostECDSAPrivateKeyFile), hostECDSAPrivateKey)
public.TLS = &config.TLS{
ACME: "letsencrypt",
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
HostPrivateKeyFiles: []string{
hostRSAPrivateKeyFile,
hostECDSAPrivateKeyFile,
},
HostPrivateRSA2048Keys: []crypto.Signer{hostRSAPrivateKey},
HostPrivateECDSAP256Keys: []crypto.Signer{hostECDSAPrivateKey},
}
public.AutoconfigHTTPS.Enabled = true
public.MTASTSHTTPS.Enabled = true
public.WebserverHTTP.Enabled = true
public.WebserverHTTPS.Enabled = true
}
2023-01-30 16:27:06 +03:00
// Suggest blocklists, but we'll comment them out after generating the config.
for _, zone := range zones {
public.SMTP.DNSBLs = append(public.SMTP.DNSBLs, zone.Name())
}
2023-01-30 16:27:06 +03:00
// Monitor DNSBLs by default, without using them for incoming deliveries.
for _, zone := range zones {
dc.MonitorDNSBLs = append(dc.MonitorDNSBLs, zone.Name())
}
2023-01-30 16:27:06 +03:00
internal := config.Listener{
IPs: privateListenerIPs,
Hostname: "localhost",
}
internal.AccountHTTP.Enabled = true
2023-01-30 16:27:06 +03:00
internal.AdminHTTP.Enabled = true
add webmail it was far down on the roadmap, but implemented earlier, because it's interesting, and to help prepare for a jmap implementation. for jmap we need to implement more client-like functionality than with just imap. internal data structures need to change. jmap has lots of other requirements, so it's already a big project. by implementing a webmail now, some of the required data structure changes become clear and can be made now, so the later jmap implementation can do things similarly to the webmail code. the webmail frontend and webmail are written together, making their interface/api much smaller and simpler than jmap. one of the internal changes is that we now keep track of per-mailbox total/unread/unseen/deleted message counts and mailbox sizes. keeping this data consistent after any change to the stored messages (through the code base) is tricky, so mox now has a consistency check that verifies the counts are correct, which runs only during tests, each time an internal account reference is closed. we have a few more internal "changes" that are propagated for the webmail frontend (that imap doesn't have a way to propagate on a connection), like changes to the special-use flags on mailboxes, and used keywords in a mailbox. more changes that will be required have revealed themselves while implementing the webmail, and will be implemented next. the webmail user interface is modeled after the mail clients i use or have used: thunderbird, macos mail, mutt; and webmails i normally only use for testing: gmail, proton, yahoo, outlook. a somewhat technical user is assumed, but still the goal is to make this webmail client easy to use for everyone. the user interface looks like most other mail clients: a list of mailboxes, a search bar, a message list view, and message details. there is a top/bottom and a left/right layout for the list/message view, default is automatic based on screen size. the panes can be resized by the user. buttons for actions are just text, not icons. clicking a button briefly shows the shortcut for the action in the bottom right, helping with learning to operate quickly. any text that is underdotted has a title attribute that causes more information to be displayed, e.g. what a button does or a field is about. to highlight potential phishing attempts, any text (anywhere in the webclient) that switches unicode "blocks" (a rough approximation to (language) scripts) within a word is underlined orange. multiple messages can be selected with familiar ui interaction: clicking while holding control and/or shift keys. keyboard navigation works with arrows/page up/down and home/end keys, and also with a few basic vi-like keys for list/message navigation. we prefer showing the text instead of html (with inlined images only) version of a message. html messages are shown in an iframe served from an endpoint with CSP headers to prevent dangerous resources (scripts, external images) from being loaded. the html is also sanitized, with javascript removed. a user can choose to load external resources (e.g. images for tracking purposes). the frontend is just (strict) typescript, no external frameworks. all incoming/outgoing data is typechecked, both the api request parameters and response types, and the data coming in over SSE. the types and checking code are generated with sherpats, which uses the api definitions generated by sherpadoc based on the Go code. so types from the backend are automatically propagated to the frontend. since there is no framework to automatically propagate properties and rerender components, changes coming in over the SSE connection are propagated explicitly with regular function calls. the ui is separated into "views", each with a "root" dom element that is added to the visible document. these views have additional functions for getting changes propagated, often resulting in the view updating its (internal) ui state (dom). we keep the frontend compilation simple, it's just a few typescript files that get compiled (combined and types stripped) into a single js file, no additional runtime code needed or complicated build processes used. the webmail is served is served from a compressed, cachable html file that includes style and the javascript, currently just over 225kb uncompressed, under 60kb compressed (not minified, including comments). we include the generated js files in the repository, to keep Go's easily buildable self-contained binaries. authentication is basic http, as with the account and admin pages. most data comes in over one long-term SSE connection to the backend. api requests signal which mailbox/search/messages are requested over the SSE connection. fetching individual messages, and making changes, are done through api calls. the operations are similar to imap, so some code has been moved from package imapserver to package store. the future jmap implementation will benefit from these changes too. more functionality will probably be moved to the store package in the future. the quickstart enables webmail on the internal listener by default (for new installs). users can enable it on the public listener if they want to. mox localserve enables it too. to enable webmail on existing installs, add settings like the following to the listeners in mox.conf, similar to AccountHTTP(S): WebmailHTTP: Enabled: true WebmailHTTPS: Enabled: true special thanks to liesbeth, gerben, andrii for early user feedback. there is plenty still to do, see the list at the top of webmail/webmail.ts. feedback welcome as always.
2023-08-07 22:57:03 +03:00
internal.WebmailHTTP.Enabled = true
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
internal.WebAPIHTTP.Enabled = true
replace http basic auth for web interfaces with session cookie & csrf-based auth the http basic auth we had was very simple to reason about, and to implement. but it has a major downside: there is no way to logout, browsers keep sending credentials. ideally, browsers themselves would show a button to stop sending credentials. a related downside: the http auth mechanism doesn't indicate for which server paths the credentials are. another downside: the original password is sent to the server with each request. though sending original passwords to web servers seems to be considered normal. our new approach uses session cookies, along with csrf values when we can. the sessions are server-side managed, automatically extended on each use. this makes it easy to invalidate sessions and keeps the frontend simpler (than with long- vs short-term sessions and refreshing). the cookies are httponly, samesite=strict, scoped to the path of the web interface. cookies are set "secure" when set over https. the cookie is set by a successful call to Login. a call to Logout invalidates a session. changing a password invalidates all sessions for a user, but keeps the session with which the password was changed alive. the csrf value is also random, and associated with the session cookie. the csrf must be sent as header for api calls, or as parameter for direct form posts (where we cannot set a custom header). rest-like calls made directly by the browser, e.g. for images, don't have a csrf protection. the csrf value is returned by the Login api call and stored in localstorage. api calls without credentials return code "user:noAuth", and with bad credentials return "user:badAuth". the api client recognizes this and triggers a login. after a login, all auth-failed api calls are automatically retried. only for "user:badAuth" is an error message displayed in the login form (e.g. session expired). in an ideal world, browsers would take care of most session management. a server would indicate authentication is needed (like http basic auth), and the browsers uses trusted ui to request credentials for the server & path. the browser could use safer mechanism than sending original passwords to the server, such as scram, along with a standard way to create sessions. for now, web developers have to do authentication themselves: from showing the login prompt, ensuring the right session/csrf cookies/localstorage/headers/etc are sent with each request. webauthn is a newer way to do authentication, perhaps we'll implement it in the future. though hardware tokens aren't an attractive option for many users, and it may be overkill as long as we still do old-fashioned authentication in smtp & imap where passwords can be sent to the server. for issue #58
2024-01-04 15:10:48 +03:00
internal.MetricsHTTP.Enabled = true
if existingWebserver {
internal.AccountHTTP.Port = 1080
replace http basic auth for web interfaces with session cookie & csrf-based auth the http basic auth we had was very simple to reason about, and to implement. but it has a major downside: there is no way to logout, browsers keep sending credentials. ideally, browsers themselves would show a button to stop sending credentials. a related downside: the http auth mechanism doesn't indicate for which server paths the credentials are. another downside: the original password is sent to the server with each request. though sending original passwords to web servers seems to be considered normal. our new approach uses session cookies, along with csrf values when we can. the sessions are server-side managed, automatically extended on each use. this makes it easy to invalidate sessions and keeps the frontend simpler (than with long- vs short-term sessions and refreshing). the cookies are httponly, samesite=strict, scoped to the path of the web interface. cookies are set "secure" when set over https. the cookie is set by a successful call to Login. a call to Logout invalidates a session. changing a password invalidates all sessions for a user, but keeps the session with which the password was changed alive. the csrf value is also random, and associated with the session cookie. the csrf must be sent as header for api calls, or as parameter for direct form posts (where we cannot set a custom header). rest-like calls made directly by the browser, e.g. for images, don't have a csrf protection. the csrf value is returned by the Login api call and stored in localstorage. api calls without credentials return code "user:noAuth", and with bad credentials return "user:badAuth". the api client recognizes this and triggers a login. after a login, all auth-failed api calls are automatically retried. only for "user:badAuth" is an error message displayed in the login form (e.g. session expired). in an ideal world, browsers would take care of most session management. a server would indicate authentication is needed (like http basic auth), and the browsers uses trusted ui to request credentials for the server & path. the browser could use safer mechanism than sending original passwords to the server, such as scram, along with a standard way to create sessions. for now, web developers have to do authentication themselves: from showing the login prompt, ensuring the right session/csrf cookies/localstorage/headers/etc are sent with each request. webauthn is a newer way to do authentication, perhaps we'll implement it in the future. though hardware tokens aren't an attractive option for many users, and it may be overkill as long as we still do old-fashioned authentication in smtp & imap where passwords can be sent to the server. for issue #58
2024-01-04 15:10:48 +03:00
internal.AccountHTTP.Forwarded = true
internal.AdminHTTP.Port = 1080
replace http basic auth for web interfaces with session cookie & csrf-based auth the http basic auth we had was very simple to reason about, and to implement. but it has a major downside: there is no way to logout, browsers keep sending credentials. ideally, browsers themselves would show a button to stop sending credentials. a related downside: the http auth mechanism doesn't indicate for which server paths the credentials are. another downside: the original password is sent to the server with each request. though sending original passwords to web servers seems to be considered normal. our new approach uses session cookies, along with csrf values when we can. the sessions are server-side managed, automatically extended on each use. this makes it easy to invalidate sessions and keeps the frontend simpler (than with long- vs short-term sessions and refreshing). the cookies are httponly, samesite=strict, scoped to the path of the web interface. cookies are set "secure" when set over https. the cookie is set by a successful call to Login. a call to Logout invalidates a session. changing a password invalidates all sessions for a user, but keeps the session with which the password was changed alive. the csrf value is also random, and associated with the session cookie. the csrf must be sent as header for api calls, or as parameter for direct form posts (where we cannot set a custom header). rest-like calls made directly by the browser, e.g. for images, don't have a csrf protection. the csrf value is returned by the Login api call and stored in localstorage. api calls without credentials return code "user:noAuth", and with bad credentials return "user:badAuth". the api client recognizes this and triggers a login. after a login, all auth-failed api calls are automatically retried. only for "user:badAuth" is an error message displayed in the login form (e.g. session expired). in an ideal world, browsers would take care of most session management. a server would indicate authentication is needed (like http basic auth), and the browsers uses trusted ui to request credentials for the server & path. the browser could use safer mechanism than sending original passwords to the server, such as scram, along with a standard way to create sessions. for now, web developers have to do authentication themselves: from showing the login prompt, ensuring the right session/csrf cookies/localstorage/headers/etc are sent with each request. webauthn is a newer way to do authentication, perhaps we'll implement it in the future. though hardware tokens aren't an attractive option for many users, and it may be overkill as long as we still do old-fashioned authentication in smtp & imap where passwords can be sent to the server. for issue #58
2024-01-04 15:10:48 +03:00
internal.AdminHTTP.Forwarded = true
add webmail it was far down on the roadmap, but implemented earlier, because it's interesting, and to help prepare for a jmap implementation. for jmap we need to implement more client-like functionality than with just imap. internal data structures need to change. jmap has lots of other requirements, so it's already a big project. by implementing a webmail now, some of the required data structure changes become clear and can be made now, so the later jmap implementation can do things similarly to the webmail code. the webmail frontend and webmail are written together, making their interface/api much smaller and simpler than jmap. one of the internal changes is that we now keep track of per-mailbox total/unread/unseen/deleted message counts and mailbox sizes. keeping this data consistent after any change to the stored messages (through the code base) is tricky, so mox now has a consistency check that verifies the counts are correct, which runs only during tests, each time an internal account reference is closed. we have a few more internal "changes" that are propagated for the webmail frontend (that imap doesn't have a way to propagate on a connection), like changes to the special-use flags on mailboxes, and used keywords in a mailbox. more changes that will be required have revealed themselves while implementing the webmail, and will be implemented next. the webmail user interface is modeled after the mail clients i use or have used: thunderbird, macos mail, mutt; and webmails i normally only use for testing: gmail, proton, yahoo, outlook. a somewhat technical user is assumed, but still the goal is to make this webmail client easy to use for everyone. the user interface looks like most other mail clients: a list of mailboxes, a search bar, a message list view, and message details. there is a top/bottom and a left/right layout for the list/message view, default is automatic based on screen size. the panes can be resized by the user. buttons for actions are just text, not icons. clicking a button briefly shows the shortcut for the action in the bottom right, helping with learning to operate quickly. any text that is underdotted has a title attribute that causes more information to be displayed, e.g. what a button does or a field is about. to highlight potential phishing attempts, any text (anywhere in the webclient) that switches unicode "blocks" (a rough approximation to (language) scripts) within a word is underlined orange. multiple messages can be selected with familiar ui interaction: clicking while holding control and/or shift keys. keyboard navigation works with arrows/page up/down and home/end keys, and also with a few basic vi-like keys for list/message navigation. we prefer showing the text instead of html (with inlined images only) version of a message. html messages are shown in an iframe served from an endpoint with CSP headers to prevent dangerous resources (scripts, external images) from being loaded. the html is also sanitized, with javascript removed. a user can choose to load external resources (e.g. images for tracking purposes). the frontend is just (strict) typescript, no external frameworks. all incoming/outgoing data is typechecked, both the api request parameters and response types, and the data coming in over SSE. the types and checking code are generated with sherpats, which uses the api definitions generated by sherpadoc based on the Go code. so types from the backend are automatically propagated to the frontend. since there is no framework to automatically propagate properties and rerender components, changes coming in over the SSE connection are propagated explicitly with regular function calls. the ui is separated into "views", each with a "root" dom element that is added to the visible document. these views have additional functions for getting changes propagated, often resulting in the view updating its (internal) ui state (dom). we keep the frontend compilation simple, it's just a few typescript files that get compiled (combined and types stripped) into a single js file, no additional runtime code needed or complicated build processes used. the webmail is served is served from a compressed, cachable html file that includes style and the javascript, currently just over 225kb uncompressed, under 60kb compressed (not minified, including comments). we include the generated js files in the repository, to keep Go's easily buildable self-contained binaries. authentication is basic http, as with the account and admin pages. most data comes in over one long-term SSE connection to the backend. api requests signal which mailbox/search/messages are requested over the SSE connection. fetching individual messages, and making changes, are done through api calls. the operations are similar to imap, so some code has been moved from package imapserver to package store. the future jmap implementation will benefit from these changes too. more functionality will probably be moved to the store package in the future. the quickstart enables webmail on the internal listener by default (for new installs). users can enable it on the public listener if they want to. mox localserve enables it too. to enable webmail on existing installs, add settings like the following to the listeners in mox.conf, similar to AccountHTTP(S): WebmailHTTP: Enabled: true WebmailHTTPS: Enabled: true special thanks to liesbeth, gerben, andrii for early user feedback. there is plenty still to do, see the list at the top of webmail/webmail.ts. feedback welcome as always.
2023-08-07 22:57:03 +03:00
internal.WebmailHTTP.Port = 1080
replace http basic auth for web interfaces with session cookie & csrf-based auth the http basic auth we had was very simple to reason about, and to implement. but it has a major downside: there is no way to logout, browsers keep sending credentials. ideally, browsers themselves would show a button to stop sending credentials. a related downside: the http auth mechanism doesn't indicate for which server paths the credentials are. another downside: the original password is sent to the server with each request. though sending original passwords to web servers seems to be considered normal. our new approach uses session cookies, along with csrf values when we can. the sessions are server-side managed, automatically extended on each use. this makes it easy to invalidate sessions and keeps the frontend simpler (than with long- vs short-term sessions and refreshing). the cookies are httponly, samesite=strict, scoped to the path of the web interface. cookies are set "secure" when set over https. the cookie is set by a successful call to Login. a call to Logout invalidates a session. changing a password invalidates all sessions for a user, but keeps the session with which the password was changed alive. the csrf value is also random, and associated with the session cookie. the csrf must be sent as header for api calls, or as parameter for direct form posts (where we cannot set a custom header). rest-like calls made directly by the browser, e.g. for images, don't have a csrf protection. the csrf value is returned by the Login api call and stored in localstorage. api calls without credentials return code "user:noAuth", and with bad credentials return "user:badAuth". the api client recognizes this and triggers a login. after a login, all auth-failed api calls are automatically retried. only for "user:badAuth" is an error message displayed in the login form (e.g. session expired). in an ideal world, browsers would take care of most session management. a server would indicate authentication is needed (like http basic auth), and the browsers uses trusted ui to request credentials for the server & path. the browser could use safer mechanism than sending original passwords to the server, such as scram, along with a standard way to create sessions. for now, web developers have to do authentication themselves: from showing the login prompt, ensuring the right session/csrf cookies/localstorage/headers/etc are sent with each request. webauthn is a newer way to do authentication, perhaps we'll implement it in the future. though hardware tokens aren't an attractive option for many users, and it may be overkill as long as we still do old-fashioned authentication in smtp & imap where passwords can be sent to the server. for issue #58
2024-01-04 15:10:48 +03:00
internal.WebmailHTTP.Forwarded = true
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
internal.WebAPIHTTP.Port = 1080
internal.WebAPIHTTP.Forwarded = true
internal.AutoconfigHTTPS.Enabled = true
internal.AutoconfigHTTPS.Port = 81
internal.AutoconfigHTTPS.NonTLS = true
internal.MTASTSHTTPS.Enabled = true
internal.MTASTSHTTPS.Port = 81
internal.MTASTSHTTPS.NonTLS = true
internal.WebserverHTTP.Enabled = true
internal.WebserverHTTP.Port = 81
}
2023-01-30 16:27:06 +03:00
sc.Listeners = map[string]config.Listener{
"public": public,
"internal": internal,
}
sc.Postmaster.Account = accountName
2023-01-30 16:27:06 +03:00
sc.Postmaster.Mailbox = "Postmaster"
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
sc.HostTLSRPT.Account = accountName
sc.HostTLSRPT.Localpart = "tls-reports"
sc.HostTLSRPT.Mailbox = "TLSRPT"
2023-01-30 16:27:06 +03:00
make mox compile on windows, without "mox serve" but with working "mox localserve" getting mox to compile required changing code in only a few places where package "syscall" was used: for accessing file access times and for umask handling. an open problem is how to start a process as an unprivileged user on windows. that's why "mox serve" isn't implemented yet. and just finding a way to implement it now may not be good enough in the near future: we may want to starting using a more complete privilege separation approach, with a process handling sensitive tasks (handling private keys, authentication), where we may want to pass file descriptors between processes. how would that work on windows? anyway, getting mox to compile for windows doesn't mean it works properly on windows. the largest issue: mox would normally open a file, rename or remove it, and finally close it. this happens during message delivery. that doesn't work on windows, the rename/remove would fail because the file is still open. so this commit swaps many "remove" and "close" calls. renames are a longer story: message delivery had two ways to deliver: with "consuming" the (temporary) message file (which would rename it to its final destination), and without consuming (by hardlinking the file, falling back to copying). the last delivery to a recipient of a message (and the only one in the common case of a single recipient) would consume the message, and the earlier recipients would not. during delivery, the already open message file was used, to parse the message. we still want to use that open message file, and the caller now stays responsible for closing it, but we no longer try to rename (consume) the file. we always hardlink (or copy) during delivery (this works on windows), and the caller is responsible for closing and removing (in that order) the original temporary file. this does cost one syscall more. but it makes the delivery code (responsibilities) a bit simpler. there is one more obvious issue: the file system path separator. mox already used the "filepath" package to join paths in many places, but not everywhere. and it still used strings with slashes for local file access. with this commit, the code now uses filepath.FromSlash for path strings with slashes, uses "filepath" in a few more places where it previously didn't. also switches from "filepath" to regular "path" package when handling mailbox names in a few places, because those always use forward slashes, regardless of local file system conventions. windows can handle forward slashes when opening files, so test code that passes path strings with forward slashes straight to go stdlib file i/o functions are left unchanged to reduce code churn. the regular non-test code, or test code that uses path strings in places other than standard i/o functions, does have the paths converted for consistent paths (otherwise we would end up with paths with mixed forward/backward slashes in log messages). windows cannot dup a listening socket. for "mox localserve", it isn't important, and we can work around the issue. the current approach for "mox serve" (forking a process and passing file descriptors of listening sockets on "privileged" ports) won't work on windows. perhaps it isn't needed on windows, and any user can listen on "privileged" ports? that would be welcome. on windows, os.Open cannot open a directory, so we cannot call Sync on it after message delivery. a cursory internet search indicates that directories cannot be synced on windows. the story is probably much more nuanced than that, with long deep technical details/discussions/disagreement/confusion, like on unix. for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
mox.ConfigStaticPath = filepath.FromSlash("config/mox.conf")
mox.ConfigDynamicPath = filepath.FromSlash("config/domains.conf")
2023-01-30 16:27:06 +03:00
mox.Conf.DynamicLastCheck = time.Now() // Prevent error logging by Make calls below.
accountConf := admin.MakeAccountConfig(addr)
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
const withMTASTS = true
confDomain, keyPaths, err := admin.MakeDomainConfig(context.Background(), domain, dnshostname, accountName, withMTASTS)
2023-01-30 16:27:06 +03:00
if err != nil {
fatalf("making domain config: %s", err)
}
cleanupPaths = append(cleanupPaths, keyPaths...)
dc.Domains = map[string]config.Domain{
domain.Name(): confDomain,
}
dc.Accounts = map[string]config.Account{
accountName: accountConf,
2023-01-30 16:27:06 +03:00
}
// Build config in memory, so we can easily comment out the DNSBLs config.
var sb strings.Builder
sc.CheckUpdates = true // Commented out below.
if err := sconf.WriteDocs(&sb, &sc); err != nil {
fatalf("generating static config: %v", err)
}
confstr := sb.String()
confstr = strings.ReplaceAll(confstr, "\nCheckUpdates: true\n", "\n#\n# RECOMMENDED: please enable to stay up to date\n#\n#CheckUpdates: true\n")
confstr = strings.ReplaceAll(confstr, "DNSBLs:\n", "#DNSBLs:\n")
for _, bl := range public.SMTP.DNSBLs {
confstr = strings.ReplaceAll(confstr, "- "+bl+"\n", "#- "+bl+"\n")
}
make mox compile on windows, without "mox serve" but with working "mox localserve" getting mox to compile required changing code in only a few places where package "syscall" was used: for accessing file access times and for umask handling. an open problem is how to start a process as an unprivileged user on windows. that's why "mox serve" isn't implemented yet. and just finding a way to implement it now may not be good enough in the near future: we may want to starting using a more complete privilege separation approach, with a process handling sensitive tasks (handling private keys, authentication), where we may want to pass file descriptors between processes. how would that work on windows? anyway, getting mox to compile for windows doesn't mean it works properly on windows. the largest issue: mox would normally open a file, rename or remove it, and finally close it. this happens during message delivery. that doesn't work on windows, the rename/remove would fail because the file is still open. so this commit swaps many "remove" and "close" calls. renames are a longer story: message delivery had two ways to deliver: with "consuming" the (temporary) message file (which would rename it to its final destination), and without consuming (by hardlinking the file, falling back to copying). the last delivery to a recipient of a message (and the only one in the common case of a single recipient) would consume the message, and the earlier recipients would not. during delivery, the already open message file was used, to parse the message. we still want to use that open message file, and the caller now stays responsible for closing it, but we no longer try to rename (consume) the file. we always hardlink (or copy) during delivery (this works on windows), and the caller is responsible for closing and removing (in that order) the original temporary file. this does cost one syscall more. but it makes the delivery code (responsibilities) a bit simpler. there is one more obvious issue: the file system path separator. mox already used the "filepath" package to join paths in many places, but not everywhere. and it still used strings with slashes for local file access. with this commit, the code now uses filepath.FromSlash for path strings with slashes, uses "filepath" in a few more places where it previously didn't. also switches from "filepath" to regular "path" package when handling mailbox names in a few places, because those always use forward slashes, regardless of local file system conventions. windows can handle forward slashes when opening files, so test code that passes path strings with forward slashes straight to go stdlib file i/o functions are left unchanged to reduce code churn. the regular non-test code, or test code that uses path strings in places other than standard i/o functions, does have the paths converted for consistent paths (otherwise we would end up with paths with mixed forward/backward slashes in log messages). windows cannot dup a listening socket. for "mox localserve", it isn't important, and we can work around the issue. the current approach for "mox serve" (forking a process and passing file descriptors of listening sockets on "privileged" ports) won't work on windows. perhaps it isn't needed on windows, and any user can listen on "privileged" ports? that would be welcome. on windows, os.Open cannot open a directory, so we cannot call Sync on it after message delivery. a cursory internet search indicates that directories cannot be synced on windows. the story is probably much more nuanced than that, with long deep technical details/discussions/disagreement/confusion, like on unix. for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
xwritefile(filepath.FromSlash("config/mox.conf"), []byte(confstr), 0660)
2023-01-30 16:27:06 +03:00
// Generate domains config, and add a commented out example for delivery to a mailing list.
var db bytes.Buffer
if err := sconf.WriteDocs(&db, &dc); err != nil {
fatalf("generating domains config: %v", err)
}
// This approach is a bit horrible, but it generates a convenient
// example that includes the comments. Though it is gone by the first
// write of the file by mox.
odests := fmt.Sprintf("\t\tDestinations:\n\t\t\t%s: nil\n", addr.String())
2023-01-30 16:27:06 +03:00
var destsExample = struct {
Destinations map[string]config.Destination
}{
Destinations: map[string]config.Destination{
addr.String(): {
2023-01-30 16:27:06 +03:00
Rulesets: []config.Ruleset{
{
VerifiedDomain: "list.example.org",
HeadersRegexp: map[string]string{
"^list-id$": `<name\.list\.example\.org>`,
},
ListAllowDomain: "list.example.org",
Mailbox: "Lists/Example",
},
},
},
},
}
var destBuf strings.Builder
if err := sconf.Describe(&destBuf, destsExample); err != nil {
fatalf("describing destination example: %v", err)
}
ndests := odests + "# If you receive email from mailing lists, you may want to configure them like the\n# example below (remove the empty/false SMTPMailRegexp and IsForward).\n# If you are receiving forwarded email, see the IsForwarded option in a Ruleset.\n"
2023-01-30 16:27:06 +03:00
for _, line := range strings.Split(destBuf.String(), "\n")[1:] {
ndests += "#\t\t" + line + "\n"
}
dconfstr := strings.ReplaceAll(db.String(), odests, ndests)
make mox compile on windows, without "mox serve" but with working "mox localserve" getting mox to compile required changing code in only a few places where package "syscall" was used: for accessing file access times and for umask handling. an open problem is how to start a process as an unprivileged user on windows. that's why "mox serve" isn't implemented yet. and just finding a way to implement it now may not be good enough in the near future: we may want to starting using a more complete privilege separation approach, with a process handling sensitive tasks (handling private keys, authentication), where we may want to pass file descriptors between processes. how would that work on windows? anyway, getting mox to compile for windows doesn't mean it works properly on windows. the largest issue: mox would normally open a file, rename or remove it, and finally close it. this happens during message delivery. that doesn't work on windows, the rename/remove would fail because the file is still open. so this commit swaps many "remove" and "close" calls. renames are a longer story: message delivery had two ways to deliver: with "consuming" the (temporary) message file (which would rename it to its final destination), and without consuming (by hardlinking the file, falling back to copying). the last delivery to a recipient of a message (and the only one in the common case of a single recipient) would consume the message, and the earlier recipients would not. during delivery, the already open message file was used, to parse the message. we still want to use that open message file, and the caller now stays responsible for closing it, but we no longer try to rename (consume) the file. we always hardlink (or copy) during delivery (this works on windows), and the caller is responsible for closing and removing (in that order) the original temporary file. this does cost one syscall more. but it makes the delivery code (responsibilities) a bit simpler. there is one more obvious issue: the file system path separator. mox already used the "filepath" package to join paths in many places, but not everywhere. and it still used strings with slashes for local file access. with this commit, the code now uses filepath.FromSlash for path strings with slashes, uses "filepath" in a few more places where it previously didn't. also switches from "filepath" to regular "path" package when handling mailbox names in a few places, because those always use forward slashes, regardless of local file system conventions. windows can handle forward slashes when opening files, so test code that passes path strings with forward slashes straight to go stdlib file i/o functions are left unchanged to reduce code churn. the regular non-test code, or test code that uses path strings in places other than standard i/o functions, does have the paths converted for consistent paths (otherwise we would end up with paths with mixed forward/backward slashes in log messages). windows cannot dup a listening socket. for "mox localserve", it isn't important, and we can work around the issue. the current approach for "mox serve" (forking a process and passing file descriptors of listening sockets on "privileged" ports) won't work on windows. perhaps it isn't needed on windows, and any user can listen on "privileged" ports? that would be welcome. on windows, os.Open cannot open a directory, so we cannot call Sync on it after message delivery. a cursory internet search indicates that directories cannot be synced on windows. the story is probably much more nuanced than that, with long deep technical details/discussions/disagreement/confusion, like on unix. for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
xwritefile(filepath.FromSlash("config/domains.conf"), []byte(dconfstr), 0660)
2023-01-30 16:27:06 +03:00
// Verify config.
loadTLSKeyCerts := !existingWebserver
mc, errs := mox.ParseConfig(context.Background(), c.log, filepath.FromSlash("config/mox.conf"), true, loadTLSKeyCerts, false)
2023-01-30 16:27:06 +03:00
if len(errs) > 0 {
if len(errs) > 1 {
log.Printf("checking generated config, multiple errors:")
for _, err := range errs {
log.Println(err)
}
fatalf("aborting due to multiple config errors")
}
fatalf("checking generated config: %s", errs[0])
}
mox.SetConfig(mc)
// NOTE: Now that we've prepared the config, we can open the account
// and set a passsword, and the public key for the DKIM private keys
// are available for generating the DKIM DNS records below.
confDomain, ok := mc.Domain(domain)
if !ok {
fatalf("cannot find domain in new config")
}
acc, _, err := store.OpenEmail(c.log, args[0])
2023-01-30 16:27:06 +03:00
if err != nil {
fatalf("open account: %s", err)
}
cleanupPaths = append(cleanupPaths, dataDir, filepath.Join(dataDir, "accounts"), filepath.Join(dataDir, "accounts", accountName), filepath.Join(dataDir, "accounts", accountName, "index.db"))
2023-01-30 16:27:06 +03:00
password := pwgen()
// Kludge to cause no logging to be printed about setting a new password.
loglevel := mox.Conf.Log[""]
mox.Conf.Log[""] = mlog.LevelWarn
mlog.SetConfig(mox.Conf.Log)
if err := acc.SetPassword(c.log, password); err != nil {
2023-01-30 16:27:06 +03:00
fatalf("setting password: %s", err)
}
mox.Conf.Log[""] = loglevel
mlog.SetConfig(mox.Conf.Log)
2023-01-30 16:27:06 +03:00
if err := acc.Close(); err != nil {
fatalf("closing account: %s", err)
}
fmt.Printf("IMAP, SMTP submission and HTTP account password for %s: %s\n\n", args[0], password)
fmt.Printf(`When configuring your email client, use the email address as username. If
autoconfig/autodiscover does not work, use these settings:
`)
2023-01-30 16:27:06 +03:00
printClientConfig(domain)
if existingWebserver {
fmt.Printf(`
Configuration files have been written to config/mox.conf and
config/domains.conf.
Create the DNS records below, by adding them to your zone file or through the
web interface of your DNS operator. The admin interface can show these same
records, and has a page to check they have been configured correctly.
You must configure your existing webserver to forward requests for:
https://mta-sts.%s/
https://autoconfig.%s/
To mox, at:
http://127.0.0.1:81
If it makes it easier to get a TLS certificate for %s, you can add a
reverse proxy for that hostname too.
You must edit mox.conf and configure the paths to the TLS certificates and keys.
The paths are relative to config/ directory that holds mox.conf! To test if your
config is valid, run:
./mox config test
The DNS records to add:
`, domain.ASCII, domain.ASCII, dnshostname.ASCII)
} else {
fmt.Printf(`
Configuration files have been written to config/mox.conf and
config/domains.conf. You should review them. Then create the DNS records below,
by adding them to your zone file or through the web interface of your DNS
operator. You can also skip creating the DNS records and start mox immediately.
The admin interface can show these same records, and has a page to check they
have been configured correctly. The DNS records to add:
`)
}
2023-01-30 16:27:06 +03:00
// We do not verify the records exist: If they don't exist, we would only be
// priming dns caches with negative/absent records, causing our "quick setup" to
// appear to fail or take longer than "quick".
records, err := admin.DomainRecords(confDomain, domain, domainDNSSECResult.Authentic, "letsencrypt.org", "")
2023-01-30 16:27:06 +03:00
if err != nil {
fatalf("making required DNS records")
}
fmt.Print("\n\n" + strings.Join(records, "\n") + "\n\n\n\n")
2023-01-30 16:27:06 +03:00
fmt.Printf(`WARNING: The configuration and DNS records above assume you do not currently
have email configured for your domain. If you do already have email configured,
or if you are sending email for your domain from other machines/services, you
should understand the consequences of the DNS records above before
continuing!
`)
if os.Getenv("MOX_DOCKER") == "" {
fmt.Printf(`
You can now start mox with "./mox serve", as root.
`)
} else {
fmt.Printf(`
You can now start the mox container.
`)
}
fmt.Printf(`
File ownership and permissions are automatically set correctly by mox when
starting up. On linux, you may want to enable mox as a systemd service.
2023-01-30 16:27:06 +03:00
`)
// For now, we only give service config instructions for linux when not running in docker.
if runtime.GOOS == "linux" && os.Getenv("MOX_DOCKER") == "" {
2023-01-30 16:27:06 +03:00
pwd, err := os.Getwd()
if err != nil {
log.Printf("current working directory: %v", err)
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
pwd = "/home/mox"
2023-01-30 16:27:06 +03:00
}
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
service := strings.ReplaceAll(moxService, "/home/mox", pwd)
2023-01-30 16:27:06 +03:00
xwritefile("mox.service", []byte(service), 0644)
cleanupPaths = append(cleanupPaths, "mox.service")
fmt.Printf(`See mox.service for a systemd service file. To enable and start:
sudo chmod 644 mox.service
sudo systemctl enable $PWD/mox.service
sudo systemctl start mox.service
sudo journalctl -f -u mox.service # See logs
`)
}
fmt.Printf(`
After starting mox, the web interfaces are served at:
add webmail it was far down on the roadmap, but implemented earlier, because it's interesting, and to help prepare for a jmap implementation. for jmap we need to implement more client-like functionality than with just imap. internal data structures need to change. jmap has lots of other requirements, so it's already a big project. by implementing a webmail now, some of the required data structure changes become clear and can be made now, so the later jmap implementation can do things similarly to the webmail code. the webmail frontend and webmail are written together, making their interface/api much smaller and simpler than jmap. one of the internal changes is that we now keep track of per-mailbox total/unread/unseen/deleted message counts and mailbox sizes. keeping this data consistent after any change to the stored messages (through the code base) is tricky, so mox now has a consistency check that verifies the counts are correct, which runs only during tests, each time an internal account reference is closed. we have a few more internal "changes" that are propagated for the webmail frontend (that imap doesn't have a way to propagate on a connection), like changes to the special-use flags on mailboxes, and used keywords in a mailbox. more changes that will be required have revealed themselves while implementing the webmail, and will be implemented next. the webmail user interface is modeled after the mail clients i use or have used: thunderbird, macos mail, mutt; and webmails i normally only use for testing: gmail, proton, yahoo, outlook. a somewhat technical user is assumed, but still the goal is to make this webmail client easy to use for everyone. the user interface looks like most other mail clients: a list of mailboxes, a search bar, a message list view, and message details. there is a top/bottom and a left/right layout for the list/message view, default is automatic based on screen size. the panes can be resized by the user. buttons for actions are just text, not icons. clicking a button briefly shows the shortcut for the action in the bottom right, helping with learning to operate quickly. any text that is underdotted has a title attribute that causes more information to be displayed, e.g. what a button does or a field is about. to highlight potential phishing attempts, any text (anywhere in the webclient) that switches unicode "blocks" (a rough approximation to (language) scripts) within a word is underlined orange. multiple messages can be selected with familiar ui interaction: clicking while holding control and/or shift keys. keyboard navigation works with arrows/page up/down and home/end keys, and also with a few basic vi-like keys for list/message navigation. we prefer showing the text instead of html (with inlined images only) version of a message. html messages are shown in an iframe served from an endpoint with CSP headers to prevent dangerous resources (scripts, external images) from being loaded. the html is also sanitized, with javascript removed. a user can choose to load external resources (e.g. images for tracking purposes). the frontend is just (strict) typescript, no external frameworks. all incoming/outgoing data is typechecked, both the api request parameters and response types, and the data coming in over SSE. the types and checking code are generated with sherpats, which uses the api definitions generated by sherpadoc based on the Go code. so types from the backend are automatically propagated to the frontend. since there is no framework to automatically propagate properties and rerender components, changes coming in over the SSE connection are propagated explicitly with regular function calls. the ui is separated into "views", each with a "root" dom element that is added to the visible document. these views have additional functions for getting changes propagated, often resulting in the view updating its (internal) ui state (dom). we keep the frontend compilation simple, it's just a few typescript files that get compiled (combined and types stripped) into a single js file, no additional runtime code needed or complicated build processes used. the webmail is served is served from a compressed, cachable html file that includes style and the javascript, currently just over 225kb uncompressed, under 60kb compressed (not minified, including comments). we include the generated js files in the repository, to keep Go's easily buildable self-contained binaries. authentication is basic http, as with the account and admin pages. most data comes in over one long-term SSE connection to the backend. api requests signal which mailbox/search/messages are requested over the SSE connection. fetching individual messages, and making changes, are done through api calls. the operations are similar to imap, so some code has been moved from package imapserver to package store. the future jmap implementation will benefit from these changes too. more functionality will probably be moved to the store package in the future. the quickstart enables webmail on the internal listener by default (for new installs). users can enable it on the public listener if they want to. mox localserve enables it too. to enable webmail on existing installs, add settings like the following to the listeners in mox.conf, similar to AccountHTTP(S): WebmailHTTP: Enabled: true WebmailHTTPS: Enabled: true special thanks to liesbeth, gerben, andrii for early user feedback. there is plenty still to do, see the list at the top of webmail/webmail.ts. feedback welcome as always.
2023-08-07 22:57:03 +03:00
http://localhost/ - account (email address as username)
http://localhost/webmail/ - webmail (email address as username)
http://localhost/admin/ - admin (empty username)
To access these from your browser, run
"ssh -L 8080:localhost:80 you@yourmachine" locally and open
http://localhost:8080/[...].
If you run into problem, have questions/feedback or found a bug, please let us
know. Mox needs your help!
Enjoy!
`)
if !existingWebserver {
fmt.Printf(`
PS: If you want to run mox along side an existing webserver that uses port 443
and 80, see "mox help quickstart" with the -existing-webserver option.
`)
}
2023-01-30 16:27:06 +03:00
cleanupPaths = nil
}