serv/mox
1
0
Fork 0
mirror of https://github.com/mjl-/mox.git synced 2024-12-27 08:53:48 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions
mox/config/config.go

580 lines
61 KiB
Go
Raw Normal View History

mox!
2023-01-30 16:27:06 +03:00
package config
import (
"crypto"
"crypto/tls"
"crypto/x509"
"net"
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
"net/url"
in account web page, allow user to configure rulesets for delivery for example, by matching incoming messags on smtp mail from, verified domains (spf/dkim), headers. then delivering to a configured mailbox. for mailing lists, if a verified domain matches, regular spam checks can be skipped. this was already possible by editing the configuration file, but only admins can edit that file. now users can manage their own rulesets.
2023-02-11 01:47:19 +03:00
"reflect"
mox!
2023-01-30 16:27:06 +03:00
"regexp"
"time"
"github.com/mjl-/mox/autotls"
"github.com/mjl-/mox/dns"
"github.com/mjl-/mox/junk"
"github.com/mjl-/mox/mtasts"
"github.com/mjl-/mox/smtp"
)
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
// todo: better default values, so less has to be specified in the config file.
mox!
2023-01-30 16:27:06 +03:00
add webmail it was far down on the roadmap, but implemented earlier, because it's interesting, and to help prepare for a jmap implementation. for jmap we need to implement more client-like functionality than with just imap. internal data structures need to change. jmap has lots of other requirements, so it's already a big project. by implementing a webmail now, some of the required data structure changes become clear and can be made now, so the later jmap implementation can do things similarly to the webmail code. the webmail frontend and webmail are written together, making their interface/api much smaller and simpler than jmap. one of the internal changes is that we now keep track of per-mailbox total/unread/unseen/deleted message counts and mailbox sizes. keeping this data consistent after any change to the stored messages (through the code base) is tricky, so mox now has a consistency check that verifies the counts are correct, which runs only during tests, each time an internal account reference is closed. we have a few more internal "changes" that are propagated for the webmail frontend (that imap doesn't have a way to propagate on a connection), like changes to the special-use flags on mailboxes, and used keywords in a mailbox. more changes that will be required have revealed themselves while implementing the webmail, and will be implemented next. the webmail user interface is modeled after the mail clients i use or have used: thunderbird, macos mail, mutt; and webmails i normally only use for testing: gmail, proton, yahoo, outlook. a somewhat technical user is assumed, but still the goal is to make this webmail client easy to use for everyone. the user interface looks like most other mail clients: a list of mailboxes, a search bar, a message list view, and message details. there is a top/bottom and a left/right layout for the list/message view, default is automatic based on screen size. the panes can be resized by the user. buttons for actions are just text, not icons. clicking a button briefly shows the shortcut for the action in the bottom right, helping with learning to operate quickly. any text that is underdotted has a title attribute that causes more information to be displayed, e.g. what a button does or a field is about. to highlight potential phishing attempts, any text (anywhere in the webclient) that switches unicode "blocks" (a rough approximation to (language) scripts) within a word is underlined orange. multiple messages can be selected with familiar ui interaction: clicking while holding control and/or shift keys. keyboard navigation works with arrows/page up/down and home/end keys, and also with a few basic vi-like keys for list/message navigation. we prefer showing the text instead of html (with inlined images only) version of a message. html messages are shown in an iframe served from an endpoint with CSP headers to prevent dangerous resources (scripts, external images) from being loaded. the html is also sanitized, with javascript removed. a user can choose to load external resources (e.g. images for tracking purposes). the frontend is just (strict) typescript, no external frameworks. all incoming/outgoing data is typechecked, both the api request parameters and response types, and the data coming in over SSE. the types and checking code are generated with sherpats, which uses the api definitions generated by sherpadoc based on the Go code. so types from the backend are automatically propagated to the frontend. since there is no framework to automatically propagate properties and rerender components, changes coming in over the SSE connection are propagated explicitly with regular function calls. the ui is separated into "views", each with a "root" dom element that is added to the visible document. these views have additional functions for getting changes propagated, often resulting in the view updating its (internal) ui state (dom). we keep the frontend compilation simple, it's just a few typescript files that get compiled (combined and types stripped) into a single js file, no additional runtime code needed or complicated build processes used. the webmail is served is served from a compressed, cachable html file that includes style and the javascript, currently just over 225kb uncompressed, under 60kb compressed (not minified, including comments). we include the generated js files in the repository, to keep Go's easily buildable self-contained binaries. authentication is basic http, as with the account and admin pages. most data comes in over one long-term SSE connection to the backend. api requests signal which mailbox/search/messages are requested over the SSE connection. fetching individual messages, and making changes, are done through api calls. the operations are similar to imap, so some code has been moved from package imapserver to package store. the future jmap implementation will benefit from these changes too. more functionality will probably be moved to the store package in the future. the quickstart enables webmail on the internal listener by default (for new installs). users can enable it on the public listener if they want to. mox localserve enables it too. to enable webmail on existing installs, add settings like the following to the listeners in mox.conf, similar to AccountHTTP(S): WebmailHTTP: Enabled: true WebmailHTTPS: Enabled: true special thanks to liesbeth, gerben, andrii for early user feedback. there is plenty still to do, see the list at the top of webmail/webmail.ts. feedback welcome as always.
2023-08-07 22:57:03 +03:00
// DefaultMaxMsgSize is the maximum message size for incoming and outgoing
// messages, in bytes. Can be overridden per listener.
const DefaultMaxMsgSize = 100 * 1024 * 1024
mox!
2023-01-30 16:27:06 +03:00
// Port returns port if non-zero, and fallback otherwise.
func Port(port, fallback int) int {
if port == 0 {
return fallback
}
return port
}
// Static is a parsed form of the mox.conf configuration file, before converting it
// into a mox.Config after additional processing.
type Static struct {
add comment about the sconf config file format at the top of the config files hopefully this helps admins editing the file and prevent mistakes about config files. for issue #56 by kikoreis, thanks!
2023-09-21 09:59:10 +03:00
DataDir string `sconf-doc:"NOTE: This config file is in 'sconf' format. Indent with tabs. Comments must be on their own line, they don't end a line. Do not escape or quote strings. Details: https://pkg.go.dev/github.com/mjl-/sconf.\n\n\nDirectory where all data is stored, e.g. queue, accounts and messages, ACME TLS certs/keys. If this is a relative path, it is relative to the directory of mox.conf."`
add two new log levels for tracing sensitive auth protocol messages, and bulk data messages named "traceauth" and "tracedata". with this, you can (almost) enable trace logging without fear of logging sensitive data or ddos'ing your log server. the caveat is that the imap login command has already printed the line as regular trace before we can decide it should not be. can be fixed soon.
2023-02-03 22:33:19 +03:00
LogLevel string `sconf-doc:"Default log level, one of: error, info, debug, trace, traceauth, tracedata. Trace logs SMTP and IMAP protocol transcripts, with traceauth also messages with passwords, and tracedata on top of that also the full data exchanges (full messages), which can be a large amount of data."`
mox!
2023-01-30 16:27:06 +03:00
PackageLogLevels map[string]string `sconf:"optional" sconf-doc:"Overrides of log level per package (e.g. queue, smtpclient, smtpserver, imapserver, spf, dkim, dmarc, dmarcdb, autotls, junk, mtasts, tlsrpt)."`
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
User string `sconf:"optional" sconf-doc:"User to switch to after binding to all sockets as root. Default: mox. If the value is not a known user, it is parsed as integer and used as uid and gid."`
NoFixPermissions bool `sconf:"optional" sconf-doc:"If true, do not automatically fix file permissions when starting up. By default, mox will ensure reasonable owner/permissions on the working, data and config directories (and files), and mox binary (if present)."`
mox!
2023-01-30 16:27:06 +03:00
Hostname string `sconf-doc:"Full hostname of system, e.g. mail.<domain>"`
HostnameDomain dns.Domain `sconf:"-" json:"-"` // Parsed form of hostname.
implement accepting dmarc & tls reports for other domains to accept reports for another domain, first add that domain to the config, leaving all options empty except DMARC/TLSRPT in which you configure a Domain. the suggested DNS DMARC/TLSRPT records will show the email address with configured domain. for DMARC, the dnscheck functionality will verify that the destination domain has opted in to receiving reports. there is a new command-line subcommand "mox dmarc checkreportaddrs" that verifies if dmarc reporting destination addresses have opted in to received reports. this also changes the suggested dns records (in quickstart, and through admin pages and cli subcommand) to take into account whether DMARC and TLSRPT is configured, and with which localpart/domain (previously it always printed records as if reporting was enabled for the domain). and when generating the suggested DNS records, the dmarc.Record and tlsrpt.Record code is used, with proper uri-escaping.
2023-08-23 15:27:21 +03:00
CheckUpdates bool `sconf:"optional" sconf-doc:"If enabled, a single DNS TXT lookup of _updates.xmox.nl is done every 24h to check for a new release. Each time a new release is found, a changelog is fetched from https://updates.xmox.nl/changelog and delivered to the postmaster mailbox."`
add pedantic mode (used by localserve) that refuses some behaviour that is invalid according to specifications and that we normally accept for compatibility
2023-03-12 17:16:01 +03:00
Pedantic bool `sconf:"optional" sconf-doc:"In pedantic mode protocol violations (that happen in the wild) for SMTP/IMAP/etc result in errors instead of accepting such behaviour."`
mox!
2023-01-30 16:27:06 +03:00
TLS struct {
CA *struct {
AdditionalToSystem bool `sconf:"optional"`
CertFiles []string `sconf:"optional"`
} `sconf:"optional"`
CertPool *x509.CertPool `sconf:"-" json:"-"`
use configured tls ca config for all tls connections, so https as well and add documentation for developers for setting up certificates with manual local CA (with cfssl) or local ACME CA (with pebble).
2023-03-10 18:25:18 +03:00
} `sconf:"optional" sconf-doc:"Global TLS configuration, e.g. for additional Certificate Authorities. Used for outgoing SMTP connections, HTTPS requests."`
mox!
2023-01-30 16:27:06 +03:00
ACME map[string]ACME `sconf:"optional" sconf-doc:"Automatic TLS configuration with ACME, e.g. through Let's Encrypt. The key is a name referenced in TLS configs, e.g. letsencrypt."`
AdminPasswordFile string `sconf:"optional" sconf-doc:"File containing hash of admin password, for authentication in the web admin pages (if enabled)."`
at startup, with acme, if the config has explicitly configured public ips (the default with the quickstart), lookup the host names allowed for acme validation and warn about ips that mox is not configured to listen on i've seen this cause acme validation failures 3 times now, so give a hint in the logs to new users. also for issue #13.
2023-03-05 18:22:23 +03:00
Listeners map[string]Listener `sconf-doc:"Listeners are groups of IP addresses and services enabled on those IP addresses, such as SMTP/IMAP or internal endpoints for administration or Prometheus metrics. All listeners with SMTP/IMAP services enabled will serve all configured domains. If the listener is named 'public', it will get a few helpful additional configuration checks, for acme automatic tls certificates and monitoring of ips in dnsbls if those are configured."`
mox!
2023-01-30 16:27:06 +03:00
Postmaster struct {
Account string
Mailbox string `sconf-doc:"E.g. Postmaster or Inbox."`
in smtpserver, accept delivery to postmaster@<hostname>, and also postmaster@ addresses for domains that don't have a postmaster address configured.
2023-04-24 13:04:46 +03:00
} `sconf-doc:"Destination for emails delivered to postmaster addresses: a plain 'postmaster' without domain, 'postmaster@<hostname>' (also for each listener with SMTP enabled), and as fallback for each domain without explicitly configured postmaster destination."`
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
HostTLSRPT struct {
Account string `sconf-doc:"Account to deliver TLS reports to. Typically same account as for postmaster."`
Mailbox string `sconf-doc:"Mailbox to deliver TLS reports to. Recommended value: TLSRPT."`
Localpart string `sconf-doc:"Localpart at hostname to accept TLS reports at. Recommended value: tls-reports."`
ParsedLocalpart smtp.Localpart `sconf:"-"`
} `sconf:"optional" sconf-doc:"Destination for per-host TLS reports (TLSRPT). TLS reports can be per recipient domain (for MTA-STS), or per MX host (for DANE). The per-domain TLS reporting configuration is in domains.conf. This is the TLS reporting configuration for this host. If absent, no host-based TLSRPT address is configured, and no host TLSRPT DNS record is suggested."`
better handling of outgoing tls reports to recipient domains vs hosts based on discussion on uta mailing list. it seems the intention of the tlsrpt is to only send reports to recipient domains. but i was able to interpret the tlsrpt rfc as sending reports to mx hosts too ("policy domain", and because it makes sense given how DANE works per MX host, not recipient domain). this change makes the behaviour of outgoing reports to recipient domains work more in line with expectations most folks may have about tls reporting (i.e. also include per-mx host tlsa policies in the report). this also keeps reports to mx hosts working, and makes them more useful by including the recipient domains of affected deliveries.
2023-11-20 13:31:46 +03:00
InitialMailboxes InitialMailboxes `sconf:"optional" sconf-doc:"Mailboxes to create for new accounts. Inbox is always created. Mailboxes can be given a 'special-use' role, which are understood by most mail clients. If absent/empty, the following mailboxes are created: Sent, Archive, Trash, Drafts and Junk."`
DefaultMailboxes []string `sconf:"optional" sconf-doc:"Deprecated in favor of InitialMailboxes. Mailboxes to create when adding an account. Inbox is always created. If no mailboxes are specified, the following are automatically created: Sent, Archive, Trash, Drafts and Junk."`
Transports map[string]Transport `sconf:"optional" sconf-doc:"Transport are mechanisms for delivering messages. Transports can be referenced from Routes in accounts, domains and the global configuration. There is always an implicit/fallback delivery transport doing direct delivery with SMTP from the outgoing message queue. Transports are typically only configured when using smarthosts, i.e. when delivering through another SMTP server. Zero or one transport methods must be set in a transport, never multiple. When using an external party to send email for a domain, keep in mind you may have to add their IP address to your domain's SPF record, and possibly additional DKIM records."`
// Awkward naming of fields to get intended default behaviour for zero values.
add per-account quota for total message size disk usage so a single user cannot fill up the disk. by default, there is (still) no limit. a default can be set in the config file for all accounts, and a per-account max size can be set that would override any global setting. this does not take into account disk usage of the index database. and also not of any file system overhead.
2023-12-20 22:54:12 +03:00
NoOutgoingDMARCReports bool `sconf:"optional" sconf-doc:"Do not send DMARC reports (aggregate only). By default, aggregate reports on DMARC evaluations are sent to domains if their DMARC policy requests them. Reports are sent at whole hours, with a minimum of 1 hour and maximum of 24 hours, rounded up so a whole number of intervals cover 24 hours, aligned at whole days in UTC. Reports are sent from the postmaster@<mailhostname> address."`
NoOutgoingTLSReports bool `sconf:"optional" sconf-doc:"Do not send TLS reports. By default, reports about failed SMTP STARTTLS connections and related MTA-STS/DANE policies are sent to domains if their TLSRPT DNS record requests them. Reports covering a 24 hour UTC interval are sent daily. Reports are sent from the postmaster address of the configured domain the mailhostname is in. If there is no such domain, or it does not have DKIM configured, no reports are sent."`
OutgoingTLSReportsForAllSuccess bool `sconf:"optional" sconf-doc:"Also send TLS reports if there were no SMTP STARTTLS connection failures. By default, reports are only sent when at least one failure occurred. If a report is sent, it does always include the successful connection counts as well."`
quota: fix handling negative max size when configured for an account, and clarify value is in bytes in config file for #115 by pmarini-nc
2024-01-12 17:02:16 +03:00
QuotaMessageSize int64 `sconf:"optional" sconf-doc:"Default maximum total message size in bytes for each individual account, only applicable if greater than zero. Can be overridden per account. Attempting to add new messages to an account beyond its maximum total size will result in an error. Useful to prevent a single account from filling storage. The quota only applies to the email message files, not to any file system overhead and also not the message index database file (account for approximately 15% overhead)."`
mox!
2023-01-30 16:27:06 +03:00
improve docs about IPs and ipv4/ipv6 used for outgoing connections based on feedback from alex on irc, thanks!
2024-04-21 18:22:00 +03:00
// All IPs that were explicitly listened on for external SMTP. Only set when there
add reverse ip checks during quickstart and in "check dns" admin page/subcommand - and don't have a global variable "d" in the big checkDomain function in http/admin.go. - and set loglevel from command-line flag again after loading the config file, for all subcommands except "serve".
2023-02-03 17:54:34 +03:00
// are no unspecified external SMTP listeners and there is at most one for IPv4 and
mox!
2023-01-30 16:27:06 +03:00
// at most one for IPv6. Used for setting the local address when making outgoing
// connections. Those IPs are assumed to be in an SPF record for the domain,
// potentially unlike other IPs on the machine. If there is only one address
// family, outgoing connections with the other address family are still made if
// possible.
SpecifiedSMTPListenIPs []net.IP `sconf:"-" json:"-"`
change mox to start as root, bind to network sockets, then drop to regular unprivileged mox user makes it easier to run on bsd's, where you cannot (easily?) let non-root users bind to ports <1024. starting as root also paves the way for future improvements with privilege separation. unfortunately, this requires changes to how you start mox. though mox will help by automatically fix up dir/file permissions/ownership. if you start mox from the systemd unit file, you should update it so it starts as root and adds a few additional capabilities: # first update the mox binary, then, as root: ./mox config printservice >mox.service systemctl daemon-reload systemctl restart mox journalctl -f -u mox & # you should see mox start up, with messages about fixing permissions on dirs/files. if you used the recommended config/ and data/ directory, in a directory just for mox, and with the mox user called "mox", this should be enough. if you don't want mox to modify dir/file permissions, set "NoFixPermissions: true" in mox.conf. if you named the mox user something else than mox, e.g. "_mox", add "User: _mox" to mox.conf. if you created a shared service user as originally suggested, you may want to get rid of that as it is no longer useful and may get in the way. e.g. if you had /home/service/mox with a "service" user, that service user can no longer access any files: only mox and root can. this also adds scripts for building mox docker images for alpine-supported platforms. the "restart" subcommand has been removed. it wasn't all that useful and got in the way. and another change: when adding a domain while mtasts isn't enabled, don't add the per-domain mtasts config, as it would cause failure to add the domain. based on report from setting up mox on openbsd from mteege. and based on issue #3. thanks for the feedback!
2023-02-27 14:19:55 +03:00
// To switch to after initialization as root.
UID uint32 `sconf:"-" json:"-"`
GID uint32 `sconf:"-" json:"-"`
mox!
2023-01-30 16:27:06 +03:00
}
remove last remnants of treating a mailbox named "Sent" specially, in favor of special-use mailbox flags a few places still looked at the name "Sent". but since we have special-use flags, we should always look at those. this also changes the config so admins can specify different names for the special-use mailboxes to create for new accounts, e.g. in a different language. the old config option is still understood, just deprecated.
2023-08-09 10:31:23 +03:00
// InitialMailboxes are mailboxes created for a new account.
type InitialMailboxes struct {
SpecialUse SpecialUseMailboxes `sconf:"optional" sconf-doc:"Special-use roles to mailbox to create."`
Regular []string `sconf:"optional" sconf-doc:"Regular, non-special-use mailboxes to create."`
}
// SpecialUseMailboxes holds mailbox names for special-use roles. Mail clients
// recognize these special-use roles, e.g. appending sent messages to whichever
// mailbox has the Sent special-use flag.
type SpecialUseMailboxes struct {
Sent string `sconf:"optional"`
Archive string `sconf:"optional"`
Trash string `sconf:"optional"`
Draft string `sconf:"optional"`
Junk string `sconf:"optional"`
}
mox!
2023-01-30 16:27:06 +03:00
// Dynamic is the parsed form of domains.conf, and is automatically reloaded when changed.
type Dynamic struct {
add comment about the sconf config file format at the top of the config files hopefully this helps admins editing the file and prevent mistakes about config files. for issue #56 by kikoreis, thanks!
2023-09-21 09:59:10 +03:00
Domains map[string]Domain `sconf-doc:"NOTE: This config file is in 'sconf' format. Indent with tabs. Comments must be on their own line, they don't end a line. Do not escape or quote strings. Details: https://pkg.go.dev/github.com/mjl-/sconf.\n\n\nDomains for which email is accepted. For internationalized domains, use their IDNA names in UTF-8."`
allow zero configured addresses for an account preventing writing out a domains.conf that is invalid and can't be parsed again. this happens when the last address was removed from an account. just a click in the admin web interface. accounts without email address cannot log in. for issue #133 by ally9335
2024-03-09 13:51:02 +03:00
Accounts map[string]Account `sconf-doc:"Accounts represent mox users, each with a password and email address(es) to which email can be delivered (possibly at different domains). Each account has its own on-disk directory holding its messages and index database. An account name is not an email address."`
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config - make builtin http handlers serve on specific domains, such as for mta-sts, so e.g. /.well-known/mta-sts.txt isn't served on all domains. - add logging of a few more fields in access logging. - small tweaks/bug fixes in webserver request handling. - add config option for redirecting entire domains to another (common enough). - split httpserver metric into two: one for duration until writing header (i.e. performance of server), another for duration until full response is sent to client (i.e. performance as perceived by users). - add admin ui, a new page for managing the configs. after making changes and hitting "save", the changes take effect immediately. the page itself doesn't look very well-designed (many input fields, makes it look messy). i have an idea to improve it (explained in admin.html as todo) by making the layout look just like the config file. not urgent though. i've already changed my websites/webapps over. the idea of adding a webserver is to take away a (the) reason for folks to want to complicate their mox setup by running an other webserver on the same machine. i think the current webserver implementation can already serve most common use cases. with a few more tweaks (feedback needed!) we should be able to get to 95% of the use cases. the reverse proxy can take care of the remaining 5%. nevertheless, a next step is still to change the quickstart to make it easier for folks to run with an existing webserver, with existing tls certs/keys. that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
WebDomainRedirects map[string]string `sconf:"optional" sconf-doc:"Redirect all requests from domain (key) to domain (value). Always redirects to HTTPS. For plain HTTP redirects, use a WebHandler with a WebRedirect."`
for WebRedirect, don't "match" when the destination URL has the same scheme,host,path, for doing http -> https redirects without loops you can already get most http to https redirects through DontRedirectPlainHTTP in WebHandler, but that needs handlers for all paths. now you can just set up a redirect for a domain and all its path to baseurl https://domain (leaving other webdirect fields empty). when the request comes in with plain http, the redirect to https is done. that next request will also evaluate the same redirect rule. but it will not cause a match because it would redirect to the same scheme,host,path. so next webhandlers get a chance to serve. also clarify in webhandlers docs that also account & admin built-in handlers run first. related to issue #16
2023-03-09 01:29:44 +03:00
WebHandlers []WebHandler `sconf:"optional" sconf-doc:"Handle webserver requests by serving static files, redirecting or reverse-proxying HTTP(s). The first matching WebHandler will handle the request. Built-in handlers, e.g. for account, admin, autoconfig and mta-sts always run first. If no handler matches, the response status code is file not found (404). If functionality you need is missng, simply forward the requests to an application that can provide the needed functionality."`
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
Routes []Route `sconf:"optional" sconf-doc:"Routes for delivering outgoing messages through the queue. Each delivery attempt evaluates account routes, domain routes and finally these global routes. The transport of the first matching route is used in the delivery attempt. If no routes match, which is the default with no configured routes, messages are delivered directly from the queue."`
implement only monitoring dns blocklists, without using them for incoming deliveries so you can still know when someone has put you on their blocklist (which may affect delivery), without using them. also query dnsbls for our ips more often when we do more outgoing connections for delivery: once every 100 messages, but at least 5 mins and at most 3 hours since the previous check.
2024-03-05 18:30:38 +03:00
MonitorDNSBLs []string `sconf:"optional" sconf-doc:"DNS blocklists to periodically check with if IPs we send from are present, without using them for checking incoming deliveries.. Also see DNSBLs in SMTP listeners in mox.conf, which specifies DNSBLs to use both for incoming deliveries and for checking our IPs against. Example DNSBLs: sbl.spamhaus.org, bl.spamcop.net."`
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config - make builtin http handlers serve on specific domains, such as for mta-sts, so e.g. /.well-known/mta-sts.txt isn't served on all domains. - add logging of a few more fields in access logging. - small tweaks/bug fixes in webserver request handling. - add config option for redirecting entire domains to another (common enough). - split httpserver metric into two: one for duration until writing header (i.e. performance of server), another for duration until full response is sent to client (i.e. performance as perceived by users). - add admin ui, a new page for managing the configs. after making changes and hitting "save", the changes take effect immediately. the page itself doesn't look very well-designed (many input fields, makes it look messy). i have an idea to improve it (explained in admin.html as todo) by making the layout look just like the config file. not urgent though. i've already changed my websites/webapps over. the idea of adding a webserver is to take away a (the) reason for folks to want to complicate their mox setup by running an other webserver on the same machine. i think the current webserver implementation can already serve most common use cases. with a few more tweaks (feedback needed!) we should be able to get to 95% of the use cases. the reverse proxy can take care of the remaining 5%. nevertheless, a next step is still to change the quickstart to make it easier for folks to run with an existing webserver, with existing tls certs/keys. that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
change javascript into typescript for webaccount and webadmin interface all ui frontend code is now in typescript. we no longer need jshint, and we build the frontend code during "make build". this also changes tlsrpt types for a Report, not encoding field names with dashes, but to keep them valid identifiers in javascript. this makes it more conveniently to work with in the frontend, and works around a sherpats limitation.
2023-12-31 13:55:22 +03:00
WebDNSDomainRedirects map[dns.Domain]dns.Domain `sconf:"-" json:"-"`
implement only monitoring dns blocklists, without using them for incoming deliveries so you can still know when someone has put you on their blocklist (which may affect delivery), without using them. also query dnsbls for our ips more often when we do more outgoing connections for delivery: once every 100 messages, but at least 5 mins and at most 3 hours since the previous check.
2024-03-05 18:30:38 +03:00
MonitorDNSBLZones []dns.Domain `sconf:"-"`
mox!
2023-01-30 16:27:06 +03:00
}
type ACME struct {
implement ACME external account binding (EAB) where a new acme account is created with a reference to an existing non-acme account known by the acme provider. some acme providers require this.
2023-12-22 12:34:55 +03:00
DirectoryURL string `sconf-doc:"For letsencrypt, use https://acme-v02.api.letsencrypt.org/directory."`
RenewBefore time.Duration `sconf:"optional" sconf-doc:"How long before expiration to renew the certificate. Default is 30 days."`
ContactEmail string `sconf-doc:"Email address to register at ACME provider. The provider can email you when certificates are about to expire. If you configure an address for which email is delivered by this server, keep in mind that TLS misconfigurations could result in such notification emails not arriving."`
Port int `sconf:"optional" sconf-doc:"TLS port for ACME validation, 443 by default. You should only override this if you cannot listen on port 443 directly. ACME will make requests to port 443, so you'll have to add an external mechanism to get the connection here, e.g. by configuring port forwarding."`
IssuerDomainName string `sconf:"optional" sconf-doc:"If set, used for suggested CAA DNS records, for restricting TLS certificate issuance to a Certificate Authority. If empty and DirectyURL is for Let's Encrypt, this value is set automatically to letsencrypt.org."`
ExternalAccountBinding *ExternalAccountBinding `sconf:"optional" sconf-doc:"ACME providers can require that a request for a new ACME account reference an existing non-ACME account known to the provider. External account binding references that account by a key id, and authorizes new ACME account requests by signing it with a key known both by the ACME client and ACME provider."`
// ../rfc/8555:2111
mox!
2023-01-30 16:27:06 +03:00
Manager *autotls.Manager `sconf:"-" json:"-"`
}
implement ACME external account binding (EAB) where a new acme account is created with a reference to an existing non-acme account known by the acme provider. some acme providers require this.
2023-12-22 12:34:55 +03:00
type ExternalAccountBinding struct {
KeyID string `sconf-doc:"Key identifier, from ACME provider."`
KeyFile string `sconf-doc:"File containing the base64url-encoded key used to sign account requests with external account binding. The ACME provider will verify the account request is correctly signed by the key. File is evaluated relative to the directory of mox.conf."`
}
mox!
2023-01-30 16:27:06 +03:00
type Listener struct {
improve docs about IPs and ipv4/ipv6 used for outgoing connections based on feedback from alex on irc, thanks!
2024-04-21 18:22:00 +03:00
IPs []string `sconf-doc:"Use 0.0.0.0 to listen on all IPv4 and/or :: to listen on all IPv6 addresses, but it is better to explicitly specify the IPs you want to use for email, as mox will make sure outgoing connections will only be made from one of those IPs. If both outgoing IPv4 and IPv6 connectivity is possible, and only one family has explicitly configured addresses, both address families are still used for outgoing connections. Use the \"direct\" transport to limit address families for outgoing connections."`
replace listener config option IPsNATed with NATIPs, and let autotls check NATIPs NATIPs lists the public IPs, so we can still do the DNS checks on them. with IPsNATed, we disabled the checks. based on feedback by kikoreis in issue #52
2023-08-11 11:13:17 +03:00
NATIPs []string `sconf:"optional" sconf-doc:"If set, the mail server is configured behind a NAT and field IPs are internal instead of the public IPs, while NATIPs lists the public IPs. Used during IP-related DNS self-checks, such as for iprev, mx, spf, autoconfig, autodiscover, and for autotls."`
IPsNATed bool `sconf:"optional" sconf-doc:"Deprecated, use NATIPs instead. If set, IPs are not the public IPs, but are NATed. Skips IP-related DNS self-checks."`
mox!
2023-01-30 16:27:06 +03:00
Hostname string `sconf:"optional" sconf-doc:"If empty, the config global Hostname is used."`
HostnameDomain dns.Domain `sconf:"-" json:"-"` // Set when parsing config.
TLS *TLS `sconf:"optional" sconf-doc:"For SMTP/IMAP STARTTLS, direct TLS and HTTPS connections."`
add webmail it was far down on the roadmap, but implemented earlier, because it's interesting, and to help prepare for a jmap implementation. for jmap we need to implement more client-like functionality than with just imap. internal data structures need to change. jmap has lots of other requirements, so it's already a big project. by implementing a webmail now, some of the required data structure changes become clear and can be made now, so the later jmap implementation can do things similarly to the webmail code. the webmail frontend and webmail are written together, making their interface/api much smaller and simpler than jmap. one of the internal changes is that we now keep track of per-mailbox total/unread/unseen/deleted message counts and mailbox sizes. keeping this data consistent after any change to the stored messages (through the code base) is tricky, so mox now has a consistency check that verifies the counts are correct, which runs only during tests, each time an internal account reference is closed. we have a few more internal "changes" that are propagated for the webmail frontend (that imap doesn't have a way to propagate on a connection), like changes to the special-use flags on mailboxes, and used keywords in a mailbox. more changes that will be required have revealed themselves while implementing the webmail, and will be implemented next. the webmail user interface is modeled after the mail clients i use or have used: thunderbird, macos mail, mutt; and webmails i normally only use for testing: gmail, proton, yahoo, outlook. a somewhat technical user is assumed, but still the goal is to make this webmail client easy to use for everyone. the user interface looks like most other mail clients: a list of mailboxes, a search bar, a message list view, and message details. there is a top/bottom and a left/right layout for the list/message view, default is automatic based on screen size. the panes can be resized by the user. buttons for actions are just text, not icons. clicking a button briefly shows the shortcut for the action in the bottom right, helping with learning to operate quickly. any text that is underdotted has a title attribute that causes more information to be displayed, e.g. what a button does or a field is about. to highlight potential phishing attempts, any text (anywhere in the webclient) that switches unicode "blocks" (a rough approximation to (language) scripts) within a word is underlined orange. multiple messages can be selected with familiar ui interaction: clicking while holding control and/or shift keys. keyboard navigation works with arrows/page up/down and home/end keys, and also with a few basic vi-like keys for list/message navigation. we prefer showing the text instead of html (with inlined images only) version of a message. html messages are shown in an iframe served from an endpoint with CSP headers to prevent dangerous resources (scripts, external images) from being loaded. the html is also sanitized, with javascript removed. a user can choose to load external resources (e.g. images for tracking purposes). the frontend is just (strict) typescript, no external frameworks. all incoming/outgoing data is typechecked, both the api request parameters and response types, and the data coming in over SSE. the types and checking code are generated with sherpats, which uses the api definitions generated by sherpadoc based on the Go code. so types from the backend are automatically propagated to the frontend. since there is no framework to automatically propagate properties and rerender components, changes coming in over the SSE connection are propagated explicitly with regular function calls. the ui is separated into "views", each with a "root" dom element that is added to the visible document. these views have additional functions for getting changes propagated, often resulting in the view updating its (internal) ui state (dom). we keep the frontend compilation simple, it's just a few typescript files that get compiled (combined and types stripped) into a single js file, no additional runtime code needed or complicated build processes used. the webmail is served is served from a compressed, cachable html file that includes style and the javascript, currently just over 225kb uncompressed, under 60kb compressed (not minified, including comments). we include the generated js files in the repository, to keep Go's easily buildable self-contained binaries. authentication is basic http, as with the account and admin pages. most data comes in over one long-term SSE connection to the backend. api requests signal which mailbox/search/messages are requested over the SSE connection. fetching individual messages, and making changes, are done through api calls. the operations are similar to imap, so some code has been moved from package imapserver to package store. the future jmap implementation will benefit from these changes too. more functionality will probably be moved to the store package in the future. the quickstart enables webmail on the internal listener by default (for new installs). users can enable it on the public listener if they want to. mox localserve enables it too. to enable webmail on existing installs, add settings like the following to the listeners in mox.conf, similar to AccountHTTP(S): WebmailHTTP: Enabled: true WebmailHTTPS: Enabled: true special thanks to liesbeth, gerben, andrii for early user feedback. there is plenty still to do, see the list at the top of webmail/webmail.ts. feedback welcome as always.
2023-08-07 22:57:03 +03:00
SMTPMaxMessageSize int64 `sconf:"optional" sconf-doc:"Maximum size in bytes for incoming and outgoing messages. Default is 100MB."`
mox!
2023-01-30 16:27:06 +03:00
SMTP struct {
Enabled bool
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
Port int `sconf:"optional" sconf-doc:"Default 25."`
NoSTARTTLS bool `sconf:"optional" sconf-doc:"Do not offer STARTTLS to secure the connection. Not recommended."`
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
RequireSTARTTLS bool `sconf:"optional" sconf-doc:"Do not accept incoming messages if STARTTLS is not active. Consider using in combination with an MTA-STS policy and/or DANE. A remote SMTP server may not support TLS and may not be able to deliver messages. Incoming messages for TLS reporting addresses ignore this setting and do not require TLS."`
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
NoRequireTLS bool `sconf:"optional" sconf-doc:"Do not announce the REQUIRETLS SMTP extension. Messages delivered using the REQUIRETLS extension should only be distributed onwards to servers also implementing the REQUIRETLS extension. In some situations, such as hosting mailing lists, this may not be feasible due to lack of support for the extension by mailing list subscribers."`
// Reoriginated messages (such as messages sent to mailing list subscribers) should
// keep REQUIRETLS. ../rfc/8689:412
implement only monitoring dns blocklists, without using them for incoming deliveries so you can still know when someone has put you on their blocklist (which may affect delivery), without using them. also query dnsbls for our ips more often when we do more outgoing connections for delivery: once every 100 messages, but at least 5 mins and at most 3 hours since the previous check.
2024-03-05 18:30:38 +03:00
DNSBLs []string `sconf:"optional" sconf-doc:"Addresses of DNS block lists for incoming messages. Block lists are only consulted for connections/messages without enough reputation to make an accept/reject decision. This prevents sending IPs of all communications to the block list provider. If any of the listed DNSBLs contains a requested IP address, the message is rejected as spam. The DNSBLs are checked for healthiness before use, at most once per 4 hours. IPs we can send from are periodically checked for being in the configured DNSBLs. See MonitorDNSBLs in domains.conf to only monitor IPs we send from, without using those DNSBLs for incoming messages. Example DNSBLs: sbl.spamhaus.org, bl.spamcop.net. See https://www.spamhaus.org/sbl/ and https://www.spamcop.net/ for more information and terms of use."`
in integration test, don't read database index files but use imap idle to get notified of message delivery, and make integration & quickstart tests faster by making first-time sender delay configurable, and using a 1s timeout instead of the default 15s we could make more types of delays configurable. the current approach isn't great, as it results in an a default value of "0s" in the config file, while the actual default is 15s (which is documented just above, but still).
2023-07-01 15:24:28 +03:00
FirstTimeSenderDelay *time.Duration `sconf:"optional" sconf-doc:"Delay before accepting a message from a first-time sender for the destination account. Default: 15s."`
DNSBLZones []dns.Domain `sconf:"-"`
mox!
2023-01-30 16:27:06 +03:00
} `sconf:"optional"`
Submission struct {
Enabled bool
Port int `sconf:"optional" sconf-doc:"Default 587."`
NoRequireSTARTTLS bool `sconf:"optional" sconf-doc:"Do not require STARTTLS. Since users must login, this means password may be sent without encryption. Not recommended."`
} `sconf:"optional" sconf-doc:"SMTP for submitting email, e.g. by email applications. Starts out in plain text, can be upgraded to TLS with the STARTTLS command. Prefer using Submissions which is always a TLS connection."`
Submissions struct {
Enabled bool
Port int `sconf:"optional" sconf-doc:"Default 465."`
} `sconf:"optional" sconf-doc:"SMTP over TLS for submitting email, by email applications. Requires a TLS config."`
IMAP struct {
Enabled bool
Port int `sconf:"optional" sconf-doc:"Default 143."`
NoRequireSTARTTLS bool `sconf:"optional" sconf-doc:"Enable this only when the connection is otherwise encrypted (e.g. through a VPN)."`
} `sconf:"optional" sconf-doc:"IMAP for reading email, by email applications. Starts out in plain text, can be upgraded to TLS with the STARTTLS command. Prefer using IMAPS instead which is always a TLS connection."`
IMAPS struct {
Enabled bool
Port int `sconf:"optional" sconf-doc:"Default 993."`
} `sconf:"optional" sconf-doc:"IMAP over TLS for reading email, by email applications. Requires a TLS config."`
replace http basic auth for web interfaces with session cookie & csrf-based auth the http basic auth we had was very simple to reason about, and to implement. but it has a major downside: there is no way to logout, browsers keep sending credentials. ideally, browsers themselves would show a button to stop sending credentials. a related downside: the http auth mechanism doesn't indicate for which server paths the credentials are. another downside: the original password is sent to the server with each request. though sending original passwords to web servers seems to be considered normal. our new approach uses session cookies, along with csrf values when we can. the sessions are server-side managed, automatically extended on each use. this makes it easy to invalidate sessions and keeps the frontend simpler (than with long- vs short-term sessions and refreshing). the cookies are httponly, samesite=strict, scoped to the path of the web interface. cookies are set "secure" when set over https. the cookie is set by a successful call to Login. a call to Logout invalidates a session. changing a password invalidates all sessions for a user, but keeps the session with which the password was changed alive. the csrf value is also random, and associated with the session cookie. the csrf must be sent as header for api calls, or as parameter for direct form posts (where we cannot set a custom header). rest-like calls made directly by the browser, e.g. for images, don't have a csrf protection. the csrf value is returned by the Login api call and stored in localstorage. api calls without credentials return code "user:noAuth", and with bad credentials return "user:badAuth". the api client recognizes this and triggers a login. after a login, all auth-failed api calls are automatically retried. only for "user:badAuth" is an error message displayed in the login form (e.g. session expired). in an ideal world, browsers would take care of most session management. a server would indicate authentication is needed (like http basic auth), and the browsers uses trusted ui to request credentials for the server & path. the browser could use safer mechanism than sending original passwords to the server, such as scram, along with a standard way to create sessions. for now, web developers have to do authentication themselves: from showing the login prompt, ensuring the right session/csrf cookies/localstorage/headers/etc are sent with each request. webauthn is a newer way to do authentication, perhaps we'll implement it in the future. though hardware tokens aren't an attractive option for many users, and it may be overkill as long as we still do old-fashioned authentication in smtp & imap where passwords can be sent to the server. for issue #58
2024-01-04 15:10:48 +03:00
AccountHTTP WebService `sconf:"optional" sconf-doc:"Account web interface, for email users wanting to change their accounts, e.g. set new password, set new delivery rulesets. Default path is /."`
AccountHTTPS WebService `sconf:"optional" sconf-doc:"Account web interface listener like AccountHTTP, but for HTTPS. Requires a TLS config."`
AdminHTTP WebService `sconf:"optional" sconf-doc:"Admin web interface, for managing domains, accounts, etc. Default path is /admin/. Preferably only enable on non-public IPs. Hint: use 'ssh -L 8080:localhost:80 you@yourmachine' and open http://localhost:8080/admin/, or set up a tunnel (e.g. WireGuard) and add its IP to the mox 'internal' listener."`
AdminHTTPS WebService `sconf:"optional" sconf-doc:"Admin web interface listener like AdminHTTP, but for HTTPS. Requires a TLS config."`
WebmailHTTP WebService `sconf:"optional" sconf-doc:"Webmail client, for reading email. Default path is /webmail/."`
WebmailHTTPS WebService `sconf:"optional" sconf-doc:"Webmail client, like WebmailHTTP, but for HTTPS. Requires a TLS config."`
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
WebAPIHTTP WebService `sconf:"optional" sconf-doc:"Like WebAPIHTTP, but with plain HTTP, without TLS."`
WebAPIHTTPS WebService `sconf:"optional" sconf-doc:"WebAPI, a simple HTTP/JSON-based API for email, with HTTPS (requires a TLS config). Default path is /webapi/."`
replace http basic auth for web interfaces with session cookie & csrf-based auth the http basic auth we had was very simple to reason about, and to implement. but it has a major downside: there is no way to logout, browsers keep sending credentials. ideally, browsers themselves would show a button to stop sending credentials. a related downside: the http auth mechanism doesn't indicate for which server paths the credentials are. another downside: the original password is sent to the server with each request. though sending original passwords to web servers seems to be considered normal. our new approach uses session cookies, along with csrf values when we can. the sessions are server-side managed, automatically extended on each use. this makes it easy to invalidate sessions and keeps the frontend simpler (than with long- vs short-term sessions and refreshing). the cookies are httponly, samesite=strict, scoped to the path of the web interface. cookies are set "secure" when set over https. the cookie is set by a successful call to Login. a call to Logout invalidates a session. changing a password invalidates all sessions for a user, but keeps the session with which the password was changed alive. the csrf value is also random, and associated with the session cookie. the csrf must be sent as header for api calls, or as parameter for direct form posts (where we cannot set a custom header). rest-like calls made directly by the browser, e.g. for images, don't have a csrf protection. the csrf value is returned by the Login api call and stored in localstorage. api calls without credentials return code "user:noAuth", and with bad credentials return "user:badAuth". the api client recognizes this and triggers a login. after a login, all auth-failed api calls are automatically retried. only for "user:badAuth" is an error message displayed in the login form (e.g. session expired). in an ideal world, browsers would take care of most session management. a server would indicate authentication is needed (like http basic auth), and the browsers uses trusted ui to request credentials for the server & path. the browser could use safer mechanism than sending original passwords to the server, such as scram, along with a standard way to create sessions. for now, web developers have to do authentication themselves: from showing the login prompt, ensuring the right session/csrf cookies/localstorage/headers/etc are sent with each request. webauthn is a newer way to do authentication, perhaps we'll implement it in the future. though hardware tokens aren't an attractive option for many users, and it may be overkill as long as we still do old-fashioned authentication in smtp & imap where passwords can be sent to the server. for issue #58
2024-01-04 15:10:48 +03:00
MetricsHTTP struct {
mox!
2023-01-30 16:27:06 +03:00
Enabled bool
Port int `sconf:"optional" sconf-doc:"Default 8010."`
} `sconf:"optional" sconf-doc:"Serve prometheus metrics, for monitoring. You should not enable this on a public IP."`
PprofHTTP struct {
Enabled bool
Port int `sconf:"optional" sconf-doc:"Default 8011."`
} `sconf:"optional" sconf-doc:"Serve /debug/pprof/ for profiling a running mox instance. Do not enable this on a public IP!"`
AutoconfigHTTPS struct {
Enabled bool
add option to handle autoconfig and mta-sts requests without TLS, for when it is reverse proxied for #5 with hints from belst & idnovic
2023-02-25 13:28:15 +03:00
Port int `sconf:"optional" sconf-doc:"TLS port, 443 by default. You should only override this if you cannot listen on port 443 directly. Autoconfig requests will be made to port 443, so you'll have to add an external mechanism to get the connection here, e.g. by configuring port forwarding."`
NonTLS bool `sconf:"optional" sconf-doc:"If set, plain HTTP instead of HTTPS is spoken on the configured port. Can be useful when the autoconfig domain is reverse proxied."`
mox!
2023-01-30 16:27:06 +03:00
} `sconf:"optional" sconf-doc:"Serve autoconfiguration/autodiscovery to simplify configuring email applications, will use port 443. Requires a TLS config."`
MTASTSHTTPS struct {
Enabled bool
add option to handle autoconfig and mta-sts requests without TLS, for when it is reverse proxied for #5 with hints from belst & idnovic
2023-02-25 13:28:15 +03:00
Port int `sconf:"optional" sconf-doc:"TLS port, 443 by default. You should only override this if you cannot listen on port 443 directly. MTA-STS requests will be made to port 443, so you'll have to add an external mechanism to get the connection here, e.g. by configuring port forwarding."`
NonTLS bool `sconf:"optional" sconf-doc:"If set, plain HTTP instead of HTTPS is spoken on the configured port. Can be useful when the mta-sts domain is reverse proxied."`
make it possible to serve https on a different port than 443 through configuration so you can run mox on openbsd with port redirects in pf.conf. in the future, starting as root, binding the sockets, and passing the bound sockets to a new unprivileged process should be implemented, but this should get openbsd users going. from discussion with mteege
2023-02-18 18:53:06 +03:00
} `sconf:"optional" sconf-doc:"Serve MTA-STS policies describing SMTP TLS requirements. Requires a TLS config."`
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
WebserverHTTP struct {
Enabled bool
Port int `sconf:"optional" sconf-doc:"Port for plain HTTP (non-TLS) webserver."`
} `sconf:"optional" sconf-doc:"All configured WebHandlers will serve on an enabled listener."`
WebserverHTTPS struct {
Enabled bool
Port int `sconf:"optional" sconf-doc:"Port for HTTPS webserver."`
} `sconf:"optional" sconf-doc:"All configured WebHandlers will serve on an enabled listener. Either ACME must be configured, or for each WebHandler domain a TLS certificate must be configured."`
mox!
2023-01-30 16:27:06 +03:00
}
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
// WebService is an internal web interface: webmail, webaccount, webadmin, webapi.
replace http basic auth for web interfaces with session cookie & csrf-based auth the http basic auth we had was very simple to reason about, and to implement. but it has a major downside: there is no way to logout, browsers keep sending credentials. ideally, browsers themselves would show a button to stop sending credentials. a related downside: the http auth mechanism doesn't indicate for which server paths the credentials are. another downside: the original password is sent to the server with each request. though sending original passwords to web servers seems to be considered normal. our new approach uses session cookies, along with csrf values when we can. the sessions are server-side managed, automatically extended on each use. this makes it easy to invalidate sessions and keeps the frontend simpler (than with long- vs short-term sessions and refreshing). the cookies are httponly, samesite=strict, scoped to the path of the web interface. cookies are set "secure" when set over https. the cookie is set by a successful call to Login. a call to Logout invalidates a session. changing a password invalidates all sessions for a user, but keeps the session with which the password was changed alive. the csrf value is also random, and associated with the session cookie. the csrf must be sent as header for api calls, or as parameter for direct form posts (where we cannot set a custom header). rest-like calls made directly by the browser, e.g. for images, don't have a csrf protection. the csrf value is returned by the Login api call and stored in localstorage. api calls without credentials return code "user:noAuth", and with bad credentials return "user:badAuth". the api client recognizes this and triggers a login. after a login, all auth-failed api calls are automatically retried. only for "user:badAuth" is an error message displayed in the login form (e.g. session expired). in an ideal world, browsers would take care of most session management. a server would indicate authentication is needed (like http basic auth), and the browsers uses trusted ui to request credentials for the server & path. the browser could use safer mechanism than sending original passwords to the server, such as scram, along with a standard way to create sessions. for now, web developers have to do authentication themselves: from showing the login prompt, ensuring the right session/csrf cookies/localstorage/headers/etc are sent with each request. webauthn is a newer way to do authentication, perhaps we'll implement it in the future. though hardware tokens aren't an attractive option for many users, and it may be overkill as long as we still do old-fashioned authentication in smtp & imap where passwords can be sent to the server. for issue #58
2024-01-04 15:10:48 +03:00
type WebService struct {
Enabled bool
Port int `sconf:"optional" sconf-doc:"Default 80 for HTTP and 443 for HTTPS."`
Path string `sconf:"optional" sconf-doc:"Path to serve requests on."`
Forwarded bool `sconf:"optional" sconf-doc:"If set, X-Forwarded-* headers are used for the remote IP address for rate limiting and for the \"secure\" status of cookies."`
}
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
// Transport is a method to delivery a message. At most one of the fields can
// be non-nil. The non-nil field represents the type of transport. For a
// transport with all fields nil, regular email delivery is done.
type Transport struct {
add TransportDirect transport The `TransportDirect` transport allows to tweak outgoing SMTP connections to remote servers. Currently, it only allows to select network IP family (ipv4, ipv6 or both). For example, to disable ipv6 for all outgoing SMTP connections: - add these lines in mox.conf to create a new transport named "disableipv6": ``` Transports: disableipv6: Direct: DisableIpv6: true ``` - then add these lines in domains.conf to use this transport: ``` Routes: - Transport: disableipv6 ``` fix #149
2024-04-08 22:50:30 +03:00
Submissions *TransportSMTP `sconf:"optional" sconf-doc:"Submission SMTP over a TLS connection to submit email to a remote queue."`
Submission *TransportSMTP `sconf:"optional" sconf-doc:"Submission SMTP over a plain TCP connection (possibly with STARTTLS) to submit email to a remote queue."`
SMTP *TransportSMTP `sconf:"optional" sconf-doc:"SMTP over a plain connection (possibly with STARTTLS), typically for old-fashioned unauthenticated relaying to a remote queue."`
Socks *TransportSocks `sconf:"optional" sconf-doc:"Like regular direct delivery, but makes outgoing connections through a SOCKS proxy."`
Direct *TransportDirect `sconf:"optional" sconf-doc:"Like regular direct delivery, but allows to tweak outgoing connections."`
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
// TransportSMTP delivers messages by "submission" (SMTP, typically
// authenticated) to the queue of a remote host (smarthost), or by relaying
// (SMTP, typically unauthenticated).
type TransportSMTP struct {
Host string `sconf-doc:"Host name to connect to and for verifying its TLS certificate."`
Port int `sconf:"optional" sconf-doc:"If unset or 0, the default port for submission(s)/smtp is used: 25 for SMTP, 465 for submissions (with TLS), 587 for submission (possibly with STARTTLS)."`
STARTTLSInsecureSkipVerify bool `sconf:"optional" sconf-doc:"If set an unverifiable remote TLS certificate during STARTTLS is accepted."`
NoSTARTTLS bool `sconf:"optional" sconf-doc:"If set for submission or smtp transport, do not attempt STARTTLS on the connection. Authentication credentials and messages will be transferred in clear text."`
Auth *SMTPAuth `sconf:"optional" sconf-doc:"If set, authentication credentials for the remote server."`
DNSHost dns.Domain `sconf:"-" json:"-"`
}
// SMTPAuth hold authentication credentials used when delivering messages
// through a smarthost.
type SMTPAuth struct {
Username string
Password string
implement the plus variants of scram, to bind the authentication exchange to the tls connection to get the security benefits (detecting mitm attempts), explicitly configure clients to use a scram plus variant, e.g. scram-sha-256-plus. unfortunately, not many clients support it yet. imapserver scram plus support seems to work with the latest imtest (imap test client) from cyrus-sasl. no success yet with mutt (with gsasl) though.
2023-12-24 01:07:21 +03:00
Mechanisms []string `sconf:"optional" sconf-doc:"Allowed authentication mechanisms. Defaults to SCRAM-SHA-256-PLUS, SCRAM-SHA-256, SCRAM-SHA-1-PLUS, SCRAM-SHA-1, CRAM-MD5. Not included by default: PLAIN. Specify the strongest mechanism known to be implemented by the server to prevent mechanism downgrade attacks."`
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
EffectiveMechanisms []string `sconf:"-" json:"-"`
}
type TransportSocks struct {
Address string `sconf-doc:"Address of SOCKS proxy, of the form host:port or ip:port."`
RemoteIPs []string `sconf-doc:"IP addresses connections from the SOCKS server will originate from. This IP addresses should be configured in the SPF record (keep in mind DNS record time to live (TTL) when adding a SOCKS proxy). Reverse DNS should be set up for these address, resolving to RemoteHostname. These are typically the IPv4 and IPv6 address for the host in the Address field."`
RemoteHostname string `sconf-doc:"Hostname belonging to RemoteIPs. This name is used during in SMTP EHLO. This is typically the hostname of the host in the Address field."`
// todo: add authentication credentials?
IPs []net.IP `sconf:"-" json:"-"` // Parsed form of RemoteIPs.
Hostname dns.Domain `sconf:"-" json:"-"` // Parsed form of RemoteHostname
}
add TransportDirect transport The `TransportDirect` transport allows to tweak outgoing SMTP connections to remote servers. Currently, it only allows to select network IP family (ipv4, ipv6 or both). For example, to disable ipv6 for all outgoing SMTP connections: - add these lines in mox.conf to create a new transport named "disableipv6": ``` Transports: disableipv6: Direct: DisableIpv6: true ``` - then add these lines in domains.conf to use this transport: ``` Routes: - Transport: disableipv6 ``` fix #149
2024-04-08 22:50:30 +03:00
type TransportDirect struct {
DisableIPv4 bool `sconf:"optional" sconf-doc:"If set, outgoing SMTP connections will *NOT* use IPv4 addresses to connect to remote SMTP servers."`
DisableIPv6 bool `sconf:"optional" sconf-doc:"If set, outgoing SMTP connections will *NOT* use IPv6 addresses to connect to remote SMTP servers."`
IPFamily string `sconf:"-" json:"-"`
}
mox!
2023-01-30 16:27:06 +03:00
type Domain struct {
Description string `sconf:"optional" sconf-doc:"Free-form description of domain."`
clarify unicode domain names in config file
2024-01-24 12:48:44 +03:00
ClientSettingsDomain string `sconf:"optional" sconf-doc:"Hostname for client settings instead of the mail server hostname. E.g. mail.<domain>. For future migration to another mail operator without requiring all clients to update their settings, it is convenient to have client settings that reference a subdomain of the hosted domain instead of the hostname of the server where the mail is currently hosted. If empty, the hostname of the mail server is used for client configurations. Unicode name."`
mox!
2023-01-30 16:27:06 +03:00
LocalpartCatchallSeparator string `sconf:"optional" sconf-doc:"If not empty, only the string before the separator is used to for email delivery decisions. For example, if set to \"+\", you+anything@example.com will be delivered to you@example.com."`
LocalpartCaseSensitive bool `sconf:"optional" sconf-doc:"If set, upper/lower case is relevant for email delivery."`
DKIM DKIM `sconf:"optional" sconf-doc:"With DKIM signing, a domain is taking responsibility for (content of) emails it sends, letting receiving mail servers build up a (hopefully positive) reputation of the domain, which can help with mail delivery."`
DMARC *DMARC `sconf:"optional" sconf-doc:"With DMARC, a domain publishes, in DNS, a policy on how other mail servers should handle incoming messages with the From-header matching this domain and/or subdomain (depending on the configured alignment). Receiving mail servers use this to build up a reputation of this domain, which can help with mail delivery. A domain can also publish an email address to which reports about DMARC verification results can be sent by verifying mail servers, useful for monitoring. Incoming DMARC reports are automatically parsed, validated, added to metrics and stored in the reporting database for later display in the admin web pages."`
webadmin: make remaining domain settings configurable via admin web interface for dmarc reporting address, tls reporting address, mtasts policy, dkim keys/selectors. should make it easier for webadmin-using admins to discover these settings. the webadmin interface is now on par with functionality you would set through the configuration file, let's keep it that way.
2024-04-19 11:23:53 +03:00
MTASTS *MTASTS `sconf:"optional" sconf-doc:"MTA-STS is a mechanism that allows publishing a policy with requirements for WebPKI-verified SMTP STARTTLS connections for email delivered to a domain. Existence of a policy is announced in a DNS TXT record (often unprotected/unverified, MTA-STS's weak spot). If a policy exists, it is fetched with a WebPKI-verified HTTPS request. The policy can indicate that WebPKI-verified SMTP STARTTLS is required, and which MX hosts (optionally with a wildcard pattern) are allowd. MX hosts to deliver to are still taken from DNS (again, not necessarily protected/verified), but messages will only be delivered to domains matching the MX hosts from the published policy. Mail servers look up the MTA-STS policy when first delivering to a domain, then keep a cached copy, periodically checking the DNS record if a new policy is available, and fetching and caching it if so. To update a policy, first serve a new policy with an updated policy ID, then update the DNS record (not the other way around). To remove an enforced policy, publish an updated policy with mode \"none\" for a long enough period so all cached policies have been refreshed (taking DNS TTL and policy max age into account), then remove the policy from DNS, wait for TTL to expire, and stop serving the policy."`
mox!
2023-01-30 16:27:06 +03:00
TLSRPT *TLSRPT `sconf:"optional" sconf-doc:"With TLSRPT a domain specifies in DNS where reports about encountered SMTP TLS behaviour should be sent. Useful for monitoring. Incoming TLS reports are automatically parsed, validated, added to metrics and stored in the reporting database for later display in the admin web pages."`
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
Routes []Route `sconf:"optional" sconf-doc:"Routes for delivering outgoing messages through the queue. Each delivery attempt evaluates account routes, these domain routes and finally global routes. The transport of the first matching route is used in the delivery attempt. If no routes match, which is the default with no configured routes, messages are delivered directly from the queue."`
mox!
2023-01-30 16:27:06 +03:00
assume a dns cname record mail.<domain>, pointing to the hostname of the mail server, for clients to connect to the autoconfig/autodiscover endpoints, and the printed client settings (in quickstart, in the admin interface) now all point to the cname record (called "client settings domain"). it is configurable per domain, and set to "mail.<domain>" by default. for existing mox installs, the domain can be added by editing the config file. this makes it easier for a domain to migrate to another server in the future. client settings don't have to be updated, the cname can just be changed. before, the hostname of the mail server was configured in email clients. migrating away would require changing settings in all clients. if a client settings domain is configured, a TLS certificate for the name will be requested through ACME, or must be configured manually.
2023-12-24 13:01:16 +03:00
Domain dns.Domain `sconf:"-" json:"-"`
ClientSettingsDNSDomain dns.Domain `sconf:"-" json:"-"`
for domains configured only for reporting, don't reject messages to that domain during smtp submission you can configure a domain only to accept dmarc/tls reports. those domains won't have addresses for that domain configured (the reporting destination address is for another domain). we already handled such domains specially in a few places. but we were considering ourselves authoritative for such domains if an smtp client would send a message to the domain during submit. and we would reject all recipient addresses. but we should be trying to deliver those messages to the actual mx hosts for the domain, which we will now do.
2024-01-26 21:51:23 +03:00
// Set when DMARC and TLSRPT (when set) has an address with different domain (we're
// hosting the reporting), and there are no destination addresses configured for
// the domain. Disables some functionality related to hosting a domain.
ReportsOnly bool `sconf:"-" json:"-"`
mox!
2023-01-30 16:27:06 +03:00
}
type DMARC struct {
Localpart string `sconf-doc:"Address-part before the @ that accepts DMARC reports. Must be non-internationalized. Recommended value: dmarc-reports."`
webadmin: make remaining domain settings configurable via admin web interface for dmarc reporting address, tls reporting address, mtasts policy, dkim keys/selectors. should make it easier for webadmin-using admins to discover these settings. the webadmin interface is now on par with functionality you would set through the configuration file, let's keep it that way.
2024-04-19 11:23:53 +03:00
Domain string `sconf:"optional" sconf-doc:"Alternative domain for reporting address, for incoming reports. Typically empty, causing the domain wherein this config exists to be used. Can be used to receive reports for domains that aren't fully hosted on this server. Configure such a domain as a hosted domain without making all the DNS changes, and configure this field with a domain that is fully hosted on this server, so the localpart and the domain of this field form a reporting address. Then only update the DMARC DNS record for the not fully hosted domain, ensuring the reporting address is specified in its \"rua\" field as shown in the suggested DNS settings. Unicode name."`
mox!
2023-01-30 16:27:06 +03:00
Account string `sconf-doc:"Account to deliver to."`
Mailbox string `sconf-doc:"Mailbox to deliver to, e.g. DMARC."`
ParsedLocalpart smtp.Localpart `sconf:"-"`
implement accepting dmarc & tls reports for other domains to accept reports for another domain, first add that domain to the config, leaving all options empty except DMARC/TLSRPT in which you configure a Domain. the suggested DNS DMARC/TLSRPT records will show the email address with configured domain. for DMARC, the dnscheck functionality will verify that the destination domain has opted in to receiving reports. there is a new command-line subcommand "mox dmarc checkreportaddrs" that verifies if dmarc reporting destination addresses have opted in to received reports. this also changes the suggested dns records (in quickstart, and through admin pages and cli subcommand) to take into account whether DMARC and TLSRPT is configured, and with which localpart/domain (previously it always printed records as if reporting was enabled for the domain). and when generating the suggested DNS records, the dmarc.Record and tlsrpt.Record code is used, with proper uri-escaping.
2023-08-23 15:27:21 +03:00
DNSDomain dns.Domain `sconf:"-"` // Effective domain, always set based on Domain field or Domain where this is configured.
mox!
2023-01-30 16:27:06 +03:00
}
type MTASTS struct {
webadmin: make remaining domain settings configurable via admin web interface for dmarc reporting address, tls reporting address, mtasts policy, dkim keys/selectors. should make it easier for webadmin-using admins to discover these settings. the webadmin interface is now on par with functionality you would set through the configuration file, let's keep it that way.
2024-04-19 11:23:53 +03:00
PolicyID string `sconf-doc:"Policies are versioned. The version must be specified in the DNS record. If you change a policy, first change it here to update the served policy, then update the DNS record with the updated policy ID."`
Mode mtasts.Mode `sconf-doc:"If set to \"enforce\", a remote SMTP server will not deliver email to us if it cannot make a WebPKI-verified SMTP STARTTLS connection. In mode \"testing\", deliveries can be done without verified TLS, but errors will be reported through TLS reporting. In mode \"none\", verified TLS is not required, used for phasing out an MTA-STS policy."`
mox!
2023-01-30 16:27:06 +03:00
MaxAge time.Duration `sconf-doc:"How long a remote mail server is allowed to cache a policy. Typically 1 or several weeks."`
MX []string `sconf:"optional" sconf-doc:"List of server names allowed for SMTP. If empty, the configured hostname is set. Host names can contain a wildcard (*) as a leading label (matching a single label, e.g. *.example matches host.example, not sub.host.example)."`
// todo: parse mx as valid mtasts.Policy.MX, with dns.ParseDomain but taking wildcard into account
}
type TLSRPT struct {
Localpart string `sconf-doc:"Address-part before the @ that accepts TLSRPT reports. Recommended value: tls-reports."`
webadmin: make remaining domain settings configurable via admin web interface for dmarc reporting address, tls reporting address, mtasts policy, dkim keys/selectors. should make it easier for webadmin-using admins to discover these settings. the webadmin interface is now on par with functionality you would set through the configuration file, let's keep it that way.
2024-04-19 11:23:53 +03:00
Domain string `sconf:"optional" sconf-doc:"Alternative domain for reporting address, for incoming reports. Typically empty, causing the domain wherein this config exists to be used. Can be used to receive reports for domains that aren't fully hosted on this server. Configure such a domain as a hosted domain without making all the DNS changes, and configure this field with a domain that is fully hosted on this server, so the localpart and the domain of this field form a reporting address. Then only update the TLSRPT DNS record for the not fully hosted domain, ensuring the reporting address is specified in its \"rua\" field as shown in the suggested DNS settings. Unicode name."`
mox!
2023-01-30 16:27:06 +03:00
Account string `sconf-doc:"Account to deliver to."`
Mailbox string `sconf-doc:"Mailbox to deliver to, e.g. TLSRPT."`
ParsedLocalpart smtp.Localpart `sconf:"-"`
implement accepting dmarc & tls reports for other domains to accept reports for another domain, first add that domain to the config, leaving all options empty except DMARC/TLSRPT in which you configure a Domain. the suggested DNS DMARC/TLSRPT records will show the email address with configured domain. for DMARC, the dnscheck functionality will verify that the destination domain has opted in to receiving reports. there is a new command-line subcommand "mox dmarc checkreportaddrs" that verifies if dmarc reporting destination addresses have opted in to received reports. this also changes the suggested dns records (in quickstart, and through admin pages and cli subcommand) to take into account whether DMARC and TLSRPT is configured, and with which localpart/domain (previously it always printed records as if reporting was enabled for the domain). and when generating the suggested DNS records, the dmarc.Record and tlsrpt.Record code is used, with proper uri-escaping.
2023-08-23 15:27:21 +03:00
DNSDomain dns.Domain `sconf:"-"` // Effective domain, always set based on Domain field or Domain where this is configured.
mox!
2023-01-30 16:27:06 +03:00
}
webadmin: make routes configurable: globally, per domain, per account this simplifies some of the code that makes modifications to the config file. a few protected functions can make changes to the dynamic config, which webadmin can use. instead of having separate functions in mox-/admin.go for each type of change. this also exports the parsed full dynamic config to webadmin, so we need fewer functions for specific config fields too.
2024-04-18 12:14:24 +03:00
type Canonicalization struct {
HeaderRelaxed bool `sconf-doc:"If set, some modifications to the headers (mostly whitespace) are allowed."`
BodyRelaxed bool `sconf-doc:"If set, some whitespace modifications to the message body are allowed."`
}
mox!
2023-01-30 16:27:06 +03:00
type Selector struct {
webadmin: make remaining domain settings configurable via admin web interface for dmarc reporting address, tls reporting address, mtasts policy, dkim keys/selectors. should make it easier for webadmin-using admins to discover these settings. the webadmin interface is now on par with functionality you would set through the configuration file, let's keep it that way.
2024-04-19 11:23:53 +03:00
Hash string `sconf:"optional" sconf-doc:"sha256 (default) or (older, not recommended) sha1."`
webadmin: make routes configurable: globally, per domain, per account this simplifies some of the code that makes modifications to the config file. a few protected functions can make changes to the dynamic config, which webadmin can use. instead of having separate functions in mox-/admin.go for each type of change. this also exports the parsed full dynamic config to webadmin, so we need fewer functions for specific config fields too.
2024-04-18 12:14:24 +03:00
HashEffective string `sconf:"-"`
Canonicalization Canonicalization `sconf:"optional"`
Headers []string `sconf:"optional" sconf-doc:"Headers to sign with DKIM. If empty, a reasonable default set of headers is selected."`
HeadersEffective []string `sconf:"-"` // Used when signing. Based on Headers from config, or the reasonable default.
DontSealHeaders bool `sconf:"optional" sconf-doc:"If set, don't prevent duplicate headers from being added. Not recommended."`
Expiration string `sconf:"optional" sconf-doc:"Period a signature is valid after signing, as duration, e.g. 72h. The period should be enough for delivery at the final destination, potentially with several hops/relays. In the order of days at least."`
PrivateKeyFile string `sconf-doc:"Either an RSA or ed25519 private key file in PKCS8 PEM form."`
mox!
2023-01-30 16:27:06 +03:00
webadmin: make remaining domain settings configurable via admin web interface for dmarc reporting address, tls reporting address, mtasts policy, dkim keys/selectors. should make it easier for webadmin-using admins to discover these settings. the webadmin interface is now on par with functionality you would set through the configuration file, let's keep it that way.
2024-04-19 11:23:53 +03:00
Algorithm string `sconf:"-"` // "ed25519", "rsa-*", based on private key.
mox!
2023-01-30 16:27:06 +03:00
ExpirationSeconds int `sconf:"-" json:"-"` // Parsed from Expiration.
Key crypto.Signer `sconf:"-" json:"-"` // As parsed with x509.ParsePKCS8PrivateKey.
Domain dns.Domain `sconf:"-" json:"-"` // Of selector only, not FQDN.
}
type DKIM struct {
Selectors map[string]Selector `sconf-doc:"Emails can be DKIM signed. Config parameters are per selector. A DNS record must be created for each selector. Add the name to Sign to use the selector for signing messages."`
Sign []string `sconf:"optional" sconf-doc:"List of selectors that emails will be signed with."`
}
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
type Route struct {
FromDomain []string `sconf:"optional" sconf-doc:"Matches if the envelope from domain matches one of the configured domains, or if the list is empty. If a domain starts with a dot, prefixes of the domain also match."`
ToDomain []string `sconf:"optional" sconf-doc:"Like FromDomain, but matching against the envelope to domain."`
MinimumAttempts int `sconf:"optional" sconf-doc:"Matches if at least this many deliveries have already been attempted. This can be used to attempt sending through a smarthost when direct delivery has failed for several times."`
Transport string `sconf:"The transport used for delivering the message that matches requirements of the above fields."`
// todo future: add ToMX, where we look up the MX record of the destination domain and check (the first, any, all?) mx host against the values in ToMX.
FromDomainASCII []string `sconf:"-"`
ToDomainASCII []string `sconf:"-"`
ResolvedTransport Transport `sconf:"-" json:"-"`
}
webmail: do not automatically mark read messages in Rejects mailbox as nonjunk
2023-11-27 09:34:18 +03:00
// todo: move RejectsMailbox to store.Mailbox.SpecialUse, possibly with "X" prefix?
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
// note: outgoing hook events are in ../queue/hooks.go, ../mox-/config.go, ../queue.go and ../webapi/gendoc.sh. keep in sync.
type OutgoingWebhook struct {
URL string `sconf-doc:"URL to POST webhooks."`
Authorization string `sconf:"optional" sconf-doc:"If not empty, value of Authorization header to add to HTTP requests."`
Events []string `sconf:"optional" sconf-doc:"Events to send outgoing delivery notifications for. If absent, all events are sent. Valid values: delivered, suppressed, delayed, failed, relayed, expanded, canceled, unrecognized."`
}
type IncomingWebhook struct {
URL string `sconf-doc:"URL to POST webhooks to for incoming deliveries over SMTP."`
Authorization string `sconf:"optional" sconf-doc:"If not empty, value of Authorization header to add to HTTP requests."`
}
use type config.Account in sherpa api for better typing, and update to latest sherpa lib typescript now knows the full types, not just "any" for account config. inline structs previously in config.Account are given their own type definition so sherpa can generate types. also update to latest sherpa lib that knows about time.Duration, to be used soon.
2024-04-14 18:18:20 +03:00
type SubjectPass struct {
Period time.Duration `sconf-doc:"How long unique values are accepted after generating, e.g. 12h."` // todo: have a reasonable default for this?
}
type AutomaticJunkFlags struct {
make more account config fields configurable through web interface so users can change it themselves, instead of requiring an admin to change the settings.
2024-04-17 22:30:54 +03:00
Enabled bool `sconf-doc:"If enabled, junk/nonjunk flags will be set automatically if they match some of the regular expressions. When two of the three mailbox regular expressions are set, the remaining one will match all unmatched messages. Messages are matched in the order 'junk', 'neutral', 'not junk', and the search stops on the first match. Mailboxes are lowercased before matching."`
use type config.Account in sherpa api for better typing, and update to latest sherpa lib typescript now knows the full types, not just "any" for account config. inline structs previously in config.Account are given their own type definition so sherpa can generate types. also update to latest sherpa lib that knows about time.Duration, to be used soon.
2024-04-14 18:18:20 +03:00
JunkMailboxRegexp string `sconf:"optional" sconf-doc:"Example: ^(junk|spam)."`
NeutralMailboxRegexp string `sconf:"optional" sconf-doc:"Example: ^(inbox|neutral|postmaster|dmarc|tlsrpt|rejects), and you may wish to add trash depending on how you use it, or leave this empty."`
NotJunkMailboxRegexp string `sconf:"optional" sconf-doc:"Example: .* or an empty string."`
}
mox!
2023-01-30 16:27:06 +03:00
type Account struct {
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
OutgoingWebhook *OutgoingWebhook `sconf:"optional" sconf-doc:"Webhooks for events about outgoing deliveries."`
IncomingWebhook *IncomingWebhook `sconf:"optional" sconf-doc:"Webhooks for events about incoming deliveries over SMTP."`
FromIDLoginAddresses []string `sconf:"optional" sconf-doc:"Login addresses that cause outgoing email to be sent with SMTP MAIL FROM addresses with a unique id after the localpart catchall separator (which must be enabled when addresses are specified here). Any delivery status notifications (DSN, e.g. for bounces), can be related to the original message and recipient with unique id's. You can login to an account with any valid email address, including variants with the localpart catchall separator. You can use this mechanism to both send outgoing messages both with and without unique fromid for a given address."`
KeepRetiredMessagePeriod time.Duration `sconf:"optional" sconf-doc:"Period to keep messages retired from the queue (delivered or failed) around. Keeping retired messages is useful for maintaining the suppression list for transactional email, for matching incoming DSNs to sent messages, and for debugging. The time at which to clean up (remove) is calculated at retire time. E.g. 168h (1 week)."`
KeepRetiredWebhookPeriod time.Duration `sconf:"optional" sconf-doc:"Period to keep webhooks retired from the queue (delivered or failed) around. Useful for debugging. The time at which to clean up (remove) is calculated at retire time. E.g. 168h (1 week)."`
use type config.Account in sherpa api for better typing, and update to latest sherpa lib typescript now knows the full types, not just "any" for account config. inline structs previously in config.Account are given their own type definition so sherpa can generate types. also update to latest sherpa lib that knows about time.Duration, to be used soon.
2024-04-14 18:18:20 +03:00
Domain string `sconf-doc:"Default domain for account. Deprecated behaviour: If a destination is not a full address but only a localpart, this domain is added to form a full address."`
Description string `sconf:"optional" sconf-doc:"Free form description, e.g. full name or alternative contact info."`
FullName string `sconf:"optional" sconf-doc:"Full name, to use in message From header when composing messages in webmail. Can be overridden per destination."`
Destinations map[string]Destination `sconf:"optional" sconf-doc:"Destinations, keys are email addresses (with IDNA domains). All destinations are allowed for logging in with IMAP/SMTP/webmail. If no destinations are configured, the account can not login. If the address is of the form '@domain', i.e. with localpart missing, it serves as a catchall for the domain, matching all messages that are not explicitly configured. Deprecated behaviour: If the address is not a full address but a localpart, it is combined with Domain to form a full address."`
SubjectPass SubjectPass `sconf:"optional" sconf-doc:"If configured, messages classified as weakly spam are rejected with instructions to retry delivery, but this time with a signed token added to the subject. During the next delivery attempt, the signed token will bypass the spam filter. Messages with a clear spam signal, such as a known bad reputation, are rejected/delayed without a signed token."`
QuotaMessageSize int64 `sconf:"optional" sconf-doc:"Default maximum total message size in bytes for the account, overriding any globally configured default maximum size if non-zero. A negative value can be used to have no limit in case there is a limit by default. Attempting to add new messages to an account beyond its maximum total size will result in an error. Useful to prevent a single account from filling storage."`
RejectsMailbox string `sconf:"optional" sconf-doc:"Mail that looks like spam will be rejected, but a copy can be stored temporarily in a mailbox, e.g. Rejects. If mail isn't coming in when you expect, you can look there. The mail still isn't accepted, so the remote mail server may retry (hopefully, if legitimate), or give up (hopefully, if indeed a spammer). Messages are automatically removed from this mailbox, so do not set it to a mailbox that has messages you want to keep."`
make more account config fields configurable through web interface so users can change it themselves, instead of requiring an admin to change the settings.
2024-04-17 22:30:54 +03:00
KeepRejects bool `sconf:"optional" sconf-doc:"Don't automatically delete mail in the RejectsMailbox listed above. This can be useful, e.g. for future spam training. It can also cause storage to fill up."`
use type config.Account in sherpa api for better typing, and update to latest sherpa lib typescript now knows the full types, not just "any" for account config. inline structs previously in config.Account are given their own type definition so sherpa can generate types. also update to latest sherpa lib that knows about time.Duration, to be used soon.
2024-04-14 18:18:20 +03:00
AutomaticJunkFlags AutomaticJunkFlags `sconf:"optional" sconf-doc:"Automatically set $Junk and $NotJunk flags based on mailbox messages are delivered/moved/copied to. Email clients typically have too limited functionality to conveniently set these flags, especially $NonJunk, but they can all move messages to a different mailbox, so this helps them."`
JunkFilter *JunkFilter `sconf:"optional" sconf-doc:"Content-based filtering, using the junk-status of individual messages to rank words in such messages as spam or ham. It is recommended you always set the applicable (non)-junk status on messages, and that you do not empty your Trash because those messages contain valuable ham/spam training information."` // todo: sane defaults for junkfilter
MaxOutgoingMessagesPerDay int `sconf:"optional" sconf-doc:"Maximum number of outgoing messages for this account in a 24 hour window. This limits the damage to recipients and the reputation of this mail server in case of account compromise. Default 1000."`
MaxFirstTimeRecipientsPerDay int `sconf:"optional" sconf-doc:"Maximum number of first-time recipients in outgoing messages for this account in a 24 hour window. This limits the damage to recipients and the reputation of this mail server in case of account compromise. Default 200."`
NoFirstTimeSenderDelay bool `sconf:"optional" sconf-doc:"Do not apply a delay to SMTP connections before accepting an incoming message from a first-time sender. Can be useful for accounts that sends automated responses and want instant replies."`
Routes []Route `sconf:"optional" sconf-doc:"Routes for delivering outgoing messages through the queue. Each delivery attempt evaluates these account routes, domain routes and finally global routes. The transport of the first matching route is used in the delivery attempt. If no routes match, which is the default with no configured routes, messages are delivered directly from the queue."`
improve training of junk filter before, we used heuristics to decide when to train/untrain a message as junk or nonjunk: the message had to be seen, be in certain mailboxes. then if a message was marked as junk, it was junk. and otherwise it was nonjunk. this wasn't good enough: you may want to keep some messages around as neither junk or nonjunk. and that wasn't possible. ideally, we would just look at the imap $Junk and $NotJunk flags. the problem is that mail clients don't set these flags, or don't make it easy. thunderbird can set the flags based on its own bayesian filter. it has a shortcut for marking Junk and moving it to the junk folder (good), but the counterpart of notjunk only marks a message as notjunk without showing in the UI that it was marked as notjunk. there is also no "move and mark as notjunk" mechanism. e.g. "archive" does not mark a message as notjunk. ios mail and mutt don't appear to have any way to see or change the $Junk and $NotJunk flags. what email clients do have is the ability to move messages to other mailboxes/folders. so mox now has a mechanism that allows you to configure mailboxes that automatically set $Junk or $NotJunk (or clear both) when a message is moved/copied/delivered to that folder. e.g. a mailbox called junk or spam or rejects marks its messags as junk. inbox, postmaster, dmarc, tlsrpt, neutral* mark their messages as neither junk or notjunk. other folders mark their messages as notjunk. e.g. list/*, archive. this functionality is optional, but enabled with the quickstart and for new accounts. also, mox now keeps track of the previous training of a message and will only untrain/train if needed. before, there probably have been duplicate or missing (un)trainings. this also includes a new subcommand "retrain" to recreate the junkfilter for an account. you should run it after updating to this version. and you should probably also modify your account config to include the AutomaticJunkFlags.
2023-02-12 01:00:12 +03:00
add a webapi and webhooks for a simple http/json-based api for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
DNSDomain dns.Domain `sconf:"-"` // Parsed form of Domain.
JunkMailbox *regexp.Regexp `sconf:"-" json:"-"`
NeutralMailbox *regexp.Regexp `sconf:"-" json:"-"`
NotJunkMailbox *regexp.Regexp `sconf:"-" json:"-"`
ParsedFromIDLoginAddresses []smtp.Address `sconf:"-" json:"-"`
mox!
2023-01-30 16:27:06 +03:00
}
type JunkFilter struct {
Threshold float64 `sconf-doc:"Approximate spaminess score between 0 and 1 above which emails are rejected as spam. Each delivery attempt adds a little noise to make it slightly harder for spammers to identify words that strongly indicate non-spaminess and use it to bypass the filter. E.g. 0.95."`
junk.Params
}
type Destination struct {
Mailbox string `sconf:"optional" sconf-doc:"Mailbox to deliver to if none of Rulesets match. Default: Inbox."`
Rulesets []Ruleset `sconf:"optional" sconf-doc:"Delivery rules based on message and SMTP transaction. You may want to match each mailing list by SMTP MailFrom address, VerifiedDomain and/or List-ID header (typically <listname.example.org> if the list address is listname@example.org), delivering them to their own mailbox."`
add webmail it was far down on the roadmap, but implemented earlier, because it's interesting, and to help prepare for a jmap implementation. for jmap we need to implement more client-like functionality than with just imap. internal data structures need to change. jmap has lots of other requirements, so it's already a big project. by implementing a webmail now, some of the required data structure changes become clear and can be made now, so the later jmap implementation can do things similarly to the webmail code. the webmail frontend and webmail are written together, making their interface/api much smaller and simpler than jmap. one of the internal changes is that we now keep track of per-mailbox total/unread/unseen/deleted message counts and mailbox sizes. keeping this data consistent after any change to the stored messages (through the code base) is tricky, so mox now has a consistency check that verifies the counts are correct, which runs only during tests, each time an internal account reference is closed. we have a few more internal "changes" that are propagated for the webmail frontend (that imap doesn't have a way to propagate on a connection), like changes to the special-use flags on mailboxes, and used keywords in a mailbox. more changes that will be required have revealed themselves while implementing the webmail, and will be implemented next. the webmail user interface is modeled after the mail clients i use or have used: thunderbird, macos mail, mutt; and webmails i normally only use for testing: gmail, proton, yahoo, outlook. a somewhat technical user is assumed, but still the goal is to make this webmail client easy to use for everyone. the user interface looks like most other mail clients: a list of mailboxes, a search bar, a message list view, and message details. there is a top/bottom and a left/right layout for the list/message view, default is automatic based on screen size. the panes can be resized by the user. buttons for actions are just text, not icons. clicking a button briefly shows the shortcut for the action in the bottom right, helping with learning to operate quickly. any text that is underdotted has a title attribute that causes more information to be displayed, e.g. what a button does or a field is about. to highlight potential phishing attempts, any text (anywhere in the webclient) that switches unicode "blocks" (a rough approximation to (language) scripts) within a word is underlined orange. multiple messages can be selected with familiar ui interaction: clicking while holding control and/or shift keys. keyboard navigation works with arrows/page up/down and home/end keys, and also with a few basic vi-like keys for list/message navigation. we prefer showing the text instead of html (with inlined images only) version of a message. html messages are shown in an iframe served from an endpoint with CSP headers to prevent dangerous resources (scripts, external images) from being loaded. the html is also sanitized, with javascript removed. a user can choose to load external resources (e.g. images for tracking purposes). the frontend is just (strict) typescript, no external frameworks. all incoming/outgoing data is typechecked, both the api request parameters and response types, and the data coming in over SSE. the types and checking code are generated with sherpats, which uses the api definitions generated by sherpadoc based on the Go code. so types from the backend are automatically propagated to the frontend. since there is no framework to automatically propagate properties and rerender components, changes coming in over the SSE connection are propagated explicitly with regular function calls. the ui is separated into "views", each with a "root" dom element that is added to the visible document. these views have additional functions for getting changes propagated, often resulting in the view updating its (internal) ui state (dom). we keep the frontend compilation simple, it's just a few typescript files that get compiled (combined and types stripped) into a single js file, no additional runtime code needed or complicated build processes used. the webmail is served is served from a compressed, cachable html file that includes style and the javascript, currently just over 225kb uncompressed, under 60kb compressed (not minified, including comments). we include the generated js files in the repository, to keep Go's easily buildable self-contained binaries. authentication is basic http, as with the account and admin pages. most data comes in over one long-term SSE connection to the backend. api requests signal which mailbox/search/messages are requested over the SSE connection. fetching individual messages, and making changes, are done through api calls. the operations are similar to imap, so some code has been moved from package imapserver to package store. the future jmap implementation will benefit from these changes too. more functionality will probably be moved to the store package in the future. the quickstart enables webmail on the internal listener by default (for new installs). users can enable it on the public listener if they want to. mox localserve enables it too. to enable webmail on existing installs, add settings like the following to the listeners in mox.conf, similar to AccountHTTP(S): WebmailHTTP: Enabled: true WebmailHTTPS: Enabled: true special thanks to liesbeth, gerben, andrii for early user feedback. there is plenty still to do, see the list at the top of webmail/webmail.ts. feedback welcome as always.
2023-08-07 22:57:03 +03:00
FullName string `sconf:"optional" sconf-doc:"Full name to use in message From header when composing messages coming from this address with webmail."`
mox!
2023-01-30 16:27:06 +03:00
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
DMARCReports bool `sconf:"-" json:"-"`
HostTLSReports bool `sconf:"-" json:"-"`
DomainTLSReports bool `sconf:"-" json:"-"`
mox!
2023-01-30 16:27:06 +03:00
}
in account web page, allow user to configure rulesets for delivery for example, by matching incoming messags on smtp mail from, verified domains (spf/dkim), headers. then delivering to a configured mailbox. for mailing lists, if a verified domain matches, regular spam checks can be skipped. this was already possible by editing the configuration file, but only admins can edit that file. now users can manage their own rulesets.
2023-02-11 01:47:19 +03:00
// Equal returns whether d and o are equal, only looking at their user-changeable fields.
func (d Destination) Equal(o Destination) bool {
if d.Mailbox != o.Mailbox || len(d.Rulesets) != len(o.Rulesets) {
return false
}
for i, rs := range d.Rulesets {
if !rs.Equal(o.Rulesets[i]) {
return false
}
}
return true
}
mox!
2023-01-30 16:27:06 +03:00
type Ruleset struct {
add flag to ruleset that indicates a message is forwarded, slightly modifying how junk analysis is done part of PR #50 by bobobo1618
2023-08-09 23:31:37 +03:00
SMTPMailFromRegexp string `sconf:"optional" sconf-doc:"Matches if this regular expression matches (a substring of) the SMTP MAIL FROM address (not the message From-header). E.g. '^user@example\\.org$'."`
webmail: when moving a single message out of/to the inbox, ask if user wants to create a rule to automatically do that server-side for future deliveries if the message has a list-id header, we assume this is a (mailing) list message, and we require a dkim/spf-verified domain (we prefer the shortest that is a suffix of the list-id value). the rule we would add will mark such messages as from a mailing list, changing filtering rules on incoming messages (not enforcing dmarc policies). messages will be matched on list-id header and will only match if they have the same dkim/spf-verified domain. if the message doesn't have a list-id header, we'll ask to match based on "message from" address. we don't ask the user in several cases: - if the destination/source mailbox is a special-use mailbox (e.g. trash,archive,sent,junk; inbox isn't included) - if the rule already exist (no point in adding it again). - if the user said "no, not for this list-id/from-address" in the past. - if the user said "no, not for messages moved to this mailbox" in the past. we'll add the rule if the message was moved out of the inbox. if the message was moved to the inbox, we check if there is a matching rule that we can remove. we now remember the "no" answers (for list-id, msg-from-addr and mailbox) in the account database. to implement the msgfrom rules, this adds support to rulesets for matching on message "from" address. before, we could match on smtp from address (and other fields). rulesets now also have a field for comments. webmail adds a note that it created the rule, with the date. manual editing of the rulesets is still in the webaccount page. this webmail functionality is just a convenient way to add/remove common rules.
2024-04-21 18:01:50 +03:00
MsgFromRegexp string `sconf:"optional" sconf-doc:"Matches if this regular expression matches (a substring of) the single address in the message From header."`
in account web page, allow user to configure rulesets for delivery for example, by matching incoming messags on smtp mail from, verified domains (spf/dkim), headers. then delivering to a configured mailbox. for mailing lists, if a verified domain matches, regular spam checks can be skipped. this was already possible by editing the configuration file, but only admins can edit that file. now users can manage their own rulesets.
2023-02-11 01:47:19 +03:00
VerifiedDomain string `sconf:"optional" sconf-doc:"Matches if this domain matches an SPF- and/or DKIM-verified (sub)domain."`
HeadersRegexp map[string]string `sconf:"optional" sconf-doc:"Matches if these header field/value regular expressions all match (substrings of) the message headers. Header fields and valuees are converted to lower case before matching. Whitespace is trimmed from the value before matching. A header field can occur multiple times in a message, only one instance has to match. For mailing lists, you could match on ^list-id$ with the value typically the mailing list address in angled brackets with @ replaced with a dot, e.g. <name\\.lists\\.example\\.org>."`
webmail: when moving a single message out of/to the inbox, ask if user wants to create a rule to automatically do that server-side for future deliveries if the message has a list-id header, we assume this is a (mailing) list message, and we require a dkim/spf-verified domain (we prefer the shortest that is a suffix of the list-id value). the rule we would add will mark such messages as from a mailing list, changing filtering rules on incoming messages (not enforcing dmarc policies). messages will be matched on list-id header and will only match if they have the same dkim/spf-verified domain. if the message doesn't have a list-id header, we'll ask to match based on "message from" address. we don't ask the user in several cases: - if the destination/source mailbox is a special-use mailbox (e.g. trash,archive,sent,junk; inbox isn't included) - if the rule already exist (no point in adding it again). - if the user said "no, not for this list-id/from-address" in the past. - if the user said "no, not for messages moved to this mailbox" in the past. we'll add the rule if the message was moved out of the inbox. if the message was moved to the inbox, we check if there is a matching rule that we can remove. we now remember the "no" answers (for list-id, msg-from-addr and mailbox) in the account database. to implement the msgfrom rules, this adds support to rulesets for matching on message "from" address. before, we could match on smtp from address (and other fields). rulesets now also have a field for comments. webmail adds a note that it created the rule, with the date. manual editing of the rulesets is still in the webaccount page. this webmail functionality is just a convenient way to add/remove common rules.
2024-04-21 18:01:50 +03:00
// todo: add a SMTPRcptTo check
mox!
2023-01-30 16:27:06 +03:00
add flag to ruleset that indicates a message is forwarded, slightly modifying how junk analysis is done part of PR #50 by bobobo1618
2023-08-09 23:31:37 +03:00
// todo: once we implement ARC, we can use dkim domains that we cannot verify but that the arc-verified forwarding mail server was able to verify.
fix typo in config field explanation
2024-01-23 18:59:08 +03:00
IsForward bool `sconf:"optional" sconf-doc:"Influences spam filtering only, this option does not change whether a message matches this ruleset. Can only be used together with SMTPMailFromRegexp and VerifiedDomain. SMTPMailFromRegexp must be set to the address used to deliver the forwarded message, e.g. '^user(|\\+.*)@forward\\.example$'. Changes to junk analysis: 1. Messages are not rejected for failing a DMARC policy, because a legitimate forwarded message without valid/intact/aligned DKIM signature would be rejected because any verified SPF domain will be 'unaligned', of the forwarding mail server. 2. The sending mail server IP address, and sending EHLO and MAIL FROM domains and matching DKIM domain aren't used in future reputation-based spam classifications (but other verified DKIM domains are) because the forwarding server is not a useful spam signal for future messages."`
add flag to ruleset that indicates a message is forwarded, slightly modifying how junk analysis is done part of PR #50 by bobobo1618
2023-08-09 23:31:37 +03:00
ListAllowDomain string `sconf:"optional" sconf-doc:"Influences spam filtering only, this option does not change whether a message matches this ruleset. If this domain matches an SPF- and/or DKIM-verified (sub)domain, the message is accepted without further spam checks, such as a junk filter or DMARC reject evaluation. DMARC rejects should not apply for mailing lists that are not configured to rewrite the From-header of messages that don't have a passing DKIM signature of the From-domain. Otherwise, by rejecting messages, you may be automatically unsubscribed from the mailing list. The assumption is that mailing lists do their own spam filtering/moderation."`
AcceptRejectsToMailbox string `sconf:"optional" sconf-doc:"Influences spam filtering only, this option does not change whether a message matches this ruleset. If a message is classified as spam, it isn't rejected during the SMTP transaction (the normal behaviour), but accepted during the SMTP transaction and delivered to the specified mailbox. The specified mailbox is not automatically cleaned up like the account global Rejects mailbox, unless set to that Rejects mailbox."`
mox!
2023-01-30 16:27:06 +03:00
in account web page, allow user to configure rulesets for delivery for example, by matching incoming messags on smtp mail from, verified domains (spf/dkim), headers. then delivering to a configured mailbox. for mailing lists, if a verified domain matches, regular spam checks can be skipped. this was already possible by editing the configuration file, but only admins can edit that file. now users can manage their own rulesets.
2023-02-11 01:47:19 +03:00
Mailbox string `sconf-doc:"Mailbox to deliver to if this ruleset matches."`
webmail: when moving a single message out of/to the inbox, ask if user wants to create a rule to automatically do that server-side for future deliveries if the message has a list-id header, we assume this is a (mailing) list message, and we require a dkim/spf-verified domain (we prefer the shortest that is a suffix of the list-id value). the rule we would add will mark such messages as from a mailing list, changing filtering rules on incoming messages (not enforcing dmarc policies). messages will be matched on list-id header and will only match if they have the same dkim/spf-verified domain. if the message doesn't have a list-id header, we'll ask to match based on "message from" address. we don't ask the user in several cases: - if the destination/source mailbox is a special-use mailbox (e.g. trash,archive,sent,junk; inbox isn't included) - if the rule already exist (no point in adding it again). - if the user said "no, not for this list-id/from-address" in the past. - if the user said "no, not for messages moved to this mailbox" in the past. we'll add the rule if the message was moved out of the inbox. if the message was moved to the inbox, we check if there is a matching rule that we can remove. we now remember the "no" answers (for list-id, msg-from-addr and mailbox) in the account database. to implement the msgfrom rules, this adds support to rulesets for matching on message "from" address. before, we could match on smtp from address (and other fields). rulesets now also have a field for comments. webmail adds a note that it created the rule, with the date. manual editing of the rulesets is still in the webaccount page. this webmail functionality is just a convenient way to add/remove common rules.
2024-04-21 18:01:50 +03:00
Comment string `sconf:"optional" sconf-doc:"Free-form comments."`
mox!
2023-01-30 16:27:06 +03:00
SMTPMailFromRegexpCompiled *regexp.Regexp `sconf:"-" json:"-"`
webmail: when moving a single message out of/to the inbox, ask if user wants to create a rule to automatically do that server-side for future deliveries if the message has a list-id header, we assume this is a (mailing) list message, and we require a dkim/spf-verified domain (we prefer the shortest that is a suffix of the list-id value). the rule we would add will mark such messages as from a mailing list, changing filtering rules on incoming messages (not enforcing dmarc policies). messages will be matched on list-id header and will only match if they have the same dkim/spf-verified domain. if the message doesn't have a list-id header, we'll ask to match based on "message from" address. we don't ask the user in several cases: - if the destination/source mailbox is a special-use mailbox (e.g. trash,archive,sent,junk; inbox isn't included) - if the rule already exist (no point in adding it again). - if the user said "no, not for this list-id/from-address" in the past. - if the user said "no, not for messages moved to this mailbox" in the past. we'll add the rule if the message was moved out of the inbox. if the message was moved to the inbox, we check if there is a matching rule that we can remove. we now remember the "no" answers (for list-id, msg-from-addr and mailbox) in the account database. to implement the msgfrom rules, this adds support to rulesets for matching on message "from" address. before, we could match on smtp from address (and other fields). rulesets now also have a field for comments. webmail adds a note that it created the rule, with the date. manual editing of the rulesets is still in the webaccount page. this webmail functionality is just a convenient way to add/remove common rules.
2024-04-21 18:01:50 +03:00
MsgFromRegexpCompiled *regexp.Regexp `sconf:"-" json:"-"`
mox!
2023-01-30 16:27:06 +03:00
VerifiedDNSDomain dns.Domain `sconf:"-"`
HeadersRegexpCompiled [][2]*regexp.Regexp `sconf:"-" json:"-"`
ListAllowDNSDomain dns.Domain `sconf:"-"`
}
in account web page, allow user to configure rulesets for delivery for example, by matching incoming messags on smtp mail from, verified domains (spf/dkim), headers. then delivering to a configured mailbox. for mailing lists, if a verified domain matches, regular spam checks can be skipped. this was already possible by editing the configuration file, but only admins can edit that file. now users can manage their own rulesets.
2023-02-11 01:47:19 +03:00
// Equal returns whether r and o are equal, only looking at their user-changeable fields.
func (r Ruleset) Equal(o Ruleset) bool {
webmail: when moving a single message out of/to the inbox, ask if user wants to create a rule to automatically do that server-side for future deliveries if the message has a list-id header, we assume this is a (mailing) list message, and we require a dkim/spf-verified domain (we prefer the shortest that is a suffix of the list-id value). the rule we would add will mark such messages as from a mailing list, changing filtering rules on incoming messages (not enforcing dmarc policies). messages will be matched on list-id header and will only match if they have the same dkim/spf-verified domain. if the message doesn't have a list-id header, we'll ask to match based on "message from" address. we don't ask the user in several cases: - if the destination/source mailbox is a special-use mailbox (e.g. trash,archive,sent,junk; inbox isn't included) - if the rule already exist (no point in adding it again). - if the user said "no, not for this list-id/from-address" in the past. - if the user said "no, not for messages moved to this mailbox" in the past. we'll add the rule if the message was moved out of the inbox. if the message was moved to the inbox, we check if there is a matching rule that we can remove. we now remember the "no" answers (for list-id, msg-from-addr and mailbox) in the account database. to implement the msgfrom rules, this adds support to rulesets for matching on message "from" address. before, we could match on smtp from address (and other fields). rulesets now also have a field for comments. webmail adds a note that it created the rule, with the date. manual editing of the rulesets is still in the webaccount page. this webmail functionality is just a convenient way to add/remove common rules.
2024-04-21 18:01:50 +03:00
if r.SMTPMailFromRegexp != o.SMTPMailFromRegexp || r.MsgFromRegexp != o.MsgFromRegexp || r.VerifiedDomain != o.VerifiedDomain || r.IsForward != o.IsForward || r.ListAllowDomain != o.ListAllowDomain || r.AcceptRejectsToMailbox != o.AcceptRejectsToMailbox || r.Mailbox != o.Mailbox || r.Comment != o.Comment {
in account web page, allow user to configure rulesets for delivery for example, by matching incoming messags on smtp mail from, verified domains (spf/dkim), headers. then delivering to a configured mailbox. for mailing lists, if a verified domain matches, regular spam checks can be skipped. this was already possible by editing the configuration file, but only admins can edit that file. now users can manage their own rulesets.
2023-02-11 01:47:19 +03:00
return false
}
if !reflect.DeepEqual(r.HeadersRegexp, o.HeadersRegexp) {
return false
}
return true
}
make it easier to run with existing webserver - make it easier to run with an existing webserver. the quickstart now has a new option for that, it generates a different mox.conf, and further instructions such as configuring the tls keys/certs and reverse proxy urls. and changes to make autoconfig work in that case too. - when starting up, request a tls cert for the hostname and for the autoconfig endpoint. the first will be requested soon anyway, and the autoconfig cert is needed early so the first autoconfig request doesn't time out (without helpful message to the user by at least thunderbird). and don't request the certificate before the servers are online. the root process was now requesting the certs, before the child process was serving on the tls port. - add examples of configs generated by the quickstart. - enable debug logging in config from quickstart, to give user more info. for issue #5
2023-03-04 02:49:02 +03:00
type KeyCert struct {
CertFile string `sconf-doc:"Certificate including intermediate CA certificates, in PEM format."`
KeyFile string `sconf-doc:"Private key for certificate, in PEM format. PKCS8 is recommended, but PKCS1 and EC private keys are recognized as well."`
}
mox!
2023-01-30 16:27:06 +03:00
type TLS struct {
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
ACME string `sconf:"optional" sconf-doc:"Name of provider from top-level configuration to use for ACME, e.g. letsencrypt."`
KeyCerts []KeyCert `sconf:"optional" sconf-doc:"Keys and certificates to use for this listener. The files are opened by the privileged root process and passed to the unprivileged mox process, so no special permissions are required on the files. If the private key will not be replaced when refreshing certificates, also consider adding the private key to HostPrivateKeyFiles and configuring DANE TLSA DNS records."`
MinVersion string `sconf:"optional" sconf-doc:"Minimum TLS version. Default: TLSv1.2."`
HostPrivateKeyFiles []string `sconf:"optional" sconf-doc:"Private keys used for ACME certificates. Specified explicitly so DANE TLSA DNS records can be generated, even before the certificates are requested. DANE is a mechanism to authenticate remote TLS certificates based on a public key or certificate specified in DNS, protected with DNSSEC. DANE is opportunistic and attempted when delivering SMTP with STARTTLS. The private key files must be in PEM format. PKCS8 is recommended, but PKCS1 and EC private keys are recognized as well. Only RSA 2048 bit and ECDSA P-256 keys are currently used. The first of each is used when requesting new certificates through ACME."`
Config *tls.Config `sconf:"-" json:"-"` // TLS config for non-ACME-verification connections, i.e. SMTP and IMAP, and not port 443.
ACMEConfig *tls.Config `sconf:"-" json:"-"` // TLS config that handles ACME verification, for serving on port 443.
HostPrivateRSA2048Keys []crypto.Signer `sconf:"-" json:"-"` // Private keys for new TLS certificates for listener host name, for new certificates with ACME, and for DANE records.
HostPrivateECDSAP256Keys []crypto.Signer `sconf:"-" json:"-"`
mox!
2023-01-30 16:27:06 +03:00
}
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
type WebHandler struct {
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config - make builtin http handlers serve on specific domains, such as for mta-sts, so e.g. /.well-known/mta-sts.txt isn't served on all domains. - add logging of a few more fields in access logging. - small tweaks/bug fixes in webserver request handling. - add config option for redirecting entire domains to another (common enough). - split httpserver metric into two: one for duration until writing header (i.e. performance of server), another for duration until full response is sent to client (i.e. performance as perceived by users). - add admin ui, a new page for managing the configs. after making changes and hitting "save", the changes take effect immediately. the page itself doesn't look very well-designed (many input fields, makes it look messy). i have an idea to improve it (explained in admin.html as todo) by making the layout look just like the config file. not urgent though. i've already changed my websites/webapps over. the idea of adding a webserver is to take away a (the) reason for folks to want to complicate their mox setup by running an other webserver on the same machine. i think the current webserver implementation can already serve most common use cases. with a few more tweaks (feedback needed!) we should be able to get to 95% of the use cases. the reverse proxy can take care of the remaining 5%. nevertheless, a next step is still to change the quickstart to make it easier for folks to run with an existing webserver, with existing tls certs/keys. that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
LogName string `sconf:"optional" sconf-doc:"Name to use in logging and metrics."`
Domain string `sconf-doc:"Both Domain and PathRegexp must match for this WebHandler to match a request. Exactly one of WebStatic, WebRedirect, WebForward must be set."`
PathRegexp string `sconf-doc:"Regular expression matched against request path, must always start with ^ to ensure matching from the start of the path. The matching prefix can optionally be stripped by WebForward. The regular expression does not have to end with $."`
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
DontRedirectPlainHTTP bool `sconf:"optional" sconf-doc:"If set, plain HTTP requests are not automatically permanently redirected (308) to HTTPS. If you don't have a HTTPS webserver configured, set this to true."`
implement transparent gzip compression in the webserver we only compress if applicable (content-type indicates likely compressible), client supports it, response doesn't already have a content-encoding). for internal handlers, we always enable compression. for reverse proxied and static files, compression must be enabled per handler. for internal & reverse proxy handlers, we do streaming compression at "bestspeed" quality (probably level 1). for static files, we have a cache based on mtime with fixed max size, where we evict based on least recently used. we compress with the default level (more cpu, better ratio).
2023-08-21 22:52:35 +03:00
Compress bool `sconf:"optional" sconf-doc:"Transparently compress responses (currently with gzip) if the client supports it, the status is 200 OK, no Content-Encoding is set on the response yet and the Content-Type of the response hints that the data is compressible (text/..., specific application/... and .../...+json and .../...+xml). For static files only, a cache with compressed files is kept."`
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
WebStatic *WebStatic `sconf:"optional" sconf-doc:"Serve static files."`
WebRedirect *WebRedirect `sconf:"optional" sconf-doc:"Redirect requests to configured URL."`
WebForward *WebForward `sconf:"optional" sconf-doc:"Forward requests to another webserver, i.e. reverse proxy."`
Name string `sconf:"-"` // Either LogName, or numeric index if LogName was empty. Used instead of LogName in logging/metrics.
DNSDomain dns.Domain `sconf:"-"`
Path *regexp.Regexp `sconf:"-" json:"-"`
}
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config - make builtin http handlers serve on specific domains, such as for mta-sts, so e.g. /.well-known/mta-sts.txt isn't served on all domains. - add logging of a few more fields in access logging. - small tweaks/bug fixes in webserver request handling. - add config option for redirecting entire domains to another (common enough). - split httpserver metric into two: one for duration until writing header (i.e. performance of server), another for duration until full response is sent to client (i.e. performance as perceived by users). - add admin ui, a new page for managing the configs. after making changes and hitting "save", the changes take effect immediately. the page itself doesn't look very well-designed (many input fields, makes it look messy). i have an idea to improve it (explained in admin.html as todo) by making the layout look just like the config file. not urgent though. i've already changed my websites/webapps over. the idea of adding a webserver is to take away a (the) reason for folks to want to complicate their mox setup by running an other webserver on the same machine. i think the current webserver implementation can already serve most common use cases. with a few more tweaks (feedback needed!) we should be able to get to 95% of the use cases. the reverse proxy can take care of the remaining 5%. nevertheless, a next step is still to change the quickstart to make it easier for folks to run with an existing webserver, with existing tls certs/keys. that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
// Equal returns if wh and o are equal, only looking at fields in the configuration file, not the derived fields.
func (wh WebHandler) Equal(o WebHandler) bool {
clean := func(x WebHandler) WebHandler {
x.Name = ""
x.DNSDomain = dns.Domain{}
x.Path = nil
x.WebStatic = nil
x.WebRedirect = nil
x.WebForward = nil
return x
}
cwh := clean(wh)
co := clean(o)
if cwh != co {
return false
}
if (wh.WebStatic == nil) != (o.WebStatic == nil) || (wh.WebRedirect == nil) != (o.WebRedirect == nil) || (wh.WebForward == nil) != (o.WebForward == nil) {
return false
}
if wh.WebStatic != nil {
return reflect.DeepEqual(wh.WebStatic, o.WebStatic)
}
if wh.WebRedirect != nil {
return wh.WebRedirect.equal(*o.WebRedirect)
}
if wh.WebForward != nil {
return wh.WebForward.equal(*o.WebForward)
}
return true
}
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
type WebStatic struct {
StripPrefix string `sconf:"optional" sconf-doc:"Path to strip from the request URL before evaluating to a local path. If the requested URL path does not start with this prefix and ContinueNotFound it is considered non-matching and next WebHandlers are tried. If ContinueNotFound is not set, a file not found (404) is returned in that case."`
Root string `sconf-doc:"Directory to serve files from for this handler. Keep in mind that relative paths are relative to the working directory of mox."`
ListFiles bool `sconf:"optional" sconf-doc:"If set, and a directory is requested, and no index.html is present that can be served, a file listing is returned. Results in 403 if ListFiles is not set. If a directory is requested and the URL does not end with a slash, the response is a redirect to the path with trailing slash."`
ContinueNotFound bool `sconf:"optional" sconf-doc:"If a requested URL does not exist, don't return a file not found (404) response, but consider this handler non-matching and continue attempts to serve with later WebHandlers, which may be a reverse proxy generating dynamic content, possibly even writing a static file for a next request to serve statically. If ContinueNotFound is set, HTTP requests other than GET and HEAD do not match. This mechanism can be used to implement the equivalent of 'try_files' in other webservers."`
ResponseHeaders map[string]string `sconf:"optional" sconf-doc:"Headers to add to the response. Useful for cache-control, content-type, etc. By default, Content-Type headers are automatically added for recognized file types, unless added explicitly through this setting. For directory listings, a content-type header is skipped."`
}
type WebRedirect struct {
for WebRedirect, don't "match" when the destination URL has the same scheme,host,path, for doing http -> https redirects without loops you can already get most http to https redirects through DontRedirectPlainHTTP in WebHandler, but that needs handlers for all paths. now you can just set up a redirect for a domain and all its path to baseurl https://domain (leaving other webdirect fields empty). when the request comes in with plain http, the redirect to https is done. that next request will also evaluate the same redirect rule. but it will not cause a match because it would redirect to the same scheme,host,path. so next webhandlers get a chance to serve. also clarify in webhandlers docs that also account & admin built-in handlers run first. related to issue #16
2023-03-09 01:29:44 +03:00
BaseURL string `sconf:"optional" sconf-doc:"Base URL to redirect to. The path must be empty and will be replaced, either by the request URL path, or by OrigPathRegexp/ReplacePath. Scheme, host, port and fragment stay intact, and query strings are combined. If empty, the response redirects to a different path through OrigPathRegexp and ReplacePath, which must then be set. Use a URL without scheme to redirect without changing the protocol, e.g. //newdomain/. If a redirect would send a request to a URL with the same scheme, host and path, the WebRedirect does not match so a next WebHandler can be tried. This can be used to redirect all plain http traffic to https."`
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
OrigPathRegexp string `sconf:"optional" sconf-doc:"Regular expression for matching path. If set and path does not match, a 404 is returned. The HTTP path used for matching always starts with a slash."`
ReplacePath string `sconf:"optional" sconf-doc:"Replacement path for destination URL based on OrigPathRegexp. Implemented with Go's Regexp.ReplaceAllString: $1 is replaced with the text of the first submatch, etc. If both OrigPathRegexp and ReplacePath are empty, BaseURL must be set and all paths are redirected unaltered."`
StatusCode int `sconf:"optional" sconf-doc:"Status code to use in redirect, e.g. 307. By default, a permanent redirect (308) is returned."`
URL *url.URL `sconf:"-" json:"-"`
OrigPath *regexp.Regexp `sconf:"-" json:"-"`
}
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config - make builtin http handlers serve on specific domains, such as for mta-sts, so e.g. /.well-known/mta-sts.txt isn't served on all domains. - add logging of a few more fields in access logging. - small tweaks/bug fixes in webserver request handling. - add config option for redirecting entire domains to another (common enough). - split httpserver metric into two: one for duration until writing header (i.e. performance of server), another for duration until full response is sent to client (i.e. performance as perceived by users). - add admin ui, a new page for managing the configs. after making changes and hitting "save", the changes take effect immediately. the page itself doesn't look very well-designed (many input fields, makes it look messy). i have an idea to improve it (explained in admin.html as todo) by making the layout look just like the config file. not urgent though. i've already changed my websites/webapps over. the idea of adding a webserver is to take away a (the) reason for folks to want to complicate their mox setup by running an other webserver on the same machine. i think the current webserver implementation can already serve most common use cases. with a few more tweaks (feedback needed!) we should be able to get to 95% of the use cases. the reverse proxy can take care of the remaining 5%. nevertheless, a next step is still to change the quickstart to make it easier for folks to run with an existing webserver, with existing tls certs/keys. that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
func (wr WebRedirect) equal(o WebRedirect) bool {
wr.URL = nil
wr.OrigPath = nil
o.URL = nil
o.OrigPath = nil
return reflect.DeepEqual(wr, o)
}
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
type WebForward struct {
StripPath bool `sconf:"optional" sconf-doc:"Strip the matching WebHandler path from the WebHandler before forwarding the request."`
add reverse proxying websocket connections if we recognize that a request for a WebForward is trying to turn the connection into a websocket, we forward it to the backend and check if the backend understands the websocket request. if so, we pass back the upgrade response and get out of the way, copying bytes between the two. we do log the total amount of bytes read from the client and written to the client. if the backend doesn't respond with a websocke response, or an invalid one, we respond with a regular non-websocket response. and we log details about the failed connection, should help with debugging and any bug reports. we don't try to parse the websocket framing, that's between the client and the backend. we could try to parse it, in part to protect the backend from bad frames, but it would be a lot of work and could be brittle in the face of extensions. this doesn't yet handle websocket connections when a http proxy is configured. we'll implement it when someone needs it. we do recognize it and fail the connection. for issue #25
2023-05-30 23:11:31 +03:00
URL string `sconf-doc:"URL to forward HTTP requests to, e.g. http://127.0.0.1:8123/base. If StripPath is false the full request path is added to the URL. Host headers are sent unmodified. New X-Forwarded-{For,Host,Proto} headers are set. Any query string in the URL is ignored. Requests are made using Go's net/http.DefaultTransport that takes environment variables HTTP_PROXY and HTTPS_PROXY into account. Websocket connections are forwarded and data is copied between client and backend without looking at the framing. The websocket 'version' and 'key'/'accept' headers are verified during the handshake, but other websocket headers, including 'origin', 'protocol' and 'extensions' headers, are not inspected and the backend is responsible for verifying/interpreting them."`
add basic webserver that can do most of what i need - serve static files, serving index.html or optionally listings for directories - redirects - reverse-proxy, forwarding requests to a backend these are configurable through the config file. a domain and path regexp have to be configured. path prefixes can be stripped. configured domains are added to the autotls allowlist, so acme automatically fetches certificates for them. all webserver requests now have (access) logging, metrics, rate limiting. on http errors, the error message prints an encrypted cid for relating with log files. this also adds a new mechanism for example config files.
2023-03-01 00:12:27 +03:00
ResponseHeaders map[string]string `sconf:"optional" sconf-doc:"Headers to add to the response. Useful for adding security- and cache-related headers."`
TargetURL *url.URL `sconf:"-" json:"-"`
}
improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config - make builtin http handlers serve on specific domains, such as for mta-sts, so e.g. /.well-known/mta-sts.txt isn't served on all domains. - add logging of a few more fields in access logging. - small tweaks/bug fixes in webserver request handling. - add config option for redirecting entire domains to another (common enough). - split httpserver metric into two: one for duration until writing header (i.e. performance of server), another for duration until full response is sent to client (i.e. performance as perceived by users). - add admin ui, a new page for managing the configs. after making changes and hitting "save", the changes take effect immediately. the page itself doesn't look very well-designed (many input fields, makes it look messy). i have an idea to improve it (explained in admin.html as todo) by making the layout look just like the config file. not urgent though. i've already changed my websites/webapps over. the idea of adding a webserver is to take away a (the) reason for folks to want to complicate their mox setup by running an other webserver on the same machine. i think the current webserver implementation can already serve most common use cases. with a few more tweaks (feedback needed!) we should be able to get to 95% of the use cases. the reverse proxy can take care of the remaining 5%. nevertheless, a next step is still to change the quickstart to make it easier for folks to run with an existing webserver, with existing tls certs/keys. that's how this relates to issue #5.
2023-03-02 20:15:54 +03:00
func (wf WebForward) equal(o WebForward) bool {
wf.TargetURL = nil
o.TargetURL = nil
return reflect.DeepEqual(wf, o)
}
Reference in a new issue Copy permalink