mox/smtpclient/client.go

1468 lines
51 KiB
Go
Raw Permalink Normal View History

// Package smtpclient is an SMTP client, for submitting to an SMTP server or
// delivering from a queue.
//
// Email clients can submit a message to SMTP server, after which the server is
// responsible for delivery to the final destination. A submission client
// typically connects with TLS, and PKIX-verifies the server's certificate. The
// client then authenticates using a SASL mechanism.
//
// Email servers manage a message queue, from which they will try to deliver
// messages. In case of temporary failures, the message is kept in the queue and
// tried again later. For delivery, no authentication is done. TLS is opportunistic
// by default (TLS certificates not verified), but TLS and certificate verification
// can be opted into by domains by specifying an MTA-STS policy for the domain, or
// DANE TLSA records for their MX hosts.
//
// Delivering a message from a queue would involve:
// 1. Looking up an MTA-STS policy, through a cache.
// 2. Resolving the MX targets for a domain, through smtpclient.GatherDestinations,
// and for each destination try delivery through:
// 3. Looking up IP addresses for the destination, with smtpclient.GatherIPs.
// 4. Looking up TLSA records for DANE, in case of authentic DNS responses
// (DNSSEC), with smtpclient.GatherTLSA.
// 5. Dialing the MX target with smtpclient.Dial.
// 6. Initializing a SMTP session with smtpclient.New, with proper TLS
// configuration based on discovered MTA-STS and DANE policies, and finally calling
// client.Deliver.
2023-01-30 16:27:06 +03:00
package smtpclient
import (
"bufio"
"bytes"
2023-01-30 16:27:06 +03:00
"context"
"crypto/tls"
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
"crypto/x509"
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
"encoding/base64"
2023-01-30 16:27:06 +03:00
"errors"
"fmt"
"io"
"log/slog"
2023-01-30 16:27:06 +03:00
"net"
"reflect"
2023-01-30 16:27:06 +03:00
"strconv"
"strings"
"time"
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
"github.com/mjl-/adns"
"github.com/mjl-/mox/dane"
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
"github.com/mjl-/mox/dns"
2023-01-30 16:27:06 +03:00
"github.com/mjl-/mox/mlog"
"github.com/mjl-/mox/moxio"
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
"github.com/mjl-/mox/sasl"
2023-01-30 16:27:06 +03:00
"github.com/mjl-/mox/smtp"
"github.com/mjl-/mox/stub"
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
"github.com/mjl-/mox/tlsrpt"
2023-01-30 16:27:06 +03:00
)
// todo future: add function to deliver message to multiple recipients. requires more elaborate return value, indicating success per message: some recipients may succeed, others may fail, and we should still deliver. to prevent backscatter, we also sometimes don't allow multiple recipients. ../rfc/5321:1144
var (
MetricCommands stub.HistogramVec = stub.HistogramVecIgnore{}
MetricTLSRequiredNoIgnored stub.CounterVec = stub.CounterVecIgnore{}
MetricPanicInc = func() {}
2023-01-30 16:27:06 +03:00
)
var (
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
ErrSize = errors.New("message too large for remote smtp server") // SMTP server announced a maximum message size and the message to be delivered exceeds it.
Err8bitmimeUnsupported = errors.New("remote smtp server does not implement 8bitmime extension, required by message")
ErrSMTPUTF8Unsupported = errors.New("remote smtp server does not implement smtputf8 extension, required by message")
ErrRequireTLSUnsupported = errors.New("remote smtp server does not implement requiretls extension, required for delivery")
ErrStatus = errors.New("remote smtp server sent unexpected response status code") // Relatively common, e.g. when a 250 OK was expected and server sent 451 temporary error.
ErrProtocol = errors.New("smtp protocol error") // After a malformed SMTP response or inconsistent multi-line response.
ErrTLS = errors.New("tls error") // E.g. handshake failure, or hostname verification was required and failed.
ErrBotched = errors.New("smtp connection is botched") // Set on a client, and returned for new operations, after an i/o error or malformed SMTP response.
ErrClosed = errors.New("client is closed")
2023-01-30 16:27:06 +03:00
)
// TLSMode indicates if TLS must, should or must not be used.
type TLSMode string
const (
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// TLS immediately ("implicit TLS"), directly starting TLS on the TCP connection,
// so not using STARTTLS. Whether PKIX and/or DANE is verified is specified
// separately.
TLSImmediate TLSMode = "immediate"
// Required TLS with STARTTLS for SMTP servers. The STARTTLS command is always
// executed, even if the server does not announce support.
// Whether PKIX and/or DANE is verified is specified separately.
TLSRequiredStartTLS TLSMode = "requiredstarttls"
// Use TLS with STARTTLS if remote claims to support it.
2023-01-30 16:27:06 +03:00
TLSOpportunistic TLSMode = "opportunistic"
// TLS must not be attempted, e.g. due to earlier TLS handshake error.
TLSSkip TLSMode = "skip"
)
// Client is an SMTP client that can deliver messages to a mail server.
//
// Use New to make a new client.
type Client struct {
// OrigConn is the original (TCP) connection. We'll read from/write to conn, which
// can be wrapped in a tls.Client. We close origConn instead of conn because
// closing the TLS connection would send a TLS close notification, which may block
// for 5s if the server isn't reading it (because it is also sending it).
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
origConn net.Conn
conn net.Conn
tlsVerifyPKIX bool
ignoreTLSVerifyErrors bool
rootCAs *x509.CertPool
implement tls client certificate authentication the imap & smtp servers now allow logging in with tls client authentication and the "external" sasl authentication mechanism. email clients like thunderbird, fairemail, k9, macos mail implement it. this seems to be the most secure among the authentication mechanism commonly implemented by clients. a useful property is that an account can have a separate tls public key for each device/email client. with tls client cert auth, authentication is also bound to the tls connection. a mitm cannot pass the credentials on to another tls connection, similar to scram-*-plus. though part of scram-*-plus is that clients verify that the server knows the client credentials. for tls client auth with imap, we send a "preauth" untagged message by default. that puts the connection in authenticated state. given the imap connection state machine, further authentication commands are not allowed. some clients don't recognize the preauth message, and try to authenticate anyway, which fails. a tls public key has a config option to disable preauth, keeping new connections in unauthenticated state, to work with such email clients. for smtp (submission), we don't require an explicit auth command. both for imap and smtp, we allow a client to authenticate with another mechanism than "external". in that case, credentials are verified, and have to be for the same account as the tls client auth, but the adress can be another one than the login address configured with the tls public key. only the public key is used to identify the account that is authenticating. we ignore the rest of the certificate. expiration dates, names, constraints, etc are not verified. no certificate authorities are involved. users can upload their own (minimal) certificate. the account web interface shows openssl commands you can run to generate a private key, minimal cert, and a p12 file (the format that email clients seem to like...) containing both private key and certificate. the imapclient & smtpclient packages can now also use tls client auth. and so does "mox sendmail", either with a pem file with private key and certificate, or with just an ed25519 private key. there are new subcommands "mox config tlspubkey ..." for adding/removing/listing tls public keys from the cli, by the admin.
2024-12-06 00:41:49 +03:00
remoteHostname dns.Domain // TLS with SNI and name verification.
daneRecords []adns.TLSA // For authenticating (START)TLS connection.
daneMoreHostnames []dns.Domain // Additional allowed names in TLS certificate for DANE-TA.
daneVerifiedRecord *adns.TLSA // If non-nil, then will be set to verified DANE record if any.
clientCert *tls.Certificate // If non-nil, tls client authentication is done.
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// TLS connection success/failure are added. These are always non-nil, regardless
// of what was passed in opts. It lets us unconditionally dereference them.
recipientDomainResult *tlsrpt.Result // Either "sts" or "no-policy-found".
hostResult *tlsrpt.Result // Either "dane" or "no-policy-found".
2023-01-30 16:27:06 +03:00
r *bufio.Reader
w *bufio.Writer
tr *moxio.TraceReader // Kept for changing trace levels between cmd/auth/data.
tw *moxio.TraceWriter
log mlog.Log
lastlog time.Time // For adding delta timestamps between log lines.
cmds []string // Last or active command, for generating errors and metrics.
cmdStart time.Time // Start of command.
tls bool // Whether connection is TLS protected.
firstReadAfterHandshake bool // To detect TLS alert error from remote just after handshake.
2023-01-30 16:27:06 +03:00
botched bool // If set, protocol is out of sync and no further commands can be sent.
needRset bool // If set, a new delivery requires an RSET command.
remoteHelo string // From 220 greeting line.
extEcodes bool // Remote server supports sending extended error codes.
extStartTLS bool // Remote server supports STARTTLS.
ext8bitmime bool
extSize bool // Remote server supports SIZE parameter. Must only be used if > 0.
maxSize int64 // Max size of email message.
extPipelining bool // Remote server supports command pipelining.
extSMTPUTF8 bool // Remote server supports SMTPUTF8 extension.
extAuthMechanisms []string // Supported authentication mechanisms.
extRequireTLS bool // Remote supports REQUIRETLS extension.
ExtLimits map[string]string // For LIMITS extension, only if present and valid, with uppercase keys.
ExtLimitMailMax int // Max "MAIL" commands in a connection, if > 0.
ExtLimitRcptMax int // Max "RCPT" commands in a transaction, if > 0.
ExtLimitRcptDomainMax int // Max unique domains in a connection, if > 0.
2023-01-30 16:27:06 +03:00
}
// Error represents a failure to deliver a message.
//
// Code, Secode, Command and Line are only set for SMTP-level errors, and are zero
// values otherwise.
type Error struct {
// Whether failure is permanent, typically because of 5xx response.
Permanent bool
// SMTP response status, e.g. 2xx for success, 4xx for transient error and 5xx for
// permanent failure.
Code int
// Short enhanced status, minus first digit and dot. Can be empty, e.g. for io
// errors or if remote does not send enhanced status codes. If remote responds with
// "550 5.7.1 ...", the Secode will be "7.1".
Secode string
// SMTP command causing failure.
Command string
// For errors due to SMTP responses, the full SMTP line excluding CRLF that caused
// the error. First line of a multi-line response.
2023-01-30 16:27:06 +03:00
Line string
// Optional additional lines in case of multi-line SMTP response. Most SMTP
// responses are single-line, leaving this field empty.
MoreLines []string
2023-01-30 16:27:06 +03:00
// Underlying error, e.g. one of the Err variables in this package, or io errors.
Err error
}
type Response Error
2023-01-30 16:27:06 +03:00
// Unwrap returns the underlying Err.
func (e Error) Unwrap() error {
return e.Err
}
// Error returns a readable error string.
func (e Error) Error() string {
s := ""
if e.Err != nil {
s = e.Err.Error() + ", "
}
if e.Permanent {
s += "permanent"
} else {
s += "transient"
}
if e.Line != "" {
s += ": " + e.Line
}
return s
}
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// Opts influence behaviour of Client.
type Opts struct {
// If auth is non-nil, authentication will be done with the returned sasl client.
// The function should select the preferred mechanism. Mechanisms are in upper
// case.
//
// The TLS connection state can be used for the SCRAM PLUS mechanisms, binding the
// authentication exchange to a TLS connection. It is only present for TLS
// connections.
//
// If no mechanism is supported, a nil client and nil error can be returned, and
// the connection will fail.
Auth func(mechanisms []string, cs *tls.ConnectionState) (sasl.Client, error)
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
DANERecords []adns.TLSA // If not nil, DANE records to verify.
DANEMoreHostnames []dns.Domain // For use with DANE, where additional certificate host names are allowed.
DANEVerifiedRecord *adns.TLSA // If non-empty, set to the DANE record that verified the TLS connection.
// If set, TLS verification errors (for DANE or PKIX) are ignored. Useful for
// delivering messages with message header "TLS-Required: No".
// Certificates are still verified, and results are still tracked for TLS
// reporting, but the connections will continue.
IgnoreTLSVerifyErrors bool
// If not nil, used instead of the system default roots for TLS PKIX verification.
RootCAs *x509.CertPool
implement tls client certificate authentication the imap & smtp servers now allow logging in with tls client authentication and the "external" sasl authentication mechanism. email clients like thunderbird, fairemail, k9, macos mail implement it. this seems to be the most secure among the authentication mechanism commonly implemented by clients. a useful property is that an account can have a separate tls public key for each device/email client. with tls client cert auth, authentication is also bound to the tls connection. a mitm cannot pass the credentials on to another tls connection, similar to scram-*-plus. though part of scram-*-plus is that clients verify that the server knows the client credentials. for tls client auth with imap, we send a "preauth" untagged message by default. that puts the connection in authenticated state. given the imap connection state machine, further authentication commands are not allowed. some clients don't recognize the preauth message, and try to authenticate anyway, which fails. a tls public key has a config option to disable preauth, keeping new connections in unauthenticated state, to work with such email clients. for smtp (submission), we don't require an explicit auth command. both for imap and smtp, we allow a client to authenticate with another mechanism than "external". in that case, credentials are verified, and have to be for the same account as the tls client auth, but the adress can be another one than the login address configured with the tls public key. only the public key is used to identify the account that is authenticating. we ignore the rest of the certificate. expiration dates, names, constraints, etc are not verified. no certificate authorities are involved. users can upload their own (minimal) certificate. the account web interface shows openssl commands you can run to generate a private key, minimal cert, and a p12 file (the format that email clients seem to like...) containing both private key and certificate. the imapclient & smtpclient packages can now also use tls client auth. and so does "mox sendmail", either with a pem file with private key and certificate, or with just an ed25519 private key. there are new subcommands "mox config tlspubkey ..." for adding/removing/listing tls public keys from the cli, by the admin.
2024-12-06 00:41:49 +03:00
// If set, the TLS client certificate authentication is done.
ClientCert *tls.Certificate
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// TLS verification successes/failures is added to these TLS reporting results.
// Once the STARTTLS handshake is attempted, a successful/failed connection is
// tracked.
RecipientDomainResult *tlsrpt.Result // MTA-STS or no policy.
HostResult *tlsrpt.Result // DANE or no policy.
}
2023-01-30 16:27:06 +03:00
// New initializes an SMTP session on the given connection, returning a client that
// can be used to deliver messages.
//
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
// New optionally starts TLS (for submission), reads the server greeting,
// identifies itself with a HELO or EHLO command, initializes TLS with STARTTLS if
// remote supports it and optionally authenticates. If successful, a client is
// returned on which eventually Close must be called. Otherwise an error is
// returned and the caller is responsible for closing the connection.
2023-01-30 16:27:06 +03:00
//
// Connecting to the correct host for delivery can be done using the Gather
// functions, and with Dial. The queue managing outgoing messages typically decides
// which host to deliver to, taking multiple MX records with preferences, other DNS
// records, MTA-STS, retries and special cases into account.
//
// tlsMode indicates if and how TLS may/must (not) be used.
//
// tlsVerifyPKIX indicates if TLS certificates must be validated against the
// PKIX/WebPKI certificate authorities (if TLS is done).
//
// DANE-verification is done when opts.DANERecords is not nil.
//
// TLS verification errors will be ignored if opts.IgnoreTLSVerification is set.
//
// If TLS is done, PKIX verification is always performed for tracking the results
// for TLS reporting, but if tlsVerifyPKIX is false, the verification result does
// not affect the connection.
2023-01-30 16:27:06 +03:00
//
// At the time of writing, delivery of email on the internet is done with
// opportunistic TLS without PKIX verification by default. Recipient domains can
// opt-in to PKIX verification by publishing an MTA-STS policy, or opt-in to DANE
// verification by publishing DNSSEC-protected TLSA records in DNS.
func New(ctx context.Context, elog *slog.Logger, conn net.Conn, tlsMode TLSMode, tlsVerifyPKIX bool, ehloHostname, remoteHostname dns.Domain, opts Opts) (*Client, error) {
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
ensureResult := func(r *tlsrpt.Result) *tlsrpt.Result {
if r == nil {
return &tlsrpt.Result{}
}
return r
}
2023-01-30 16:27:06 +03:00
c := &Client{
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
origConn: conn,
tlsVerifyPKIX: tlsVerifyPKIX,
ignoreTLSVerifyErrors: opts.IgnoreTLSVerifyErrors,
rootCAs: opts.RootCAs,
remoteHostname: remoteHostname,
daneRecords: opts.DANERecords,
daneMoreHostnames: opts.DANEMoreHostnames,
daneVerifiedRecord: opts.DANEVerifiedRecord,
implement tls client certificate authentication the imap & smtp servers now allow logging in with tls client authentication and the "external" sasl authentication mechanism. email clients like thunderbird, fairemail, k9, macos mail implement it. this seems to be the most secure among the authentication mechanism commonly implemented by clients. a useful property is that an account can have a separate tls public key for each device/email client. with tls client cert auth, authentication is also bound to the tls connection. a mitm cannot pass the credentials on to another tls connection, similar to scram-*-plus. though part of scram-*-plus is that clients verify that the server knows the client credentials. for tls client auth with imap, we send a "preauth" untagged message by default. that puts the connection in authenticated state. given the imap connection state machine, further authentication commands are not allowed. some clients don't recognize the preauth message, and try to authenticate anyway, which fails. a tls public key has a config option to disable preauth, keeping new connections in unauthenticated state, to work with such email clients. for smtp (submission), we don't require an explicit auth command. both for imap and smtp, we allow a client to authenticate with another mechanism than "external". in that case, credentials are verified, and have to be for the same account as the tls client auth, but the adress can be another one than the login address configured with the tls public key. only the public key is used to identify the account that is authenticating. we ignore the rest of the certificate. expiration dates, names, constraints, etc are not verified. no certificate authorities are involved. users can upload their own (minimal) certificate. the account web interface shows openssl commands you can run to generate a private key, minimal cert, and a p12 file (the format that email clients seem to like...) containing both private key and certificate. the imapclient & smtpclient packages can now also use tls client auth. and so does "mox sendmail", either with a pem file with private key and certificate, or with just an ed25519 private key. there are new subcommands "mox config tlspubkey ..." for adding/removing/listing tls public keys from the cli, by the admin.
2024-12-06 00:41:49 +03:00
clientCert: opts.ClientCert,
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
lastlog: time.Now(),
cmds: []string{"(none)"},
recipientDomainResult: ensureResult(opts.RecipientDomainResult),
hostResult: ensureResult(opts.HostResult),
2023-01-30 16:27:06 +03:00
}
c.log = mlog.New("smtpclient", elog).WithFunc(func() []slog.Attr {
2023-01-30 16:27:06 +03:00
now := time.Now()
l := []slog.Attr{
slog.Duration("delta", now.Sub(c.lastlog)),
2023-01-30 16:27:06 +03:00
}
c.lastlog = now
return l
})
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
if tlsMode == TLSImmediate {
config := c.tlsConfig()
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
tlsconn := tls.Client(conn, config)
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// The tlsrpt tracking isn't used by caller, but won't hurt.
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
if err := tlsconn.HandshakeContext(ctx); err != nil {
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
c.tlsResultAdd(0, 1, err)
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
return nil, err
}
c.firstReadAfterHandshake = true
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
c.tlsResultAdd(1, 0, nil)
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
c.conn = tlsconn
tlsversion, ciphersuite := moxio.TLSInfo(tlsconn)
c.log.Debug("tls client handshake done",
slog.String("tls", tlsversion),
slog.String("ciphersuite", ciphersuite),
slog.Any("servername", remoteHostname))
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
c.tls = true
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
} else {
c.conn = conn
}
2023-01-30 16:27:06 +03:00
// We don't wrap reads in a timeoutReader for fear of an optional TLS wrapper doing
// reads without the client asking for it. Such reads could result in a timeout
// error.
c.tr = moxio.NewTraceReader(c.log, "RS: ", c.conn)
c.r = bufio.NewReader(c.tr)
2023-01-30 16:27:06 +03:00
// We use a single write timeout of 30 seconds.
// todo future: use different timeouts ../rfc/5321:3610
c.tw = moxio.NewTraceWriter(c.log, "LC: ", timeoutWriter{c.conn, 30 * time.Second, c.log})
c.w = bufio.NewWriter(c.tw)
2023-01-30 16:27:06 +03:00
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
if err := c.hello(ctx, tlsMode, ehloHostname, opts.Auth); err != nil {
2023-01-30 16:27:06 +03:00
return nil, err
}
return c, nil
}
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// reportedError wraps an error while indicating it was already tracked for TLS
// reporting.
type reportedError struct{ err error }
func (e reportedError) Error() string {
return e.err.Error()
}
func (e reportedError) Unwrap() error {
return e.err
}
func (c *Client) tlsConfig() *tls.Config {
// We always manage verification ourselves: We need to report in detail about
// failures. And we may have to verify both PKIX and DANE, record errors for
// each, and possibly ignore the errors.
verifyConnection := func(cs tls.ConnectionState) error {
// Collect verification errors. If there are none at the end, TLS validation
// succeeded. We may find validation problems below, record them for a TLS report
// but continue due to policies. We track the TLS reporting result in this
// function, wrapping errors in a reportedError.
var daneErr, pkixErr error
// DANE verification.
// daneRecords can be non-nil and empty, that's intended.
if c.daneRecords != nil {
verified, record, err := dane.Verify(c.log.Logger, c.daneRecords, cs, c.remoteHostname, c.daneMoreHostnames, c.rootCAs)
c.log.Debugx("dane verification", err, slog.Bool("verified", verified), slog.Any("record", record))
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
if verified {
if c.daneVerifiedRecord != nil {
*c.daneVerifiedRecord = record
}
} else {
// Track error for reports.
// todo spec: may want to propose adding a result for no-dane-match. dane allows multiple records, some mismatching/failing isn't fatal and reporting on each record is probably not productive. ../rfc/8460:541
fd := c.tlsrptFailureDetails(tlsrpt.ResultValidationFailure, "dane-no-match")
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
if err != nil {
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// todo future: potentially add more details. e.g. dane-ta verification errors. tlsrpt does not have "result types" to indicate those kinds of errors. we would probably have to pass c.daneResult to dane.Verify.
// We may have encountered errors while evaluation some of the TLSA records.
fd.FailureReasonCode += "+errors"
}
c.hostResult.Add(0, 0, fd)
if c.ignoreTLSVerifyErrors {
// We ignore the failure and continue the connection.
c.log.Infox("verifying dane failed, continuing with connection", err)
MetricTLSRequiredNoIgnored.IncLabels("daneverification")
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
} else {
// This connection will fail.
daneErr = dane.ErrNoMatch
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
}
}
}
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// PKIX verification.
opts := x509.VerifyOptions{
DNSName: cs.ServerName,
Intermediates: x509.NewCertPool(),
Roots: c.rootCAs,
}
for _, cert := range cs.PeerCertificates[1:] {
opts.Intermediates.AddCert(cert)
}
if _, err := cs.PeerCertificates[0].Verify(opts); err != nil {
resultType, reasonCode := tlsrpt.TLSFailureDetails(err)
fd := c.tlsrptFailureDetails(resultType, reasonCode)
c.recipientDomainResult.Add(0, 0, fd)
if c.tlsVerifyPKIX && !c.ignoreTLSVerifyErrors {
pkixErr = err
}
}
if daneErr != nil && pkixErr != nil {
return reportedError{errors.Join(daneErr, pkixErr)}
} else if daneErr != nil {
return reportedError{daneErr}
} else if pkixErr != nil {
return reportedError{pkixErr}
}
return nil
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
}
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
implement tls client certificate authentication the imap & smtp servers now allow logging in with tls client authentication and the "external" sasl authentication mechanism. email clients like thunderbird, fairemail, k9, macos mail implement it. this seems to be the most secure among the authentication mechanism commonly implemented by clients. a useful property is that an account can have a separate tls public key for each device/email client. with tls client cert auth, authentication is also bound to the tls connection. a mitm cannot pass the credentials on to another tls connection, similar to scram-*-plus. though part of scram-*-plus is that clients verify that the server knows the client credentials. for tls client auth with imap, we send a "preauth" untagged message by default. that puts the connection in authenticated state. given the imap connection state machine, further authentication commands are not allowed. some clients don't recognize the preauth message, and try to authenticate anyway, which fails. a tls public key has a config option to disable preauth, keeping new connections in unauthenticated state, to work with such email clients. for smtp (submission), we don't require an explicit auth command. both for imap and smtp, we allow a client to authenticate with another mechanism than "external". in that case, credentials are verified, and have to be for the same account as the tls client auth, but the adress can be another one than the login address configured with the tls public key. only the public key is used to identify the account that is authenticating. we ignore the rest of the certificate. expiration dates, names, constraints, etc are not verified. no certificate authorities are involved. users can upload their own (minimal) certificate. the account web interface shows openssl commands you can run to generate a private key, minimal cert, and a p12 file (the format that email clients seem to like...) containing both private key and certificate. the imapclient & smtpclient packages can now also use tls client auth. and so does "mox sendmail", either with a pem file with private key and certificate, or with just an ed25519 private key. there are new subcommands "mox config tlspubkey ..." for adding/removing/listing tls public keys from the cli, by the admin.
2024-12-06 00:41:49 +03:00
var certs []tls.Certificate
if c.clientCert != nil {
certs = []tls.Certificate{*c.clientCert}
}
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
return &tls.Config{
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
ServerName: c.remoteHostname.ASCII, // For SNI.
// todo: possibly accept older TLS versions for TLSOpportunistic? or would our private key be at risk?
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
MinVersion: tls.VersionTLS12, // ../rfc/8996:31 ../rfc/8997:66
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
InsecureSkipVerify: true, // VerifyConnection below is called and will do all verification.
VerifyConnection: verifyConnection,
implement tls client certificate authentication the imap & smtp servers now allow logging in with tls client authentication and the "external" sasl authentication mechanism. email clients like thunderbird, fairemail, k9, macos mail implement it. this seems to be the most secure among the authentication mechanism commonly implemented by clients. a useful property is that an account can have a separate tls public key for each device/email client. with tls client cert auth, authentication is also bound to the tls connection. a mitm cannot pass the credentials on to another tls connection, similar to scram-*-plus. though part of scram-*-plus is that clients verify that the server knows the client credentials. for tls client auth with imap, we send a "preauth" untagged message by default. that puts the connection in authenticated state. given the imap connection state machine, further authentication commands are not allowed. some clients don't recognize the preauth message, and try to authenticate anyway, which fails. a tls public key has a config option to disable preauth, keeping new connections in unauthenticated state, to work with such email clients. for smtp (submission), we don't require an explicit auth command. both for imap and smtp, we allow a client to authenticate with another mechanism than "external". in that case, credentials are verified, and have to be for the same account as the tls client auth, but the adress can be another one than the login address configured with the tls public key. only the public key is used to identify the account that is authenticating. we ignore the rest of the certificate. expiration dates, names, constraints, etc are not verified. no certificate authorities are involved. users can upload their own (minimal) certificate. the account web interface shows openssl commands you can run to generate a private key, minimal cert, and a p12 file (the format that email clients seem to like...) containing both private key and certificate. the imapclient & smtpclient packages can now also use tls client auth. and so does "mox sendmail", either with a pem file with private key and certificate, or with just an ed25519 private key. there are new subcommands "mox config tlspubkey ..." for adding/removing/listing tls public keys from the cli, by the admin.
2024-12-06 00:41:49 +03:00
Certificates: certs,
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
}
}
2023-01-30 16:27:06 +03:00
// xbotchf generates a temporary error and marks the client as botched. e.g. for
// i/o errors or invalid protocol messages.
func (c *Client) xbotchf(code int, secode string, firstLine string, moreLines []string, format string, args ...any) {
panic(c.botchf(code, secode, firstLine, moreLines, format, args...))
}
// botchf generates a temporary error and marks the client as botched. e.g. for
// i/o errors or invalid protocol messages.
func (c *Client) botchf(code int, secode string, firstLine string, moreLines []string, format string, args ...any) error {
2023-01-30 16:27:06 +03:00
c.botched = true
return c.errorf(false, code, secode, firstLine, moreLines, format, args...)
2023-01-30 16:27:06 +03:00
}
func (c *Client) errorf(permanent bool, code int, secode, firstLine string, moreLines []string, format string, args ...any) error {
2023-01-30 16:27:06 +03:00
var cmd string
if len(c.cmds) > 0 {
cmd = c.cmds[0]
}
return Error{permanent, code, secode, cmd, firstLine, moreLines, fmt.Errorf(format, args...)}
}
func (c *Client) xerrorf(permanent bool, code int, secode, firstLine string, moreLines []string, format string, args ...any) {
panic(c.errorf(permanent, code, secode, firstLine, moreLines, format, args...))
2023-01-30 16:27:06 +03:00
}
// timeoutWriter passes each Write on to conn after setting a write deadline on conn based on
// timeout.
type timeoutWriter struct {
conn net.Conn
timeout time.Duration
log mlog.Log
2023-01-30 16:27:06 +03:00
}
func (w timeoutWriter) Write(buf []byte) (int, error) {
if err := w.conn.SetWriteDeadline(time.Now().Add(w.timeout)); err != nil {
w.log.Errorx("setting write deadline", err)
}
return w.conn.Write(buf)
}
var bufs = moxio.NewBufpool(8, 2*1024)
func (c *Client) readline() (string, error) {
2023-01-30 16:27:06 +03:00
// todo: could have per-operation timeouts. and rfc suggests higher minimum timeouts. ../rfc/5321:3610
if err := c.conn.SetReadDeadline(time.Now().Add(30 * time.Second)); err != nil {
c.log.Errorx("setting read deadline", err)
}
line, err := bufs.Readline(c.log, c.r)
2023-01-30 16:27:06 +03:00
if err != nil {
// See if this is a TLS alert from remote, and one other than 0 (which notifies
// that the connection is being closed. If so, we register a TLS connection
// failure. This handles TLS alerts that happen just after a successful handshake.
var netErr *net.OpError
if c.firstReadAfterHandshake && errors.As(err, &netErr) && netErr.Op == "remote error" && netErr.Err != nil && reflect.ValueOf(netErr.Err).Kind() == reflect.Uint8 && reflect.ValueOf(netErr.Err).Uint() != 0 {
resultType, reasonCode := tlsrpt.TLSFailureDetails(err)
// We count -1 success to compensate for the assumed success right after the handshake.
c.tlsResultAddFailureDetails(-1, 1, c.tlsrptFailureDetails(resultType, reasonCode))
}
return line, c.botchf(0, "", "", nil, "%s: %w", strings.Join(c.cmds, ","), err)
2023-01-30 16:27:06 +03:00
}
c.firstReadAfterHandshake = false
return line, nil
2023-01-30 16:27:06 +03:00
}
func (c *Client) xtrace(level slog.Level) func() {
c.xflush()
c.tr.SetTrace(level)
c.tw.SetTrace(level)
return func() {
c.xflush()
c.tr.SetTrace(mlog.LevelTrace)
c.tw.SetTrace(mlog.LevelTrace)
}
}
2023-01-30 16:27:06 +03:00
func (c *Client) xwritelinef(format string, args ...any) {
c.xbwritelinef(format, args...)
c.xflush()
}
func (c *Client) xwriteline(line string) {
c.xbwriteline(line)
c.xflush()
}
func (c *Client) xbwritelinef(format string, args ...any) {
c.xbwriteline(fmt.Sprintf(format, args...))
}
func (c *Client) xbwriteline(line string) {
_, err := fmt.Fprintf(c.w, "%s\r\n", line)
if err != nil {
c.xbotchf(0, "", "", nil, "write: %w", err)
2023-01-30 16:27:06 +03:00
}
}
func (c *Client) xflush() {
err := c.w.Flush()
if err != nil {
c.xbotchf(0, "", "", nil, "writes: %w", err)
2023-01-30 16:27:06 +03:00
}
}
// read response, possibly multiline, with supporting extended codes based on configuration in client.
func (c *Client) xread() (code int, secode, firstLine string, moreLines []string) {
var err error
code, secode, firstLine, moreLines, err = c.read()
if err != nil {
panic(err)
}
return
}
func (c *Client) read() (code int, secode, firstLine string, moreLines []string, rerr error) {
code, secode, _, firstLine, moreLines, _, rerr = c.readecode(c.extEcodes)
return
2023-01-30 16:27:06 +03:00
}
// read response, possibly multiline.
// if ecodes, extended codes are parsed.
func (c *Client) readecode(ecodes bool) (code int, secode, lastText, firstLine string, moreLines, moreTexts []string, rerr error) {
first := true
2023-01-30 16:27:06 +03:00
for {
co, sec, text, line, last, err := c.read1(ecodes)
if first {
firstLine = line
first = false
} else if line != "" {
moreLines = append(moreLines, line)
if text != "" {
moreTexts = append(moreTexts, text)
}
}
if err != nil {
rerr = err
return
}
2023-01-30 16:27:06 +03:00
if code != 0 && co != code {
// ../rfc/5321:2771
err := c.botchf(0, "", firstLine, moreLines, "%w: multiline response with different codes, previous %d, last %d", ErrProtocol, code, co)
return 0, "", "", "", nil, nil, err
2023-01-30 16:27:06 +03:00
}
code = co
if last {
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
if code != smtp.C334ContinueAuth {
cmd := ""
if len(c.cmds) > 0 {
cmd = c.cmds[0]
// We only keep the last, so we're not creating new slices all the time.
if len(c.cmds) > 1 {
c.cmds = c.cmds[1:]
}
2023-01-30 16:27:06 +03:00
}
MetricCommands.ObserveLabels(float64(time.Since(c.cmdStart))/float64(time.Second), cmd, fmt.Sprintf("%d", co), sec)
c.log.Debug("smtpclient command result",
slog.String("cmd", cmd),
slog.Int("code", co),
slog.String("secode", sec),
slog.Duration("duration", time.Since(c.cmdStart)))
2023-01-30 16:27:06 +03:00
}
return co, sec, text, firstLine, moreLines, moreTexts, nil
2023-01-30 16:27:06 +03:00
}
}
}
func (c *Client) xreadecode(ecodes bool) (code int, secode, lastText, firstLine string, moreLines, moreTexts []string) {
var err error
code, secode, lastText, firstLine, moreLines, moreTexts, err = c.readecode(ecodes)
if err != nil {
panic(err)
}
return
}
2023-01-30 16:27:06 +03:00
// read single response line.
// if ecodes, extended codes are parsed.
func (c *Client) read1(ecodes bool) (code int, secode, text, line string, last bool, rerr error) {
line, rerr = c.readline()
if rerr != nil {
return
}
2023-01-30 16:27:06 +03:00
i := 0
for ; i < len(line) && line[i] >= '0' && line[i] <= '9'; i++ {
}
if i != 3 {
rerr = c.botchf(0, "", line, nil, "%w: expected response code: %s", ErrProtocol, line)
return
2023-01-30 16:27:06 +03:00
}
v, err := strconv.ParseInt(line[:i], 10, 32)
if err != nil {
rerr = c.botchf(0, "", line, nil, "%w: bad response code (%s): %s", ErrProtocol, err, line)
return
2023-01-30 16:27:06 +03:00
}
code = int(v)
major := code / 100
s := line[3:]
if strings.HasPrefix(s, "-") || strings.HasPrefix(s, " ") {
last = s[0] == ' '
s = s[1:]
} else if s == "" {
// Allow missing space. ../rfc/5321:2570 ../rfc/5321:2612
last = true
} else {
rerr = c.botchf(0, "", line, nil, "%w: expected space or dash after response code: %s", ErrProtocol, line)
return
2023-01-30 16:27:06 +03:00
}
if ecodes {
secode, s = parseEcode(major, s)
}
return code, secode, s, line, last, nil
2023-01-30 16:27:06 +03:00
}
func parseEcode(major int, s string) (secode string, remain string) {
o := 0
bad := false
take := func(need bool, a, b byte) bool {
if !bad && o < len(s) && s[o] >= a && s[o] <= b {
o++
return true
}
bad = bad || need
return false
}
digit := func(need bool) bool {
return take(need, '0', '9')
}
dot := func() bool {
return take(true, '.', '.')
}
digit(true)
dot()
xo := o
digit(true)
for digit(false) {
}
dot()
digit(true)
for digit(false) {
}
secode = s[xo:o]
take(false, ' ', ' ')
if bad || int(s[0])-int('0') != major {
return "", s
}
return secode, s[o:]
}
func (c *Client) recover(rerr *error) {
x := recover()
if x == nil {
return
}
cerr, ok := x.(Error)
if !ok {
MetricPanicInc()
2023-01-30 16:27:06 +03:00
panic(x)
}
*rerr = cerr
}
func (c *Client) hello(ctx context.Context, tlsMode TLSMode, ehloHostname dns.Domain, auth func(mechanisms []string, cs *tls.ConnectionState) (sasl.Client, error)) (rerr error) {
2023-01-30 16:27:06 +03:00
defer c.recover(&rerr)
// perform EHLO handshake, falling back to HELO if server does not appear to
// implement EHLO.
hello := func(heloOK bool) {
// Write EHLO and parse the supported extensions.
// ../rfc/5321:987
c.cmds[0] = "ehlo"
c.cmdStart = time.Now()
// Syntax: ../rfc/5321:1827
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
c.xwritelinef("EHLO %s", ehloHostname.ASCII)
code, _, _, firstLine, moreLines, moreTexts := c.xreadecode(false)
2023-01-30 16:27:06 +03:00
switch code {
// ../rfc/5321:997
// ../rfc/5321:3098
case smtp.C500BadSyntax, smtp.C501BadParamSyntax, smtp.C502CmdNotImpl, smtp.C503BadCmdSeq, smtp.C504ParamNotImpl:
if !heloOK {
c.xerrorf(true, code, "", firstLine, moreLines, "%w: remote claims ehlo is not supported", ErrProtocol)
2023-01-30 16:27:06 +03:00
}
// ../rfc/5321:996
c.cmds[0] = "helo"
c.cmdStart = time.Now()
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
c.xwritelinef("HELO %s", ehloHostname.ASCII)
code, _, _, firstLine, _, _ = c.xreadecode(false)
2023-01-30 16:27:06 +03:00
if code != smtp.C250Completed {
c.xerrorf(code/100 == 5, code, "", firstLine, moreLines, "%w: expected 250 to HELO, got %d", ErrStatus, code)
2023-01-30 16:27:06 +03:00
}
return
case smtp.C250Completed:
default:
c.xerrorf(code/100 == 5, code, "", firstLine, moreLines, "%w: expected 250, got %d", ErrStatus, code)
2023-01-30 16:27:06 +03:00
}
for _, s := range moreTexts {
2023-01-30 16:27:06 +03:00
// ../rfc/5321:1869
s = strings.ToUpper(strings.TrimSpace(s))
switch s {
case "STARTTLS":
c.extStartTLS = true
case "ENHANCEDSTATUSCODES":
c.extEcodes = true
case "8BITMIME":
c.ext8bitmime = true
case "PIPELINING":
c.extPipelining = true
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
case "REQUIRETLS":
c.extRequireTLS = true
2023-01-30 16:27:06 +03:00
default:
// For SMTPUTF8 we must ignore any parameter. ../rfc/6531:207
if s == "SMTPUTF8" || strings.HasPrefix(s, "SMTPUTF8 ") {
c.extSMTPUTF8 = true
} else if strings.HasPrefix(s, "SIZE ") {
// ../rfc/1870:77
2023-01-30 16:27:06 +03:00
c.extSize = true
if v, err := strconv.ParseInt(s[len("SIZE "):], 10, 64); err == nil {
c.maxSize = v
}
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
} else if strings.HasPrefix(s, "AUTH ") {
c.extAuthMechanisms = strings.Split(s[len("AUTH "):], " ")
} else if strings.HasPrefix(s, "LIMITS ") {
c.ExtLimits, c.ExtLimitMailMax, c.ExtLimitRcptMax, c.ExtLimitRcptDomainMax = parseLimits([]byte(s[len("LIMITS"):]))
2023-01-30 16:27:06 +03:00
}
}
}
}
// Read greeting.
c.cmds = []string{"(greeting)"}
c.cmdStart = time.Now()
code, _, _, firstLine, moreLines, _ := c.xreadecode(false)
2023-01-30 16:27:06 +03:00
if code != smtp.C220ServiceReady {
c.xerrorf(code/100 == 5, code, "", firstLine, moreLines, "%w: expected 220, got %d", ErrStatus, code)
2023-01-30 16:27:06 +03:00
}
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// ../rfc/5321:2588
_, c.remoteHelo, _ = strings.Cut(firstLine, " ")
2023-01-30 16:27:06 +03:00
// Write EHLO, falling back to HELO if server doesn't appear to support it.
hello(true)
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
// Attempt TLS if remote understands STARTTLS and we aren't doing immediate TLS or if caller requires it.
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
if c.extStartTLS && tlsMode == TLSOpportunistic || tlsMode == TLSRequiredStartTLS {
c.log.Debug("starting tls client", slog.Any("tlsmode", tlsMode), slog.Any("servername", c.remoteHostname))
2023-01-30 16:27:06 +03:00
c.cmds[0] = "starttls"
c.cmdStart = time.Now()
c.xwritelinef("STARTTLS")
code, secode, firstLine, _ := c.xread()
2023-01-30 16:27:06 +03:00
// ../rfc/3207:107
if code != smtp.C220ServiceReady {
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
c.tlsResultAddFailureDetails(0, 1, c.tlsrptFailureDetails(tlsrpt.ResultSTARTTLSNotSupported, fmt.Sprintf("smtp-starttls-reply-code-%d", code)))
c.xerrorf(code/100 == 5, code, secode, firstLine, moreLines, "%w: STARTTLS: got %d, expected 220", ErrTLS, code)
2023-01-30 16:27:06 +03:00
}
// We don't want to do TLS on top of c.r because it also prints protocol traces: We
// don't want to log the TLS stream. So we'll do TLS on the underlying connection,
// but make sure any bytes already read and in the buffer are used for the TLS
// handshake.
conn := c.conn
if n := c.r.Buffered(); n > 0 {
conn = &moxio.PrefixConn{
PrefixReader: io.LimitReader(c.r, int64(n)),
Conn: conn,
}
}
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
tlsConfig := c.tlsConfig()
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
nconn := tls.Client(conn, tlsConfig)
2023-01-30 16:27:06 +03:00
c.conn = nconn
nctx, cancel := context.WithTimeout(ctx, time.Minute)
defer cancel()
err := nconn.HandshakeContext(nctx)
if err != nil {
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// For each STARTTLS failure, we track a failed TLS session. For deliveries with
// multiple MX targets, we may add multiple failures, and delivery may succeed with
// a later MX target with which we can do STARTTLS. ../rfc/8460:524
c.tlsResultAdd(0, 1, err)
c.xerrorf(false, 0, "", "", nil, "%w: STARTTLS TLS handshake: %s", ErrTLS, err)
2023-01-30 16:27:06 +03:00
}
c.firstReadAfterHandshake = true
2023-01-30 16:27:06 +03:00
cancel()
c.tr = moxio.NewTraceReader(c.log, "RS: ", c.conn)
c.tw = moxio.NewTraceWriter(c.log, "LC: ", c.conn) // No need to wrap in timeoutWriter, it would just set the timeout on the underlying connection, which is still active.
c.r = bufio.NewReader(c.tr)
c.w = bufio.NewWriter(c.tw)
2023-01-30 16:27:06 +03:00
tlsversion, ciphersuite := moxio.TLSInfo(nconn)
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
c.log.Debug("starttls client handshake done",
slog.Any("tlsmode", tlsMode),
slog.Bool("verifypkix", c.tlsVerifyPKIX),
slog.Bool("verifydane", c.daneRecords != nil),
slog.Bool("ignoretlsverifyerrors", c.ignoreTLSVerifyErrors),
slog.String("tls", tlsversion),
slog.String("ciphersuite", ciphersuite),
slog.Any("servername", c.remoteHostname),
slog.Any("danerecord", c.daneVerifiedRecord))
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
c.tls = true
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
// Track successful TLS connection. ../rfc/8460:515
c.tlsResultAdd(1, 0, nil)
2023-01-30 16:27:06 +03:00
hello(false)
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
} else if tlsMode == TLSOpportunistic {
// Result: ../rfc/8460:538
c.tlsResultAddFailureDetails(0, 0, c.tlsrptFailureDetails(tlsrpt.ResultSTARTTLSNotSupported, ""))
2023-01-30 16:27:06 +03:00
}
if auth != nil {
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
return c.auth(auth)
}
return
}
// parse text after "LIMITS", including leading space.
func parseLimits(b []byte) (map[string]string, int, int, int) {
// ../rfc/9422:150
var o int
// Read next " name=value".
pair := func() ([]byte, []byte) {
if o >= len(b) || b[o] != ' ' {
return nil, nil
}
o++
ns := o
for o < len(b) {
c := b[o]
if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-' || c == '_' {
o++
} else {
break
}
}
es := o
if ns == es || o >= len(b) || b[o] != '=' {
return nil, nil
}
o++
vs := o
for o < len(b) {
c := b[o]
if c > 0x20 && c < 0x7f && c != ';' {
o++
} else {
break
}
}
if vs == o {
return nil, nil
}
return b[ns:es], b[vs:o]
}
limits := map[string]string{}
var mailMax, rcptMax, rcptDomainMax int
for o < len(b) {
name, value := pair()
if name == nil {
// We skip the entire LIMITS extension for syntax errors. ../rfc/9422:232
return nil, 0, 0, 0
}
k := strings.ToUpper(string(name))
if _, ok := limits[k]; ok {
// Not specified, but we treat duplicates as error.
return nil, 0, 0, 0
}
limits[k] = string(value)
// For individual value syntax errors, we skip that value, leaving the default 0.
// ../rfc/9422:254
switch string(name) {
case "MAILMAX":
if v, err := strconv.Atoi(string(value)); err == nil && v > 0 && len(value) <= 6 {
mailMax = v
}
case "RCPTMAX":
if v, err := strconv.Atoi(string(value)); err == nil && v > 0 && len(value) <= 6 {
rcptMax = v
}
case "RCPTDOMAINMAX":
if v, err := strconv.Atoi(string(value)); err == nil && v > 0 && len(value) <= 6 {
rcptDomainMax = v
}
}
}
return limits, mailMax, rcptMax, rcptDomainMax
}
implement outgoing tls reports we were already accepting, processing and displaying incoming tls reports. now we start tracking TLS connection and security-policy-related errors for outgoing message deliveries as well. we send reports once a day, to the reporting addresses specified in TLSRPT records (rua) of a policy domain. these reports are about MTA-STS policies and/or DANE policies, and about STARTTLS-related failures. sending reports is enabled by default, but can be disabled through setting NoOutgoingTLSReports in mox.conf. only at the end of the implementation process came the realization that the TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy for the recipient domain, and that MTA-STS and DANE TLS/policy results are typically delivered in separate reports. so MX hosts need their own TLSRPT policies. config for the per-host TLSRPT policy should be added to mox.conf for existing installs, in field HostTLSRPT. it is automatically configured by quickstart for new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin pages now suggest the per-host TLSRPT record. by creating that record, you're requesting TLS reports about your MX host. gathering all the TLS/policy results is somewhat tricky. the tentacles go throughout the code. the positive result is that the TLS/policy-related code had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect reality better, with independent settings about whether PKIX and/or DANE verification has to be done, and/or whether verification errors have to be ignored (e.g. for tls-required: no header). also, cached mtasts policies of mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:40:46 +03:00
func addrIP(addr net.Addr) string {
if t, ok := addr.(*net.TCPAddr); ok {
return t.IP.String()
}
host, _, _ := net.SplitHostPort(addr.String())
ip := net.ParseIP(host)
if ip == nil {
return "" // For pipe during tests.
}
return ip.String()
}
// tlsrptFailureDetails returns FailureDetails with connection details (such as
// IP addresses) for inclusion in a TLS report.
func (c *Client) tlsrptFailureDetails(resultType tlsrpt.ResultType, reasonCode string) tlsrpt.FailureDetails {
return tlsrpt.FailureDetails{
ResultType: resultType,
SendingMTAIP: addrIP(c.origConn.LocalAddr()),
ReceivingMXHostname: c.remoteHostname.ASCII,
ReceivingMXHelo: c.remoteHelo,
ReceivingIP: addrIP(c.origConn.RemoteAddr()),
FailedSessionCount: 1,
FailureReasonCode: reasonCode,
}
}
// tlsResultAdd adds TLS success/failure to all results.
func (c *Client) tlsResultAdd(success, failure int64, err error) {
// Only track failure if not already done so in tls.Config.VerifyConnection.
var fds []tlsrpt.FailureDetails
var repErr reportedError
if err != nil && !errors.As(err, &repErr) {
resultType, reasonCode := tlsrpt.TLSFailureDetails(err)
fd := c.tlsrptFailureDetails(resultType, reasonCode)
fds = []tlsrpt.FailureDetails{fd}
}
c.tlsResultAddFailureDetails(success, failure, fds...)
}
func (c *Client) tlsResultAddFailureDetails(success, failure int64, fds ...tlsrpt.FailureDetails) {
c.recipientDomainResult.Add(success, failure, fds...)
c.hostResult.Add(success, failure, fds...)
}
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
// ../rfc/4954:139
func (c *Client) auth(auth func(mechanisms []string, cs *tls.ConnectionState) (sasl.Client, error)) (rerr error) {
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
defer c.recover(&rerr)
c.cmds[0] = "auth"
c.cmdStart = time.Now()
mechanisms := make([]string, len(c.extAuthMechanisms))
for i, m := range c.extAuthMechanisms {
mechanisms[i] = strings.ToUpper(m)
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
a, err := auth(mechanisms, c.TLSConnectionState())
if err != nil {
c.xerrorf(true, 0, "", "", nil, "get authentication mechanism: %s, server supports %s", err, strings.Join(c.extAuthMechanisms, ", "))
} else if a == nil {
c.xerrorf(true, 0, "", "", nil, "no matching authentication mechanisms, server supports %s", strings.Join(c.extAuthMechanisms, ", "))
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
name, cleartextCreds := a.Info()
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
abort := func() (int, string, string, []string) {
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
// Abort authentication. ../rfc/4954:193
c.xwriteline("*")
// Server must respond with 501. // ../rfc/4954:195
code, secode, firstLine, moreLines := c.xread()
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
if code != smtp.C501BadParamSyntax {
c.botched = true
2023-01-30 16:27:06 +03:00
}
return code, secode, firstLine, moreLines
2023-01-30 16:27:06 +03:00
}
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
toserver, last, err := a.Next(nil)
if err != nil {
c.xerrorf(false, 0, "", "", nil, "initial step in auth mechanism %s: %w", name, err)
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
if cleartextCreds {
defer c.xtrace(mlog.LevelTraceauth)()
}
if toserver == nil {
c.xwriteline("AUTH " + name)
} else if len(toserver) == 0 {
c.xwriteline("AUTH " + name + " =") // ../rfc/4954:214
} else {
c.xwriteline("AUTH " + name + " " + base64.StdEncoding.EncodeToString(toserver))
}
for {
if cleartextCreds && last {
c.xtrace(mlog.LevelTrace) // Restore.
}
code, secode, lastText, firstLine, moreLines, _ := c.xreadecode(last)
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
if code == smtp.C235AuthSuccess {
if !last {
c.xerrorf(false, code, secode, firstLine, moreLines, "server completed authentication earlier than client expected")
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
return nil
} else if code == smtp.C334ContinueAuth {
if last {
c.xerrorf(false, code, secode, firstLine, moreLines, "server requested unexpected continuation of authentication")
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
if len(moreLines) > 0 {
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
abort()
c.xerrorf(false, code, secode, firstLine, moreLines, "server responded with multiline contination")
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
fromserver, err := base64.StdEncoding.DecodeString(lastText)
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
if err != nil {
abort()
c.xerrorf(false, code, secode, firstLine, moreLines, "malformed base64 data in authentication continuation response")
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
toserver, last, err = a.Next(fromserver)
if err != nil {
// For failing SCRAM, the client stops due to message about invalid proof. The
// server still sends an authentication result (it probably should send 501
// instead).
xcode, xsecode, xfirstLine, xmoreLines := abort()
c.xerrorf(false, xcode, xsecode, xfirstLine, xmoreLines, "client aborted authentication: %w", err)
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
c.xwriteline(base64.StdEncoding.EncodeToString(toserver))
} else {
c.xerrorf(code/100 == 5, code, secode, firstLine, moreLines, "unexpected response during authentication, expected 334 continue or 235 auth success")
new feature: when delivering messages from the queue, make it possible to use a "transport" the default transport is still just "direct delivery", where we connect to the destination domain's MX servers. other transports are: - regular smtp without authentication, this is relaying to a smarthost. - submission with authentication, e.g. to a third party email sending service. - direct delivery, but with with connections going through a socks proxy. this can be helpful if your ip is blocked, you need to get email out, and you have another IP that isn't blocked. keep in mind that for all of the above, appropriate SPF/DKIM settings have to be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM requirements cannot really be checked. which transport is used can be configured through routes. routes can be set on an account, a domain, or globally. the routes are evaluated in that order, with the first match selecting the transport. these routes are evaluated for each delivery attempt. common selection criteria are recipient domain and sender domain, but also which delivery attempt this is. you could configured mox to attempt sending through a 3rd party from the 4th attempt onwards. routes and transports are optional. if no route matches, or an empty/zero transport is selected, normal direct delivery is done. we could already "submit" emails with 3rd party accounts with "sendmail". but we now support more SASL authentication mechanisms with SMTP (not only PLAIN, but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also supports. sendmail will use the most secure mechanism supported by the server, or the explicitly configured mechanism. for issue #36 by dmikushin. also based on earlier discussion on hackernews.
2023-06-16 19:38:28 +03:00
}
}
2023-01-30 16:27:06 +03:00
}
// Supports8BITMIME returns whether the SMTP server supports the 8BITMIME
// extension, needed for sending data with non-ASCII bytes.
func (c *Client) Supports8BITMIME() bool {
return c.ext8bitmime
}
// SupportsSMTPUTF8 returns whether the SMTP server supports the SMTPUTF8
// extension, needed for sending messages with UTF-8 in headers or in an (SMTP)
// address.
func (c *Client) SupportsSMTPUTF8() bool {
return c.extSMTPUTF8
}
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
// SupportsStartTLS returns whether the SMTP server supports the STARTTLS
// extension.
func (c *Client) SupportsStartTLS() bool {
return c.extStartTLS
}
// SupportsRequireTLS returns whether the SMTP server supports the REQUIRETLS
// extension. The REQUIRETLS extension is only announced after enabling
// STARTTLS.
func (c *Client) SupportsRequireTLS() bool {
return c.extRequireTLS
}
// TLSConnectionState returns TLS details if TLS is enabled, and nil otherwise.
func (c *Client) TLSConnectionState() *tls.ConnectionState {
if tlsConn, ok := c.conn.(*tls.Conn); ok {
cs := tlsConn.ConnectionState()
return &cs
}
return nil
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
}
2023-01-30 16:27:06 +03:00
// Deliver attempts to deliver a message to a mail server.
//
// mailFrom must be an email address, or empty in case of a DSN. rcptTo must be
// an email address.
//
// If the message contains bytes with the high bit set, req8bitmime must be true. If
// set, the remote server must support the 8BITMIME extension or delivery will
// fail.
//
// If the message is internationalized, e.g. when headers contain non-ASCII
// character, or when UTF-8 is used in a localpart, reqSMTPUTF8 must be true. If set,
// the remote server must support the SMTPUTF8 extension or delivery will fail.
//
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
// If requireTLS is true, the remote server must support the REQUIRETLS
// extension, or delivery will fail.
//
2023-01-30 16:27:06 +03:00
// Deliver uses the following SMTP extensions if the remote server supports them:
// 8BITMIME, SMTPUTF8, SIZE, PIPELINING, ENHANCEDSTATUSCODES, STARTTLS.
//
// Returned errors can be of type Error, one of the Err-variables in this package
// or other underlying errors, e.g. for i/o. Use errors.Is to check.
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
func (c *Client) Deliver(ctx context.Context, mailFrom string, rcptTo string, msgSize int64, msg io.Reader, req8bitmime, reqSMTPUTF8, requireTLS bool) (rerr error) {
_, err := c.DeliverMultiple(ctx, mailFrom, []string{rcptTo}, msgSize, msg, req8bitmime, reqSMTPUTF8, requireTLS)
return err
}
var errNoRecipientsPipelined = errors.New("no recipients accepted in pipelined transaction")
var errNoRecipients = errors.New("no recipients accepted in transaction")
// DeliverMultiple is like Deliver, but attempts to deliver a message to multiple
// recipients. Errors about the entire transaction, such as i/o errors or error
// responses to the MAIL FROM or DATA commands, are returned by a non-nil rerr. If
// rcptTo has a single recipient, an error to the RCPT TO command is returned in
// rerr instead of rcptResps. Otherwise, the SMTP response for each recipient is
// returned in rcptResps.
//
// The caller should take extLimit* into account when sending. And recognize
// recipient response code "452" to mean that a recipient limit was reached,
// another transaction can be attempted immediately after instead of marking the
// delivery attempt as failed. Also code "552" must be treated like temporary error
// code "452" for historic reasons.
func (c *Client) DeliverMultiple(ctx context.Context, mailFrom string, rcptTo []string, msgSize int64, msg io.Reader, req8bitmime, reqSMTPUTF8, requireTLS bool) (rcptResps []Response, rerr error) {
2023-01-30 16:27:06 +03:00
defer c.recover(&rerr)
if len(rcptTo) == 0 {
return nil, fmt.Errorf("need at least one recipient")
}
2023-01-30 16:27:06 +03:00
if c.origConn == nil {
return nil, ErrClosed
2023-01-30 16:27:06 +03:00
} else if c.botched {
return nil, ErrBotched
2023-01-30 16:27:06 +03:00
} else if c.needRset {
if err := c.Reset(); err != nil {
return nil, err
2023-01-30 16:27:06 +03:00
}
}
if !c.ext8bitmime && req8bitmime {
// Temporary error, e.g. OpenBSD spamd does not announce 8bitmime support, but once
// you get through, the mail server behind it probably does. Just needs a few
// retries.
c.xerrorf(false, 0, "", "", nil, "%w", Err8bitmimeUnsupported)
2023-01-30 16:27:06 +03:00
}
if !c.extSMTPUTF8 && reqSMTPUTF8 {
// ../rfc/6531:313
c.xerrorf(false, 0, "", "", nil, "%w", ErrSMTPUTF8Unsupported)
2023-01-30 16:27:06 +03:00
}
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
if !c.extRequireTLS && requireTLS {
c.xerrorf(false, 0, "", "", nil, "%w", ErrRequireTLSUnsupported)
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
}
2023-01-30 16:27:06 +03:00
// Max size enforced, only when not zero. ../rfc/1870:79
if c.extSize && c.maxSize > 0 && msgSize > c.maxSize {
c.xerrorf(true, 0, "", "", nil, "%w: message is %d bytes, remote has a %d bytes maximum size", ErrSize, msgSize, c.maxSize)
2023-01-30 16:27:06 +03:00
}
var mailSize, bodyType string
if c.extSize {
mailSize = fmt.Sprintf(" SIZE=%d", msgSize)
}
if c.ext8bitmime {
if req8bitmime {
bodyType = " BODY=8BITMIME"
} else {
bodyType = " BODY=7BIT"
}
}
var smtputf8Arg string
if reqSMTPUTF8 {
// ../rfc/6531:213
smtputf8Arg = " SMTPUTF8"
}
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
var requiretlsArg string
if requireTLS {
// ../rfc/8689:155
requiretlsArg = " REQUIRETLS"
}
2023-01-30 16:27:06 +03:00
// Transaction overview: ../rfc/5321:1015
// MAIL FROM: ../rfc/5321:1879
// RCPT TO: ../rfc/5321:1916
// DATA: ../rfc/5321:1992
implement "requiretls", rfc 8689 with requiretls, the tls verification mode/rules for email deliveries can be changed by the sender/submitter. in two ways: 1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or dnssec+dane), along the entire delivery path until delivery into the final destination mailbox (so entire transport is verified-tls-protected). 2. "tls-required: no" message header, to ignore any tls and tls verification errors even if the recipient domain has a policy that requires tls verification (mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in case of misconfiguration/interoperability issues (at least useful for sending tls reports). we enable requiretls by default (only when tls is active), for smtp and submission. it can be disabled through the config. for each delivery attempt, we now store (per recipient domain, in the account of the sender) whether the smtp server supports starttls and requiretls. this support is shown (after having sent a first message) in the webmail when sending a message (the previous 3 bars under the address input field are now 5 bars, the first for starttls support, the last for requiretls support). when all recipient domains for a message are known to implement requiretls, requiretls is automatically selected for sending (instead of "default" tls behaviour). users can also select the "fallback to insecure" to add the "tls-required: no" header. new metrics are added for insight into requiretls errors and (some, not yet all) cases where tls-required-no ignored a tls/verification error. the admin can change the requiretls status for messages in the queue. so with default delivery attempts, when verified tls is required by failing, an admin could potentially change the field to "tls-required: no"-behaviour. messages received (over smtp) with the requiretls option, get a comment added to their Received header line, just before "id", after "with".
2023-10-24 11:06:16 +03:00
lineMailFrom := fmt.Sprintf("MAIL FROM:<%s>%s%s%s%s", mailFrom, mailSize, bodyType, smtputf8Arg, requiretlsArg)
2023-01-30 16:27:06 +03:00
// We are going into a transaction. We'll clear this when done.
c.needRset = true
if c.extPipelining {
c.cmds = make([]string, 1+len(rcptTo)+1)
c.cmds[0] = "mailfrom"
for i := range rcptTo {
c.cmds[1+i] = "rcptto"
}
c.cmds[len(c.cmds)-1] = "data"
2023-01-30 16:27:06 +03:00
c.cmdStart = time.Now()
// Write and read in separte goroutines. Otherwise, writing a large recipient list
// could block when a server doesn't read more commands before we read their
// response.
errc := make(chan error, 1)
// Make sure we don't return before we're done writing to the connection.
defer func() {
if errc != nil {
<-errc
}
}()
go func() {
var b bytes.Buffer
b.WriteString(lineMailFrom)
b.WriteString("\r\n")
for _, rcpt := range rcptTo {
b.WriteString("RCPT TO:<")
b.WriteString(rcpt)
b.WriteString(">\r\n")
}
b.WriteString("DATA\r\n")
_, err := c.w.Write(b.Bytes())
if err == nil {
err = c.w.Flush()
}
errc <- err
}()
// Read response to MAIL FROM.
mfcode, mfsecode, mffirstLine, mfmoreLines := c.xread()
// We read the response to RCPT TOs and DATA without panic on read error. Servers
// may be aborting the connection after a failed MAIL FROM, e.g. outlook when it
// has blocklisted your IP. We don't want the read for the response to RCPT TO to
// cause a read error as it would result in an unhelpful error message and a
// temporary instead of permanent error code.
// Read responses to RCPT TO.
rcptResps = make([]Response, len(rcptTo))
nok := 0
for i := 0; i < len(rcptTo); i++ {
code, secode, firstLine, moreLines, err := c.read()
// 552 should be treated as temporary historically, ../rfc/5321:3576
permanent := code/100 == 5 && code != smtp.C552MailboxFull
rcptResps[i] = Response{permanent, code, secode, "rcptto", firstLine, moreLines, err}
if code == smtp.C250Completed {
nok++
}
}
// Read response to DATA.
datacode, datasecode, datafirstLine, datamoreLines, dataerr := c.read()
2023-01-30 16:27:06 +03:00
writeerr := <-errc
errc = nil
// If MAIL FROM failed, it's an error for the entire transaction. We may have been
// blocked.
2023-01-30 16:27:06 +03:00
if mfcode != smtp.C250Completed {
if writeerr != nil || dataerr != nil {
c.botched = true
}
c.xerrorf(mfcode/100 == 5, mfcode, mfsecode, mffirstLine, mfmoreLines, "%w: got %d, expected 2xx", ErrStatus, mfcode)
2023-01-30 16:27:06 +03:00
}
// If there was an i/o error writing the commands, there is no point continuing.
if writeerr != nil {
c.xbotchf(0, "", "", nil, "writing pipelined mail/rcpt/data: %w", writeerr)
2023-01-30 16:27:06 +03:00
}
// If remote closed the connection before writing a DATA response, and the RCPT
// TO's failed (e.g. after deciding we're on a blocklist), use the last response
// for a rcptto as result.
if dataerr != nil && errors.Is(dataerr, io.ErrUnexpectedEOF) && nok == 0 {
c.botched = true
r := rcptResps[len(rcptResps)-1]
c.xerrorf(r.Permanent, r.Code, r.Secode, r.Line, r.MoreLines, "%w: server closed connection just before responding to data command", ErrStatus)
}
// If the data command had an i/o or protocol error, it's also a failure for the
// entire transaction.
if dataerr != nil {
panic(dataerr)
}
// If we didn't have any successful recipient, there is no point in continuing.
if nok == 0 {
// Servers may return success for a DATA without valid recipients. Write a dot to
// end DATA and restore the connection to a known state.
// ../rfc/2920:328
if datacode == smtp.C354Continue {
_, doterr := fmt.Fprintf(c.w, ".\r\n")
if doterr == nil {
doterr = c.w.Flush()
}
if doterr == nil {
_, _, _, _, doterr = c.read()
}
if doterr != nil {
c.botched = true
}
}
if len(rcptTo) == 1 {
panic(Error(rcptResps[0]))
}
c.xerrorf(false, 0, "", "", nil, "%w", errNoRecipientsPipelined)
}
2023-01-30 16:27:06 +03:00
if datacode != smtp.C354Continue {
c.xerrorf(datacode/100 == 5, datacode, datasecode, datafirstLine, datamoreLines, "%w: got %d, expected 354", ErrStatus, datacode)
2023-01-30 16:27:06 +03:00
}
2023-01-30 16:27:06 +03:00
} else {
c.cmds[0] = "mailfrom"
c.cmdStart = time.Now()
c.xwriteline(lineMailFrom)
code, secode, firstLine, moreLines := c.xread()
2023-01-30 16:27:06 +03:00
if code != smtp.C250Completed {
c.xerrorf(code/100 == 5, code, secode, firstLine, moreLines, "%w: got %d, expected 2xx", ErrStatus, code)
2023-01-30 16:27:06 +03:00
}
rcptResps = make([]Response, len(rcptTo))
nok := 0
for i, rcpt := range rcptTo {
c.cmds[0] = "rcptto"
c.cmdStart = time.Now()
c.xwriteline(fmt.Sprintf("RCPT TO:<%s>", rcpt))
code, secode, firstLine, moreLines = c.xread()
if i > 0 && (code == smtp.C452StorageFull || code == smtp.C552MailboxFull) {
// Remote doesn't accept more recipients for this transaction. Don't send more, give
// remaining recipients the same error result.
for j := i; j < len(rcptTo); j++ {
rcptResps[j] = Response{false, code, secode, "rcptto", firstLine, moreLines, fmt.Errorf("no more recipients accepted in transaction")}
}
break
}
var err error
if code == smtp.C250Completed {
nok++
} else {
err = fmt.Errorf("%w: got %d, expected 2xx", ErrStatus, code)
}
rcptResps[i] = Response{code/100 == 5, code, secode, "rcptto", firstLine, moreLines, err}
}
if nok == 0 {
if len(rcptTo) == 1 {
panic(Error(rcptResps[0]))
}
c.xerrorf(false, 0, "", "", nil, "%w", errNoRecipients)
2023-01-30 16:27:06 +03:00
}
c.cmds[0] = "data"
c.cmdStart = time.Now()
c.xwriteline("DATA")
code, secode, firstLine, moreLines = c.xread()
2023-01-30 16:27:06 +03:00
if code != smtp.C354Continue {
c.xerrorf(code/100 == 5, code, secode, firstLine, moreLines, "%w: got %d, expected 354", ErrStatus, code)
2023-01-30 16:27:06 +03:00
}
}
// For a DATA write, the suggested timeout is 3 minutes, we use 30 seconds for all
// writes through timeoutWriter. ../rfc/5321:3651
defer c.xtrace(mlog.LevelTracedata)()
2023-01-30 16:27:06 +03:00
err := smtp.DataWrite(c.w, msg)
if err != nil {
c.xbotchf(0, "", "", nil, "writing message as smtp data: %w", err)
2023-01-30 16:27:06 +03:00
}
c.xflush()
c.xtrace(mlog.LevelTrace) // Restore.
code, secode, firstLine, moreLines := c.xread()
2023-01-30 16:27:06 +03:00
if code != smtp.C250Completed {
c.xerrorf(code/100 == 5, code, secode, firstLine, moreLines, "%w: got %d, expected 2xx", ErrStatus, code)
2023-01-30 16:27:06 +03:00
}
c.needRset = false
return
}
// Reset sends an SMTP RSET command to reset the message transaction state. Deliver
// automatically sends it if needed.
func (c *Client) Reset() (rerr error) {
if c.origConn == nil {
return ErrClosed
} else if c.botched {
return ErrBotched
}
defer c.recover(&rerr)
// ../rfc/5321:2079
c.cmds[0] = "rset"
c.cmdStart = time.Now()
c.xwriteline("RSET")
code, secode, firstLine, moreLines := c.xread()
2023-01-30 16:27:06 +03:00
if code != smtp.C250Completed {
c.xerrorf(code/100 == 5, code, secode, firstLine, moreLines, "%w: got %d, expected 2xx", ErrStatus, code)
2023-01-30 16:27:06 +03:00
}
c.needRset = false
return
}
// Botched returns whether this connection is botched, e.g. a protocol error
// occurred and the connection is in unknown state, and cannot be used for message
// delivery.
func (c *Client) Botched() bool {
return c.botched || c.origConn == nil
}
// Close cleans up the client, closing the underlying connection.
//
// If the connection is initialized and not botched, a QUIT command is sent and the
// response read with a short timeout before closing the underlying connection.
2023-01-30 16:27:06 +03:00
//
// Close returns any error encountered during QUIT and closing.
func (c *Client) Close() (rerr error) {
if c.origConn == nil {
return ErrClosed
}
defer c.recover(&rerr)
if !c.botched {
// ../rfc/5321:2205
c.cmds[0] = "quit"
c.cmdStart = time.Now()
c.xwriteline("QUIT")
if err := c.conn.SetReadDeadline(time.Now().Add(5 * time.Second)); err != nil {
c.log.Infox("setting read deadline for reading quit response", err)
} else if _, err := bufs.Readline(c.log, c.r); err != nil {
2023-01-30 16:27:06 +03:00
rerr = fmt.Errorf("reading response to quit command: %v", err)
c.log.Debugx("reading quit response", err)
}
}
err := c.origConn.Close()
if c.conn != c.origConn {
// This is the TLS connection. Close will attempt to write a close notification.
// But it will fail quickly because the underlying socket was closed.
c.conn.Close()
}
c.origConn = nil
c.conn = nil
if rerr != nil {
rerr = err
}
return
}
implement dnssec-awareness throughout code, and dane for incoming/outgoing mail delivery the vendored dns resolver code is a copy of the go stdlib dns resolver, with awareness of the "authentic data" (i.e. dnssec secure) added, as well as support for enhanced dns errors, and looking up tlsa records (for dane). ideally it would be upstreamed, but the chances seem slim. dnssec-awareness is added to all packages, e.g. spf, dkim, dmarc, iprev. their dnssec status is added to the Received message headers for incoming email. but the main reason to add dnssec was for implementing dane. with dane, the verification of tls certificates can be done through certificates/public keys published in dns (in the tlsa records). this only makes sense (is trustworthy) if those dns records can be verified to be authentic. mox now applies dane to delivering messages over smtp. mox already implemented mta-sts for webpki/pkix-verification of certificates against the (large) pool of CA's, and still enforces those policies when present. but it now also checks for dane records, and will verify those if present. if dane and mta-sts are both absent, the regular opportunistic tls with starttls is still done. and the fallback to plaintext is also still done. mox also makes it easy to setup dane for incoming deliveries, so other servers can deliver with dane tls certificate verification. the quickstart now generates private keys that are used when requesting certificates with acme. the private keys are pre-generated because they must be static and known during setup, because their public keys must be published in tlsa records in dns. autocert would generate private keys on its own, so had to be forked to add the option to provide the private key when requesting a new certificate. hopefully upstream will accept the change and we can drop the fork. with this change, using the quickstart to setup a new mox instance, the checks at internet.nl result in a 100% score, provided the domain is dnssec-signed and the network doesn't have any issues.
2023-10-10 13:09:35 +03:00
// Conn returns the connection with initialized SMTP session. Once the caller uses
// this connection it is in control, and responsible for closing the connection,
// and other functions on the client must not be called anymore.
func (c *Client) Conn() (net.Conn, error) {
if err := c.conn.SetDeadline(time.Time{}); err != nil {
return nil, fmt.Errorf("clearing io deadlines: %w", err)
}
return c.conn, nil
}