new feature: when delivering messages from the queue, make it possible to use a "transport"

the default transport is still just "direct delivery", where we connect to the
destination domain's MX servers.

other transports are:

- regular smtp without authentication, this is relaying to a smarthost.
- submission with authentication, e.g. to a third party email sending service.
- direct delivery, but with with connections going through a socks proxy. this
  can be helpful if your ip is blocked, you need to get email out, and you have
  another IP that isn't blocked.

keep in mind that for all of the above, appropriate SPF/DKIM settings have to
be configured. the "dnscheck" for a domain does a check for any SOCKS IP in the
SPF record. SPF for smtp/submission (ranges? includes?) and any DKIM
requirements cannot really be checked.

which transport is used can be configured through routes. routes can be set on
an account, a domain, or globally. the routes are evaluated in that order, with
the first match selecting the transport. these routes are evaluated for each
delivery attempt. common selection criteria are recipient domain and sender
domain, but also which delivery attempt this is. you could configured mox to
attempt sending through a 3rd party from the 4th attempt onwards.

routes and transports are optional. if no route matches, or an empty/zero
transport is selected, normal direct delivery is done.

we could already "submit" emails with 3rd party accounts with "sendmail". but
we now support more SASL authentication mechanisms with SMTP (not only PLAIN,
but also SCRAM-SHA-256, SCRAM-SHA-1 and CRAM-MD5), which sendmail now also
supports. sendmail will use the most secure mechanism supported by the server,
or the explicitly configured mechanism.

for issue #36 by dmikushin. also based on earlier discussion on hackernews.

README.md

@@ -5,7 +5,7 @@ See Quickstart below to get started.
 ## Features
 
 - Quick and easy to start/maintain mail server, for your own domain(s).
-- SMTP (with extensions) for receiving and submitting email.
+- SMTP (with extensions) for receiving, submitting and delivering email.
 - IMAP4 (with extensions) for giving email clients access to email.
 - Automatic TLS with ACME, for use with Let's Encrypt and other CA's.
 - SPF, verifying that a remote host is allowed to sent email for a domain.

config/config.go

@@ -53,7 +53,8 @@ type Static struct {
 		Account string
 		Mailbox string `sconf-doc:"E.g. Postmaster or Inbox."`
 	} `sconf-doc:"Destination for emails delivered to postmaster addresses: a plain 'postmaster' without domain, 'postmaster@<hostname>' (also for each listener with SMTP enabled), and as fallback for each domain without explicitly configured postmaster destination."`
-	DefaultMailboxes []string `sconf:"optional" sconf-doc:"Mailboxes to create when adding an account. Inbox is always created. If no mailboxes are specified, the following are automatically created: Sent, Archive, Trash, Drafts and Junk."`
+	DefaultMailboxes []string             `sconf:"optional" sconf-doc:"Mailboxes to create when adding an account. Inbox is always created. If no mailboxes are specified, the following are automatically created: Sent, Archive, Trash, Drafts and Junk."`
+	Transports       map[string]Transport `sconf:"optional" sconf-doc:"Transport are mechanisms for delivering messages. Transports can be referenced from Routes in accounts, domains and the global configuration. There is always an implicit/fallback delivery transport doing direct delivery with SMTP from the outgoing message queue. Transports are typically only configured when using smarthosts, i.e. when delivering through another SMTP server. Zero or one transport methods must be set in a transport, never multiple. When using an external party to send email for a domain, keep in mind you may have to add their IP address to your domain's SPF record, and possibly additional DKIM records."`
 
 	// All IPs that were explicitly listen on for external SMTP. Only set when there
 	// are no unspecified external SMTP listeners and there is at most one for IPv4 and
@@ -75,6 +76,7 @@ type Dynamic struct {
 	Accounts           map[string]Account `sconf-doc:"Accounts to which email can be delivered. An account can accept email for multiple domains, for multiple localparts, and deliver to multiple mailboxes."`
 	WebDomainRedirects map[string]string  `sconf:"optional" sconf-doc:"Redirect all requests from domain (key) to domain (value). Always redirects to HTTPS. For plain HTTP redirects, use a WebHandler with a WebRedirect."`
 	WebHandlers        []WebHandler       `sconf:"optional" sconf-doc:"Handle webserver requests by serving static files, redirecting or reverse-proxying HTTP(s). The first matching WebHandler will handle the request. Built-in handlers, e.g. for account, admin, autoconfig and mta-sts always run first. If no handler matches, the response status code is file not found (404). If functionality you need is missng, simply forward the requests to an application that can provide the needed functionality."`
+	Routes             []Route            `sconf:"optional" sconf-doc:"Routes for delivering outgoing messages through the queue. Each delivery attempt evaluates account routes, domain routes and finally these global routes. The transport of the first matching route is used in the delivery attempt. If no routes match, which is the default with no configured routes, messages are delivered directly from the queue."`
 
 	WebDNSDomainRedirects map[dns.Domain]dns.Domain `sconf:"-"`
 }
@@ -170,6 +172,50 @@ type Listener struct {
 	} `sconf:"optional" sconf-doc:"All configured WebHandlers will serve on an enabled listener. Either ACME must be configured, or for each WebHandler domain a TLS certificate must be configured."`
 }
 
+// Transport is a method to delivery a message. At most one of the fields can
+// be non-nil. The non-nil field represents the type of transport. For a
+// transport with all fields nil, regular email delivery is done.
+type Transport struct {
+	Submissions *TransportSMTP  `sconf:"optional" sconf-doc:"Submission SMTP over a TLS connection to submit email to a remote queue."`
+	Submission  *TransportSMTP  `sconf:"optional" sconf-doc:"Submission SMTP over a plain TCP connection (possibly with STARTTLS) to submit email to a remote queue."`
+	SMTP        *TransportSMTP  `sconf:"optional" sconf-doc:"SMTP over a plain connection (possibly with STARTTLS), typically for old-fashioned unauthenticated relaying to a remote queue."`
+	Socks       *TransportSocks `sconf:"optional" sconf-doc:"Like regular direct delivery, but makes outgoing connections through a SOCKS proxy."`
+}
+
+// TransportSMTP delivers messages by "submission" (SMTP, typically
+// authenticated) to the queue of a remote host (smarthost), or by relaying
+// (SMTP, typically unauthenticated).
+type TransportSMTP struct {
+	Host                       string    `sconf-doc:"Host name to connect to and for verifying its TLS certificate."`
+	Port                       int       `sconf:"optional" sconf-doc:"If unset or 0, the default port for submission(s)/smtp is used: 25 for SMTP, 465 for submissions (with TLS), 587 for submission (possibly with STARTTLS)."`
+	STARTTLSInsecureSkipVerify bool      `sconf:"optional" sconf-doc:"If set an unverifiable remote TLS certificate during STARTTLS is accepted."`
+	NoSTARTTLS                 bool      `sconf:"optional" sconf-doc:"If set for submission or smtp transport, do not attempt STARTTLS on the connection. Authentication credentials and messages will be transferred in clear text."`
+	Auth                       *SMTPAuth `sconf:"optional" sconf-doc:"If set, authentication credentials for the remote server."`
+
+	DNSHost dns.Domain `sconf:"-" json:"-"`
+}
+
+// SMTPAuth hold authentication credentials used when delivering messages
+// through a smarthost.
+type SMTPAuth struct {
+	Username   string
+	Password   string
+	Mechanisms []string `sconf:"optional" sconf-doc:"Allowed authentication mechanisms. Defaults to SCRAM-SHA-256, SCRAM-SHA-1, CRAM-MD5. Not included by default: PLAIN."`
+
+	EffectiveMechanisms []string `sconf:"-" json:"-"`
+}
+
+type TransportSocks struct {
+	Address        string   `sconf-doc:"Address of SOCKS proxy, of the form host:port or ip:port."`
+	RemoteIPs      []string `sconf-doc:"IP addresses connections from the SOCKS server will originate from. This IP addresses should be configured in the SPF record (keep in mind DNS record time to live (TTL) when adding a SOCKS proxy). Reverse DNS should be set up for these address, resolving to RemoteHostname. These are typically the IPv4 and IPv6 address for the host in the Address field."`
+	RemoteHostname string   `sconf-doc:"Hostname belonging to RemoteIPs. This name is used during in SMTP EHLO. This is typically the hostname of the host in the Address field."`
+
+	// todo: add authentication credentials?
+
+	IPs      []net.IP   `sconf:"-" json:"-"` // Parsed form of RemoteIPs.
+	Hostname dns.Domain `sconf:"-" json:"-"` // Parsed form of RemoteHostname
+}
+
 type Domain struct {
 	Description                string  `sconf:"optional" sconf-doc:"Free-form description of domain."`
 	LocalpartCatchallSeparator string  `sconf:"optional" sconf-doc:"If not empty, only the string before the separator is used to for email delivery decisions. For example, if set to \"+\", you+anything@example.com will be delivered to you@example.com."`
@@ -178,6 +224,7 @@ type Domain struct {
 	DMARC                      *DMARC  `sconf:"optional" sconf-doc:"With DMARC, a domain publishes, in DNS, a policy on how other mail servers should handle incoming messages with the From-header matching this domain and/or subdomain (depending on the configured alignment). Receiving mail servers use this to build up a reputation of this domain, which can help with mail delivery. A domain can also publish an email address to which reports about DMARC verification results can be sent by verifying mail servers, useful for monitoring. Incoming DMARC reports are automatically parsed, validated, added to metrics and stored in the reporting database for later display in the admin web pages."`
 	MTASTS                     *MTASTS `sconf:"optional" sconf-doc:"With MTA-STS a domain publishes, in DNS, presence of a policy for using/requiring TLS for SMTP connections. The policy is served over HTTPS."`
 	TLSRPT                     *TLSRPT `sconf:"optional" sconf-doc:"With TLSRPT a domain specifies in DNS where reports about encountered SMTP TLS behaviour should be sent. Useful for monitoring. Incoming TLS reports are automatically parsed, validated, added to metrics and stored in the reporting database for later display in the admin web pages."`
+	Routes                     []Route `sconf:"optional" sconf-doc:"Routes for delivering outgoing messages through the queue. Each delivery attempt evaluates account routes, these domain routes and finally global routes. The transport of the first matching route is used in the delivery attempt. If no routes match, which is the default with no configured routes, messages are delivered directly from the queue."`
 
 	Domain dns.Domain `sconf:"-" json:"-"`
 }
@@ -229,6 +276,19 @@ type DKIM struct {
 	Sign      []string            `sconf:"optional" sconf-doc:"List of selectors that emails will be signed with."`
 }
 
+type Route struct {
+	FromDomain      []string `sconf:"optional" sconf-doc:"Matches if the envelope from domain matches one of the configured domains, or if the list is empty. If a domain starts with a dot, prefixes of the domain also match."`
+	ToDomain        []string `sconf:"optional" sconf-doc:"Like FromDomain, but matching against the envelope to domain."`
+	MinimumAttempts int      `sconf:"optional" sconf-doc:"Matches if at least this many deliveries have already been attempted. This can be used to attempt sending through a smarthost when direct delivery has failed for several times."`
+	Transport       string   `sconf:"The transport used for delivering the message that matches requirements of the above fields."`
+
+	// todo future: add ToMX, where we look up the MX record of the destination domain and check (the first, any, all?) mx host against the values in ToMX.
+
+	FromDomainASCII   []string  `sconf:"-"`
+	ToDomainASCII     []string  `sconf:"-"`
+	ResolvedTransport Transport `sconf:"-" json:"-"`
+}
+
 type Account struct {
 	Domain       string                 `sconf-doc:"Default domain for account. Deprecated behaviour: If a destination is not a full address but only a localpart, this domain is added to form a full address."`
 	Description  string                 `sconf:"optional" sconf-doc:"Free form description, e.g. full name or alternative contact info."`
@@ -246,6 +306,7 @@ type Account struct {
 	JunkFilter                   *JunkFilter `sconf:"optional" sconf-doc:"Content-based filtering, using the junk-status of individual messages to rank words in such messages as spam or ham. It is recommended you always set the applicable (non)-junk status on messages, and that you do not empty your Trash because those messages contain valuable ham/spam training information."` // todo: sane defaults for junkfilter
 	MaxOutgoingMessagesPerDay    int         `sconf:"optional" sconf-doc:"Maximum number of outgoing messages for this account in a 24 hour window. This limits the damage to recipients and the reputation of this mail server in case of account compromise. Default 1000."`
 	MaxFirstTimeRecipientsPerDay int         `sconf:"optional" sconf-doc:"Maximum number of first-time recipients in outgoing messages for this account in a 24 hour window. This limits the damage to recipients and the reputation of this mail server in case of account compromise. Default 200."`
+	Routes                       []Route     `sconf:"optional" sconf-doc:"Routes for delivering outgoing messages through the queue. Each delivery attempt evaluates these account routes, domain routes and finally global routes. The transport of the first matching route is used in the delivery attempt. If no routes match, which is the default with no configured routes, messages are delivered directly from the queue."`
 
 	DNSDomain      dns.Domain     `sconf:"-"` // Parsed form of Domain.
 	JunkMailbox    *regexp.Regexp `sconf:"-" json:"-"`

config/doc.go

@@ -339,6 +339,130 @@ describe-static" and "mox config describe-domains":
 	DefaultMailboxes:
 		-
 
+	# Transport are mechanisms for delivering messages. Transports can be referenced
+	# from Routes in accounts, domains and the global configuration. There is always
+	# an implicit/fallback delivery transport doing direct delivery with SMTP from the
+	# outgoing message queue. Transports are typically only configured when using
+	# smarthosts, i.e. when delivering through another SMTP server. Zero or one
+	# transport methods must be set in a transport, never multiple. When using an
+	# external party to send email for a domain, keep in mind you may have to add
+	# their IP address to your domain's SPF record, and possibly additional DKIM
+	# records. (optional)
+	Transports:
+		x:
+
+			# Submission SMTP over a TLS connection to submit email to a remote queue.
+			# (optional)
+			Submissions:
+
+				# Host name to connect to and for verifying its TLS certificate.
+				Host:
+
+				# If unset or 0, the default port for submission(s)/smtp is used: 25 for SMTP, 465
+				# for submissions (with TLS), 587 for submission (possibly with STARTTLS).
+				# (optional)
+				Port: 0
+
+				# If set an unverifiable remote TLS certificate during STARTTLS is accepted.
+				# (optional)
+				STARTTLSInsecureSkipVerify: false
+
+				# If set for submission or smtp transport, do not attempt STARTTLS on the
+				# connection. Authentication credentials and messages will be transferred in clear
+				# text. (optional)
+				NoSTARTTLS: false
+
+				# If set, authentication credentials for the remote server. (optional)
+				Auth:
+					Username:
+					Password:
+
+					# Allowed authentication mechanisms. Defaults to SCRAM-SHA-256, SCRAM-SHA-1,
+					# CRAM-MD5. Not included by default: PLAIN. (optional)
+					Mechanisms:
+						-
+
+			# Submission SMTP over a plain TCP connection (possibly with STARTTLS) to submit
+			# email to a remote queue. (optional)
+			Submission:
+
+				# Host name to connect to and for verifying its TLS certificate.
+				Host:
+
+				# If unset or 0, the default port for submission(s)/smtp is used: 25 for SMTP, 465
+				# for submissions (with TLS), 587 for submission (possibly with STARTTLS).
+				# (optional)
+				Port: 0
+
+				# If set an unverifiable remote TLS certificate during STARTTLS is accepted.
+				# (optional)
+				STARTTLSInsecureSkipVerify: false
+
+				# If set for submission or smtp transport, do not attempt STARTTLS on the
+				# connection. Authentication credentials and messages will be transferred in clear
+				# text. (optional)
+				NoSTARTTLS: false
+
+				# If set, authentication credentials for the remote server. (optional)
+				Auth:
+					Username:
+					Password:
+
+					# Allowed authentication mechanisms. Defaults to SCRAM-SHA-256, SCRAM-SHA-1,
+					# CRAM-MD5. Not included by default: PLAIN. (optional)
+					Mechanisms:
+						-
+
+			# SMTP over a plain connection (possibly with STARTTLS), typically for
+			# old-fashioned unauthenticated relaying to a remote queue. (optional)
+			SMTP:
+
+				# Host name to connect to and for verifying its TLS certificate.
+				Host:
+
+				# If unset or 0, the default port for submission(s)/smtp is used: 25 for SMTP, 465
+				# for submissions (with TLS), 587 for submission (possibly with STARTTLS).
+				# (optional)
+				Port: 0
+
+				# If set an unverifiable remote TLS certificate during STARTTLS is accepted.
+				# (optional)
+				STARTTLSInsecureSkipVerify: false
+
+				# If set for submission or smtp transport, do not attempt STARTTLS on the
+				# connection. Authentication credentials and messages will be transferred in clear
+				# text. (optional)
+				NoSTARTTLS: false
+
+				# If set, authentication credentials for the remote server. (optional)
+				Auth:
+					Username:
+					Password:
+
+					# Allowed authentication mechanisms. Defaults to SCRAM-SHA-256, SCRAM-SHA-1,
+					# CRAM-MD5. Not included by default: PLAIN. (optional)
+					Mechanisms:
+						-
+
+			# Like regular direct delivery, but makes outgoing connections through a SOCKS
+			# proxy. (optional)
+			Socks:
+
+				# Address of SOCKS proxy, of the form host:port or ip:port.
+				Address:
+
+				# IP addresses connections from the SOCKS server will originate from. This IP
+				# addresses should be configured in the SPF record (keep in mind DNS record time
+				# to live (TTL) when adding a SOCKS proxy). Reverse DNS should be set up for these
+				# address, resolving to RemoteHostname. These are typically the IPv4 and IPv6
+				# address for the host in the Address field.
+				RemoteIPs:
+					-
+
+				# Hostname belonging to RemoteIPs. This name is used during in SMTP EHLO. This is
+				# typically the hostname of the host in the Address field.
+				RemoteHostname:
+
 # domains.conf
 
 	# Domains for which email is accepted. For internationalized domains, use their
@@ -461,6 +585,30 @@ describe-static" and "mox config describe-domains":
 				# Mailbox to deliver to, e.g. TLSRPT.
 				Mailbox:
 
+			# Routes for delivering outgoing messages through the queue. Each delivery attempt
+			# evaluates account routes, these domain routes and finally global routes. The
+			# transport of the first matching route is used in the delivery attempt. If no
+			# routes match, which is the default with no configured routes, messages are
+			# delivered directly from the queue. (optional)
+			Routes:
+				-
+
+					# Matches if the envelope from domain matches one of the configured domains, or if
+					# the list is empty. If a domain starts with a dot, prefixes of the domain also
+					# match. (optional)
+					FromDomain:
+						-
+
+					# Like FromDomain, but matching against the envelope to domain. (optional)
+					ToDomain:
+						-
+
+					# Matches if at least this many deliveries have already been attempted. This can
+					# be used to attempt sending through a smarthost when direct delivery has failed
+					# for several times. (optional)
+					MinimumAttempts: 0
+					Transport:
+
 	# Accounts to which email can be delivered. An account can accept email for
 	# multiple domains, for multiple localparts, and deliver to multiple mailboxes.
 	Accounts:
@@ -611,6 +759,30 @@ describe-static" and "mox config describe-domains":
 			# this mail server in case of account compromise. Default 200. (optional)
 			MaxFirstTimeRecipientsPerDay: 0
 
+			# Routes for delivering outgoing messages through the queue. Each delivery attempt
+			# evaluates these account routes, domain routes and finally global routes. The
+			# transport of the first matching route is used in the delivery attempt. If no
+			# routes match, which is the default with no configured routes, messages are
+			# delivered directly from the queue. (optional)
+			Routes:
+				-
+
+					# Matches if the envelope from domain matches one of the configured domains, or if
+					# the list is empty. If a domain starts with a dot, prefixes of the domain also
+					# match. (optional)
+					FromDomain:
+						-
+
+					# Like FromDomain, but matching against the envelope to domain. (optional)
+					ToDomain:
+						-
+
+					# Matches if at least this many deliveries have already been attempted. This can
+					# be used to attempt sending through a smarthost when direct delivery has failed
+					# for several times. (optional)
+					MinimumAttempts: 0
+					Transport:
+
 	# Redirect all requests from domain (key) to domain (value). Always redirects to
 	# HTTPS. For plain HTTP redirects, use a WebHandler with a WebRedirect. (optional)
 	WebDomainRedirects:
@@ -729,6 +901,30 @@ describe-static" and "mox config describe-domains":
 				ResponseHeaders:
 					x:
 
+	# Routes for delivering outgoing messages through the queue. Each delivery attempt
+	# evaluates account routes, domain routes and finally these global routes. The
+	# transport of the first matching route is used in the delivery attempt. If no
+	# routes match, which is the default with no configured routes, messages are
+	# delivered directly from the queue. (optional)
+	Routes:
+		-
+
+			# Matches if the envelope from domain matches one of the configured domains, or if
+			# the list is empty. If a domain starts with a dot, prefixes of the domain also
+			# match. (optional)
+			FromDomain:
+				-
+
+			# Like FromDomain, but matching against the envelope to domain. (optional)
+			ToDomain:
+				-
+
+			# Matches if at least this many deliveries have already been attempted. This can
+			# be used to attempt sending through a smarthost when direct delivery has failed
+			# for several times. (optional)
+			MinimumAttempts: 0
+			Transport:
+
 # Examples
 
 Mox includes configuration files to illustrate common setups. You can see these

ctl.go

@@ -404,9 +404,39 @@ func servectlcmd(ctx context.Context, log *mlog.Log, ctl *ctl, shutdown func())
 		}
 		xw.xclose()
 
-	case "queuekick", "queuedrop":
+	case "queuekick":
 		/* protocol:
-		> "queuekick" or "queuedrop"
+		> "queuekick"
+		> id
+		> todomain
+		> recipient
+		> transport // if empty, transport is left unchanged; in future, we may want to differtiate between "leave unchanged" and "set to empty string".
+		< count
+		< "ok" or error
+		*/
+
+		idstr := ctl.xread()
+		todomain := ctl.xread()
+		recipient := ctl.xread()
+		transport := ctl.xread()
+		id, err := strconv.ParseInt(idstr, 10, 64)
+		if err != nil {
+			ctl.xwrite("0")
+			ctl.xcheck(err, "parsing id")
+		}
+
+		var xtransport *string
+		if transport != "" {
+			xtransport = &transport
+		}
+		count, err := queue.Kick(ctx, id, todomain, recipient, xtransport)
+		ctl.xcheck(err, "kicking queue")
+		ctl.xwrite(fmt.Sprintf("%d", count))
+		ctl.xwriteok()
+
+	case "queuedrop":
+		/* protocol:
+		> "queuedrop"
 		> id
 		> todomain
 		> recipient
@@ -423,14 +453,8 @@ func servectlcmd(ctx context.Context, log *mlog.Log, ctl *ctl, shutdown func())
 			ctl.xcheck(err, "parsing id")
 		}
 
-		var count int
-		if cmd == "queuekick" {
-			count, err = queue.Kick(ctx, id, todomain, recipient)
-			ctl.xcheck(err, "kicking queue")
-		} else {
-			count, err = queue.Drop(ctx, id, todomain, recipient)
-			ctl.xcheck(err, "dropping messages from queue")
-		}
+		count, err := queue.Drop(ctx, id, todomain, recipient)
+		ctl.xcheck(err, "dropping messages from queue")
 		ctl.xwrite(fmt.Sprintf("%d", count))
 		ctl.xwriteok()
 

doc.go

@@ -20,7 +20,7 @@ low-maintenance self-hosted email.
 	mox setadminpassword
 	mox loglevels [level [pkg]]
 	mox queue list
-	mox queue kick [-id id] [-todomain domain] [-recipient address]
+	mox queue kick [-id id] [-todomain domain] [-recipient address] [-transport transport]
 	mox queue drop [-id id] [-todomain domain] [-recipient address]
 	mox queue dump id
 	mox import maildir accountname mailboxname maildir
@@ -192,13 +192,19 @@ retry after 7.5 minutes, and doubling each time. Kicking messages sets their
 next scheduled attempt to now, it can cause delivery to fail earlier than
 without rescheduling.
 
-	usage: mox queue kick [-id id] [-todomain domain] [-recipient address]
+With the -transport flag, future delivery attempts are done using the specified
+transport. Transports can be configured in mox.conf, e.g. to submit to a remote
+queue over SMTP.
+
+	usage: mox queue kick [-id id] [-todomain domain] [-recipient address] [-transport transport]
 	  -id int
 	    	id of message in queue
 	  -recipient string
 	    	recipient email address
 	  -todomain string
 	    	destination domain of messages
+	  -transport string
+	    	transport to use for the next delivery
 
 # mox queue drop
 

gentestdata.go

@@ -233,7 +233,7 @@ Accounts:
 	const qmsg = "From: <test0@mox.example>\r\nTo: <other@remote.example>\r\nSubject: test\r\n\r\nthe message...\r\n"
 	_, err = fmt.Fprint(mf, qmsg)
 	xcheckf(err, "writing message")
-	err = queue.Add(ctxbg, mlog.New("gentestdata"), "test0", mailfrom, rcptto, false, false, int64(len(qmsg)), prefix, mf, nil, true)
+	_, err = queue.Add(ctxbg, mlog.New("gentestdata"), "test0", mailfrom, rcptto, false, false, int64(len(qmsg)), prefix, mf, nil, true)
 	xcheckf(err, "enqueue message")
 
 	// Create three accounts.

http/admin.go

@@ -339,8 +339,15 @@ func logPanic(ctx context.Context) {
 }
 
 // return IPs we may be listening on.
-func xlistenIPs(ctx context.Context) []net.IP {
-	ips, err := mox.IPs(ctx)
+func xlistenIPs(ctx context.Context, receiveOnly bool) []net.IP {
+	ips, err := mox.IPs(ctx, receiveOnly)
+	xcheckf(ctx, err, "listing ips")
+	return ips
+}
+
+// return IPs from which we may be sending.
+func xsendingIPs(ctx context.Context) []net.IP {
+	ips, err := mox.IPs(ctx, false)
 	xcheckf(ctx, err, "listing ips")
 	return ips
 }
@@ -366,7 +373,7 @@ func checkDomain(ctx context.Context, resolver dns.Resolver, dialer *net.Dialer,
 		panic(&sherpa.Error{Code: "user:notFound", Message: "domain not found"})
 	}
 
-	listenIPs := xlistenIPs(ctx)
+	listenIPs := xlistenIPs(ctx, true)
 	isListenIP := func(ip net.IP) bool {
 		for _, lip := range listenIPs {
 			if ip.Equal(lip) {
@@ -442,7 +449,7 @@ func checkDomain(ctx context.Context, resolver dns.Resolver, dialer *net.Dialer,
 
 		// For each mox.Conf.SpecifiedSMTPListenIPs, and each address for
 		// mox.Conf.HostnameDomain, check if they resolve back to the host name.
-		var ips []net.IP
+		hostIPs := map[dns.Domain][]net.IP{}
 		ips, err := resolver.LookupIP(ctx, "ip", mox.Conf.Static.HostnameDomain.ASCII+".")
 		if err != nil {
 			addf(&r.IPRev.Errors, "Looking up IPs for hostname: %s", err)
@@ -458,32 +465,59 @@ func checkDomain(ctx context.Context, resolver dns.Resolver, dialer *net.Dialer,
 				ips = append(ips, ip)
 			}
 		}
+		hostIPs[mox.Conf.Static.HostnameDomain] = ips
+
+		iplist := func(ips []net.IP) string {
+			var ipstrs []string
+			for _, ip := range ips {
+				ipstrs = append(ipstrs, ip.String())
+			}
+			return strings.Join(ipstrs, ", ")
+		}
+
+		r.IPRev.Hostname = mox.Conf.Static.HostnameDomain
+		r.IPRev.Instructions = []string{
+			fmt.Sprintf("Ensure IPs %s have reverse address %s.", iplist(ips), mox.Conf.Static.HostnameDomain.ASCII),
+		}
+
+		// If we have a socks transport, also check its host and IP.
+		for tname, t := range mox.Conf.Static.Transports {
+			if t.Socks != nil {
+				hostIPs[t.Socks.Hostname] = append(hostIPs[t.Socks.Hostname], t.Socks.IPs...)
+				instr := fmt.Sprintf("For SOCKS transport %s, ensure IPs %s have reverse address %s.", tname, iplist(t.Socks.IPs), t.Socks.Hostname)
+				r.IPRev.Instructions = append(r.IPRev.Instructions, instr)
+			}
+		}
 
 		type result struct {
+			Host  dns.Domain
 			IP    string
 			Addrs []string
 			Err   error
 		}
 		results := make(chan result)
-		var ipstrs []string
-		for _, ip := range ips {
-			s := ip.String()
-			go func() {
-				addrs, err := resolver.LookupAddr(ctx, s)
-				results <- result{s, addrs, err}
-			}()
-			ipstrs = append(ipstrs, s)
+		n := 0
+		for host, ips := range hostIPs {
+			for _, ip := range ips {
+				n++
+				s := ip.String()
+				host := host
+				go func() {
+					addrs, err := resolver.LookupAddr(ctx, s)
+					results <- result{host, s, addrs, err}
+				}()
+			}
 		}
 		r.IPRev.IPNames = map[string][]string{}
-		for i := 0; i < len(ips); i++ {
+		for i := 0; i < n; i++ {
 			lr := <-results
-			addrs, ip, err := lr.Addrs, lr.IP, lr.Err
+			host, addrs, ip, err := lr.Host, lr.Addrs, lr.IP, lr.Err
 			if err != nil {
-				addf(&r.IPRev.Errors, "Looking up reverse name for %s: %v", ip, err)
+				addf(&r.IPRev.Errors, "Looking up reverse name for %s of %s: %v", ip, host, err)
 				continue
 			}
 			if len(addrs) != 1 {
-				addf(&r.IPRev.Errors, "Expected exactly 1 name for %s, got %d (%v)", ip, len(addrs), addrs)
+				addf(&r.IPRev.Errors, "Expected exactly 1 name for %s of %s, got %d (%v)", ip, host, len(addrs), addrs)
 			}
 			var match bool
 			for i, a := range addrs {
@@ -493,20 +527,15 @@ func checkDomain(ctx context.Context, resolver dns.Resolver, dialer *net.Dialer,
 				if err != nil {
 					addf(&r.IPRev.Errors, "Parsing reverse name %q for %s: %v", a, ip, err)
 				}
-				if ad == mox.Conf.Static.HostnameDomain {
+				if ad == host {
 					match = true
 				}
 			}
 			if !match {
-				addf(&r.IPRev.Errors, "Reverse name(s) %s for ip %s do not match hostname %s, which will cause other mail servers to reject incoming messages from this IP.", strings.Join(addrs, ","), ip, mox.Conf.Static.HostnameDomain)
+				addf(&r.IPRev.Errors, "Reverse name(s) %s for ip %s do not match hostname %s, which will cause other mail servers to reject incoming messages from this IP.", strings.Join(addrs, ","), ip, host)
 			}
 			r.IPRev.IPNames[ip] = addrs
 		}
-
-		r.IPRev.Hostname = mox.Conf.Static.HostnameDomain
-		r.IPRev.Instructions = []string{
-			fmt.Sprintf("Ensure IPs %s have reverse address %s.", strings.Join(ipstrs, ", "), mox.Conf.Static.HostnameDomain.ASCII),
-		}
 	}()
 
 	// MX
@@ -648,6 +677,7 @@ func checkDomain(ctx context.Context, resolver dns.Resolver, dialer *net.Dialer,
 	}()
 
 	// SPF
+	// todo: add warnings if we have Transports with submission? admin should ensure their IPs are in the SPF record. it may be an IP(net), or an include. that means we cannot easily check for it. and should we first check the transport can be used from this domain (or an account that has this domain?). also see DKIM.
 	wg.Add(1)
 	go func() {
 		defer logPanic(ctx)
@@ -667,38 +697,51 @@ func checkDomain(ctx context.Context, resolver dns.Resolver, dialer *net.Dialer,
 			spfr := spf.Record{
 				Version: "spf1",
 			}
+
+			checkSPFIP := func(ip net.IP) {
+				mechanism := "ip4"
+				if ip.To4() == nil {
+					mechanism = "ip6"
+				}
+				spfr.Directives = append(spfr.Directives, spf.Directive{Mechanism: mechanism, IP: ip})
+
+				if record == nil {
+					return
+				}
+
+				args := spf.Args{
+					RemoteIP:          ip,
+					MailFromLocalpart: "postmaster",
+					MailFromDomain:    domain,
+					HelloDomain:       dns.IPDomain{Domain: domain},
+					LocalIP:           net.ParseIP("127.0.0.1"),
+					LocalHostname:     dns.Domain{ASCII: "localhost"},
+				}
+				status, mechanism, expl, err := spf.Evaluate(ctx, record, resolver, args)
+				if err != nil {
+					addf(&r.SPF.Errors, "Evaluating IP %q against %s SPF record: %s", ip, kind, err)
+				} else if status != spf.StatusPass {
+					addf(&r.SPF.Errors, "IP %q does not pass %s SPF evaluation, status not \"pass\" but %q (mechanism %q, explanation %q)", ip, kind, status, mechanism, expl)
+				}
+			}
+
 			for _, l := range mox.Conf.Static.Listeners {
 				if !l.SMTP.Enabled || l.IPsNATed {
 					continue
 				}
 				for _, ipstr := range l.IPs {
 					ip := net.ParseIP(ipstr)
-					mechanism := "ip4"
-					if ip.To4() == nil {
-						mechanism = "ip6"
-					}
-					spfr.Directives = append(spfr.Directives, spf.Directive{Mechanism: mechanism, IP: ip})
-
-					if record == nil {
-						continue
-					}
-
-					args := spf.Args{
-						RemoteIP:          ip,
-						MailFromLocalpart: "postmaster",
-						MailFromDomain:    domain,
-						HelloDomain:       dns.IPDomain{Domain: domain},
-						LocalIP:           net.ParseIP("127.0.0.1"),
-						LocalHostname:     dns.Domain{ASCII: "localhost"},
-					}
-					status, mechanism, expl, err := spf.Evaluate(ctx, record, resolver, args)
-					if err != nil {
-						addf(&r.SPF.Errors, "Evaluating IP %q against %s SPF record: %s", ip, kind, err)
-					} else if status != spf.StatusPass {
-						addf(&r.SPF.Errors, "IP %q does not pass %s SPF evaluation, status not \"pass\" but %q (mechanism %q, explanation %q)", ip, kind, status, mechanism, expl)
+					checkSPFIP(ip)
+				}
+			}
+			for _, t := range mox.Conf.Static.Transports {
+				if t.Socks != nil {
+					for _, ip := range t.Socks.IPs {
+						checkSPFIP(ip)
 					}
 				}
 			}
+
 			spfr.Directives = append(spfr.Directives, spf.Directive{Qualifier: "-", Mechanism: "all"})
 			return txt, xrecord, spfr
 		}
@@ -722,6 +765,7 @@ func checkDomain(ctx context.Context, resolver dns.Resolver, dialer *net.Dialer,
 	}()
 
 	// DKIM
+	// todo: add warnings if we have Transports with submission? admin should ensure DKIM records exist. we cannot easily check if they actually exist though. and should we first check the transport can be used from this domain (or an account that has this domain?). also see SPF.
 	wg.Add(1)
 	go func() {
 		defer logPanic(ctx)
@@ -1392,7 +1436,7 @@ func dnsblsStatus(ctx context.Context, resolver dns.Resolver) map[string]map[str
 	}
 
 	r := map[string]map[string]string{}
-	for _, ip := range xlistenIPs(ctx) {
+	for _, ip := range xsendingIPs(ctx) {
 		if ip.IsLoopback() || ip.IsPrivate() {
 			continue
 		}
@@ -1517,9 +1561,10 @@ func (Admin) QueueSize(ctx context.Context) int {
 	return n
 }
 
-// QueueKick initiates delivery of a message from the queue.
-func (Admin) QueueKick(ctx context.Context, id int64) {
-	n, err := queue.Kick(ctx, id, "", "")
+// QueueKick initiates delivery of a message from the queue and sets the transport
+// to use for delivery.
+func (Admin) QueueKick(ctx context.Context, id int64, transport string) {
+	n, err := queue.Kick(ctx, id, "", "", &transport)
 	if err == nil && n == 0 {
 		err = errors.New("message not found")
 	}
@@ -1632,3 +1677,8 @@ func (Admin) WebserverConfigSave(ctx context.Context, oldConf, newConf Webserver
 	savedConf.WebDomainRedirects = nil
 	return savedConf
 }
+
+// Transports returns the configured transports, for sending email.
+func (Admin) Transports(ctx context.Context) map[string]config.Transport {
+	return mox.Conf.Static.Transports
+}

http/admin.html

@@ -195,9 +195,9 @@ const formatSize = n => {
 
 const index = async () => {
 	const [domains, queueSize, checkUpdatesEnabled] = await Promise.all([
-		await api.Domains(),
-		await api.QueueSize(),
-		await api.CheckUpdatesEnabled(),
+		api.Domains(),
+		api.QueueSize(),
+		api.CheckUpdatesEnabled(),
 	])
 
 	let fieldset, domain, account, localpart
@@ -265,6 +265,7 @@ const index = async () => {
 		dom.div(dom.a('MTA-STS policies', attr({href: '#mtasts'}))),
 		// todo: outgoing DMARC findings
 		// todo: outgoing TLSRPT findings
+		// todo: routing, globally, per domain and per account
 		dom.br(),
 		dom.h2('DNS blocklist status'),
 		dom.div(dom.a('DNSBL status', attr({href: '#dnsbl'}))),
@@ -1525,9 +1526,13 @@ const dnsbl = async () => {
 }
 
 const queueList = async () => {
-	const msgs = await api.QueueList()
+	const [msgs, transports] = await Promise.all([
+		api.QueueList(),
+		api.Transports(),
+	])
 
 	const nowSecs = new Date().getTime()/1000
+	let transport
 
 	const page = document.getElementById('page')
 	dom._kids(page,
@@ -1550,7 +1555,8 @@ const queueList = async () => {
 						dom.th('Next attempt'),
 						dom.th('Last attempt'),
 						dom.th('Last error'),
-						dom.th('Action'),
+						dom.th('Transport/Retry'),
+						dom.th('Remove'),
 					),
 				),
 				dom.tbody(
@@ -1565,11 +1571,17 @@ const queueList = async () => {
 						dom.td(m.LastAttempt ? age(new Date(m.LastAttempt), false, nowSecs) : '-'),
 						dom.td(m.LastError || '-'),
 						dom.td(
-							dom.button('Try now', async function click(e) {
+							transport=dom.select(
+								attr({title: 'Transport to use for delivery attempts. The default is direct delivery, connecting to the MX hosts of the domain.'}),
+								dom.option('(default)', attr({value: ''})),
+								Object.keys(transports).sort().map(t => dom.option(t, m.Transport === t ? attr({checked: ''}) : [])),
+							),
+							' ',
+							dom.button('Retry now', async function click(e) {
 								e.preventDefault()
 								try {
 									e.target.disabled = true
-									await api.QueueKick(m.ID)
+									await api.QueueKick(m.ID, transport.value)
 								} catch (err) {
 									console.log({err})
 									window.alert('Error: ' + err.message)
@@ -1579,7 +1591,8 @@ const queueList = async () => {
 								}
 								window.location.reload() // todo: only refresh the list
 							}),
-							' ',
+						),
+						dom.td(
 							dom.button('Remove', async function click(e) {
 								e.preventDefault()
 								if (!window.confirm('Are you sure you want to remove this message? It will be removed completely.')) {

http/adminapi.json

@@ -592,13 +592,19 @@
 		},
 		{
 			"Name": "QueueKick",
-			"Docs": "QueueKick initiates delivery of a message from the queue.",
+			"Docs": "QueueKick initiates delivery of a message from the queue and sets the transport\nto use for delivery.",
 			"Params": [
 				{
 					"Name": "id",
 					"Typewords": [
 						"int64"
 					]
+				},
+				{
+					"Name": "transport",
+					"Typewords": [
+						"string"
+					]
 				}
 			],
 			"Returns": []
@@ -713,6 +719,20 @@
 					]
 				}
 			]
+		},
+		{
+			"Name": "Transports",
+			"Docs": "Transports returns the configured transports, for sending email.",
+			"Params": [],
+			"Returns": [
+				{
+					"Name": "r0",
+					"Typewords": [
+						"{}",
+						"Transport"
+					]
+				}
+			]
 		}
 	],
 	"Sections": [],
@@ -2826,7 +2846,7 @@
 				},
 				{
 					"Name": "SenderAccount",
-					"Docs": "Failures are delivered back to this local account.",
+					"Docs": "Failures are delivered back to this local account. Also used for routing.",
 					"Typewords": [
 						"string"
 					]
@@ -2940,6 +2960,13 @@
 						"[]",
 						"uint8"
 					]
+				},
+				{
+					"Name": "Transport",
+					"Docs": "If non-empty, the transport to use for this message. Can be set through cli or admin interface. If empty (the default for a submitted message), regular routing rules apply.",
+					"Typewords": [
+						"string"
+					]
 				}
 			]
 		},
@@ -3170,6 +3197,142 @@
 					]
 				}
 			]
+		},
+		{
+			"Name": "Transport",
+			"Docs": "Transport is a method to delivery a message. At most one of the fields can\nbe non-nil. The non-nil field represents the type of transport. For a\ntransport with all fields nil, regular email delivery is done.",
+			"Fields": [
+				{
+					"Name": "Submissions",
+					"Docs": "",
+					"Typewords": [
+						"nullable",
+						"TransportSMTP"
+					]
+				},
+				{
+					"Name": "Submission",
+					"Docs": "",
+					"Typewords": [
+						"nullable",
+						"TransportSMTP"
+					]
+				},
+				{
+					"Name": "SMTP",
+					"Docs": "",
+					"Typewords": [
+						"nullable",
+						"TransportSMTP"
+					]
+				},
+				{
+					"Name": "Socks",
+					"Docs": "",
+					"Typewords": [
+						"nullable",
+						"TransportSocks"
+					]
+				}
+			]
+		},
+		{
+			"Name": "TransportSMTP",
+			"Docs": "TransportSMTP delivers messages by \"submission\" (SMTP, typically\nauthenticated) to the queue of a remote host (smarthost), or by relaying\n(SMTP, typically unauthenticated).",
+			"Fields": [
+				{
+					"Name": "Host",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "Port",
+					"Docs": "",
+					"Typewords": [
+						"int32"
+					]
+				},
+				{
+					"Name": "STARTTLSInsecureSkipVerify",
+					"Docs": "",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "NoSTARTTLS",
+					"Docs": "",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "Auth",
+					"Docs": "",
+					"Typewords": [
+						"nullable",
+						"SMTPAuth"
+					]
+				}
+			]
+		},
+		{
+			"Name": "SMTPAuth",
+			"Docs": "SMTPAuth hold authentication credentials used when delivering messages\nthrough a smarthost.",
+			"Fields": [
+				{
+					"Name": "Username",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "Password",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "Mechanisms",
+					"Docs": "",
+					"Typewords": [
+						"[]",
+						"string"
+					]
+				}
+			]
+		},
+		{
+			"Name": "TransportSocks",
+			"Docs": "",
+			"Fields": [
+				{
+					"Name": "Address",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "RemoteIPs",
+					"Docs": "",
+					"Typewords": [
+						"[]",
+						"string"
+					]
+				},
+				{
+					"Name": "RemoteHostname",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				}
+			]
 		}
 	],
 	"Ints": [],

integration_test.go

@@ -5,9 +5,7 @@
 package main
 
 import (
-	"bytes"
 	"context"
-	"encoding/base64"
 	"errors"
 	"fmt"
 	"log"
@@ -22,8 +20,10 @@ import (
 
 	"github.com/mjl-/bstore"
 
+	"github.com/mjl-/mox/dns"
 	"github.com/mjl-/mox/mlog"
 	"github.com/mjl-/mox/mox-"
+	"github.com/mjl-/mox/sasl"
 	"github.com/mjl-/mox/smtpclient"
 	"github.com/mjl-/mox/store"
 )
@@ -133,8 +133,6 @@ func TestDeliver(t *testing.T) {
 		tcheck(t, err, "dial submission")
 		defer conn.Close()
 
-		// todo: this is "aware" (hopefully) of the config smtpclient/client.go sets up... tricky
-		mox.Conf.Static.HostnameDomain.ASCII = desthost
 		msg := fmt.Sprintf(`From: <%s>
 To: <%s>
 Subject: test message
@@ -142,9 +140,8 @@ Subject: test message
 This is the message.
 `, mailfrom, rcptto)
 		msg = strings.ReplaceAll(msg, "\n", "\r\n")
-		auth := bytes.Join([][]byte{nil, []byte(mailfrom), []byte(password)}, []byte{0})
-		authLine := fmt.Sprintf("AUTH PLAIN %s", base64.StdEncoding.EncodeToString(auth))
-		c, err := smtpclient.New(mox.Context, mlog.New("test"), conn, smtpclient.TLSOpportunistic, desthost, authLine)
+		auth := []sasl.Client{sasl.NewClientPlain(mailfrom, password)}
+		c, err := smtpclient.New(mox.Context, mlog.New("test"), conn, smtpclient.TLSOpportunistic, mox.Conf.Static.HostnameDomain, dns.Domain{ASCII: desthost}, auth)
 		tcheck(t, err, "smtp hello")
 		err = c.Deliver(mox.Context, mailfrom, rcptto, int64(len(msg)), strings.NewReader(msg), false, false)
 		tcheck(t, err, "deliver with smtp")

main.go

@@ -1140,19 +1140,24 @@ error.
 }
 
 func cmdQueueKick(c *cmd) {
-	c.params = "[-id id] [-todomain domain] [-recipient address]"
+	c.params = "[-id id] [-todomain domain] [-recipient address] [-transport transport]"
 	c.help = `Schedule matching messages in the queue for immediate delivery.
 
 Messages deliveries are normally attempted with exponential backoff. The first
 retry after 7.5 minutes, and doubling each time. Kicking messages sets their
 next scheduled attempt to now, it can cause delivery to fail earlier than
 without rescheduling.
+
+With the -transport flag, future delivery attempts are done using the specified
+transport. Transports can be configured in mox.conf, e.g. to submit to a remote
+queue over SMTP.
 `
 	var id int64
-	var todomain, recipient string
+	var todomain, recipient, transport string
 	c.flag.Int64Var(&id, "id", 0, "id of message in queue")
 	c.flag.StringVar(&todomain, "todomain", "", "destination domain of messages")
 	c.flag.StringVar(&recipient, "recipient", "", "recipient email address")
+	c.flag.StringVar(&transport, "transport", "", "transport to use for the next delivery")
 	if len(c.Parse()) != 0 {
 		c.Usage()
 	}
@@ -1163,6 +1168,7 @@ without rescheduling.
 	ctl.xwrite(fmt.Sprintf("%d", id))
 	ctl.xwrite(todomain)
 	ctl.xwrite(recipient)
+	ctl.xwrite(transport)
 	count := ctl.xread()
 	line := ctl.xread()
 	if line == "ok" {
@@ -2041,27 +2047,3 @@ func cmdBumpUIDValidity(c *cmd) {
 	})
 	xcheckf(err, "updating database")
 }
-
-var submitconf struct {
-	LocalHostname      string `sconf-doc:"Hosts don't always have an FQDN, set it explicitly, for EHLO."`
-	Host               string `sconf-doc:"Host to dial for delivery, e.g. mail.<domain>."`
-	Port               int    `sconf-doc:"Port to dial for delivery, e.g. 465 for submissions, 587 for submission, or perhaps 25 for smtp."`
-	TLS                bool   `sconf-doc:"Connect with TLS. Usually for connections to port 465."`
-	STARTTLS           bool   `sconf-doc:"After starting in plain text, use STARTTLS to enable TLS. For port 587 and 25."`
-	Username           string `sconf-doc:"For SMTP plain auth."`
-	Password           string `sconf-doc:"For SMTP plain auth."`
-	AuthMethod         string `sconf-doc:"Ignored for now, regardless of value, AUTH PLAIN is done. This will change in the future."`
-	From               string `sconf-doc:"Address for MAIL FROM in SMTP and From-header in message."`
-	DefaultDestination string `sconf:"optional" sconf-doc:"Used when specified address does not contain an @ and may be a local user (eg root)."`
-}
-
-func cmdConfigDescribeSendmail(c *cmd) {
-	c.params = ">/etc/moxsubmit.conf"
-	c.help = `Describe configuration for mox when invoked as sendmail.`
-	if len(c.Parse()) != 0 {
-		c.Usage()
-	}
-
-	err := sconf.Describe(os.Stdout, submitconf)
-	xcheckf(err, "describe config")
-}

mox-/admin.go

@@ -923,8 +923,8 @@ func ClientConfigDomain(d dns.Domain) (ClientConfig, error) {
 	return c, nil
 }
 
-// return IPs we may be listening on or connecting from to the outside.
-func IPs(ctx context.Context) ([]net.IP, error) {
+// return IPs we may be listening/receiving mail on or connecting/sending from to the outside.
+func IPs(ctx context.Context, receiveOnly bool) ([]net.IP, error) {
 	log := xlog.WithContext(ctx)
 
 	// Try to gather all IPs we are listening on by going through the config.
@@ -982,5 +982,16 @@ func IPs(ctx context.Context) ([]net.IP, error) {
 			}
 		}
 	}
+
+	if receiveOnly {
+		return ips, nil
+	}
+
+	for _, t := range Conf.Static.Transports {
+		if t.Socks != nil {
+			ips = append(ips, t.Socks.IPs...)
+		}
+	}
+
 	return ips, nil
 }

mox-/config.go

@@ -217,6 +217,19 @@ func (c *Config) WebServer() (r map[dns.Domain]dns.Domain, l []config.WebHandler
 	return r, l
 }
 
+func (c *Config) Routes(accountName string, domain dns.Domain) (accountRoutes, domainRoutes, globalRoutes []config.Route) {
+	c.withDynamicLock(func() {
+		acc := c.Dynamic.Accounts[accountName]
+		accountRoutes = acc.Routes
+
+		dom := c.Dynamic.Domains[domain.Name()]
+		domainRoutes = dom.Routes
+
+		globalRoutes = c.Dynamic.Routes
+	})
+	return
+}
+
 func (c *Config) allowACMEHosts(checkACMEHosts bool) {
 	for _, l := range c.Static.Listeners {
 		if l.TLS == nil || l.TLS.ACME == "" {
@@ -408,12 +421,12 @@ func ParseConfig(ctx context.Context, p string, checkOnly, doLoadTLSKeyCerts, ch
 // PrepareStaticConfig parses the static config file and prepares data structures
 // for starting mox. If checkOnly is set no substantial changes are made, like
 // creating an ACME registration.
-func PrepareStaticConfig(ctx context.Context, configFile string, config *Config, checkOnly, doLoadTLSKeyCerts bool) (errs []error) {
+func PrepareStaticConfig(ctx context.Context, configFile string, conf *Config, checkOnly, doLoadTLSKeyCerts bool) (errs []error) {
 	addErrorf := func(format string, args ...any) {
 		errs = append(errs, fmt.Errorf(format, args...))
 	}
 
-	c := &config.Static
+	c := &conf.Static
 
 	// check that mailbox is in unicode NFC normalized form.
 	checkMailboxNormf := func(mailbox string, format string, args ...any) {
@@ -426,13 +439,13 @@ func PrepareStaticConfig(ctx context.Context, configFile string, config *Config,
 
 	// Post-process logging config.
 	if logLevel, ok := mlog.Levels[c.LogLevel]; ok {
-		config.Log = map[string]mlog.Level{"": logLevel}
+		conf.Log = map[string]mlog.Level{"": logLevel}
 	} else {
 		addErrorf("invalid log level %q", c.LogLevel)
 	}
 	for pkg, s := range c.PackageLogLevels {
 		if logLevel, ok := mlog.Levels[s]; ok {
-			config.Log[pkg] = logLevel
+			conf.Log[pkg] = logLevel
 		} else {
 			addErrorf("invalid package log level %q", s)
 		}
@@ -635,6 +648,87 @@ func PrepareStaticConfig(ctx context.Context, configFile string, config *Config,
 		checkMailboxNormf(mb, "default mailbox")
 	}
 
+	checkTransportSMTP := func(name string, isTLS bool, t *config.TransportSMTP) {
+		var err error
+		t.DNSHost, err = dns.ParseDomain(t.Host)
+		if err != nil {
+			addErrorf("transport %s: bad host %s: %v", name, t.Host, err)
+		}
+
+		if isTLS && t.STARTTLSInsecureSkipVerify {
+			addErrorf("transport %s: cannot have STARTTLSInsecureSkipVerify with immediate TLS")
+		}
+		if isTLS && t.NoSTARTTLS {
+			addErrorf("transport %s: cannot have NoSTARTTLS with immediate TLS")
+		}
+
+		if t.Auth == nil {
+			return
+		}
+		seen := map[string]bool{}
+		for _, m := range t.Auth.Mechanisms {
+			if seen[m] {
+				addErrorf("transport %s: duplicate authentication mechanism %s", name, m)
+			}
+			seen[m] = true
+			switch m {
+			case "SCRAM-SHA-256":
+			case "SCRAM-SHA-1":
+			case "CRAM-MD5":
+			case "PLAIN":
+			default:
+				addErrorf("transport %s: unknown authentication mechanism %s", name, m)
+			}
+		}
+
+		t.Auth.EffectiveMechanisms = t.Auth.Mechanisms
+		if len(t.Auth.EffectiveMechanisms) == 0 {
+			t.Auth.EffectiveMechanisms = []string{"SCRAM-SHA-256", "SCRAM-SHA-1", "CRAM-MD5"}
+		}
+	}
+
+	checkTransportSocks := func(name string, t *config.TransportSocks) {
+		_, _, err := net.SplitHostPort(t.Address)
+		if err != nil {
+			addErrorf("transport %s: bad address %s: %v", name, t.Address, err)
+		}
+		for _, ipstr := range t.RemoteIPs {
+			ip := net.ParseIP(ipstr)
+			if ip == nil {
+				addErrorf("transport %s: bad ip %s", name, ipstr)
+			} else {
+				t.IPs = append(t.IPs, ip)
+			}
+		}
+		t.Hostname, err = dns.ParseDomain(t.RemoteHostname)
+		if err != nil {
+			addErrorf("transport %s: bad hostname %s: %v", name, t.RemoteHostname, err)
+		}
+	}
+
+	for name, t := range c.Transports {
+		n := 0
+		if t.Submissions != nil {
+			n++
+			checkTransportSMTP(name, true, t.Submissions)
+		}
+		if t.Submission != nil {
+			n++
+			checkTransportSMTP(name, false, t.Submission)
+		}
+		if t.SMTP != nil {
+			n++
+			checkTransportSMTP(name, false, t.SMTP)
+		}
+		if t.Socks != nil {
+			n++
+			checkTransportSocks(name, t.Socks)
+		}
+		if n > 1 {
+			addErrorf("transport %s: cannot have multiple methods in a transport", name)
+		}
+	}
+
 	// Load CA certificate pool.
 	if c.TLS.CA != nil {
 		if c.TLS.CA.AdditionalToSystem {
@@ -718,6 +812,41 @@ func prepareDynamicConfig(ctx context.Context, dynamicPath string, static config
 		}
 	}
 
+	checkRoutes := func(descr string, routes []config.Route) {
+		parseRouteDomains := func(l []string) []string {
+			var r []string
+			for _, e := range l {
+				if e == "." {
+					r = append(r, e)
+					continue
+				}
+				prefix := ""
+				if strings.HasPrefix(e, ".") {
+					prefix = "."
+					e = e[1:]
+				}
+				d, err := dns.ParseDomain(e)
+				if err != nil {
+					addErrorf("%s: invalid domain %s: %v", descr, e, err)
+				}
+				r = append(r, prefix+d.ASCII)
+			}
+			return r
+		}
+
+		for i := range routes {
+			routes[i].FromDomainASCII = parseRouteDomains(routes[i].FromDomain)
+			routes[i].ToDomainASCII = parseRouteDomains(routes[i].ToDomain)
+			var ok bool
+			routes[i].ResolvedTransport, ok = static.Transports[routes[i].Transport]
+			if !ok {
+				addErrorf("%s: route references undefined transport %s", descr, routes[i].Transport)
+			}
+		}
+	}
+
+	checkRoutes("global routes", c.Routes)
+
 	// Validate domains.
 	for d, domain := range c.Domains {
 		dnsdomain, err := dns.ParseDomain(d)
@@ -836,6 +965,8 @@ func prepareDynamicConfig(ctx context.Context, dynamicPath string, static config
 			}
 		}
 
+		checkRoutes("routes for domain", domain.Routes)
+
 		c.Domains[d] = domain
 	}
 
@@ -1006,6 +1137,8 @@ func prepareDynamicConfig(ctx context.Context, dynamicPath string, static config
 				delete(acc.Destinations, lp)
 			}
 		}
+
+		checkRoutes("routes for account", acc.Routes)
 	}
 
 	// Set DMARC destinations.

queue/direct.go

@@ -0,0 +1,366 @@
+package queue
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/mjl-/bstore"
+
+	"github.com/mjl-/mox/dns"
+	"github.com/mjl-/mox/dsn"
+	"github.com/mjl-/mox/mlog"
+	"github.com/mjl-/mox/mox-"
+	"github.com/mjl-/mox/mtasts"
+	"github.com/mjl-/mox/mtastsdb"
+	"github.com/mjl-/mox/smtpclient"
+	"github.com/mjl-/mox/store"
+)
+
+// todo: rename function, perhaps put some of the params in a delivery struct so we don't pass all the params all the time?
+func fail(qlog *mlog.Log, m Msg, backoff time.Duration, permanent bool, remoteMTA dsn.NameIP, secodeOpt, errmsg string) {
+	if permanent || m.Attempts >= 8 {
+		qlog.Errorx("permanent failure delivering from queue", errors.New(errmsg))
+		queueDSNFailure(qlog, m, remoteMTA, secodeOpt, errmsg)
+
+		if err := queueDelete(context.Background(), m.ID); err != nil {
+			qlog.Errorx("deleting message from queue after permanent failure", err)
+		}
+		return
+	}
+
+	qup := bstore.QueryDB[Msg](context.Background(), DB)
+	qup.FilterID(m.ID)
+	if _, err := qup.UpdateNonzero(Msg{LastError: errmsg, DialedIPs: m.DialedIPs}); err != nil {
+		qlog.Errorx("storing delivery error", err, mlog.Field("deliveryerror", errmsg))
+	}
+
+	if m.Attempts == 5 {
+		// We've attempted deliveries at these intervals: 0, 7.5m, 15m, 30m, 1h, 2u.
+		// Let sender know delivery is delayed.
+		qlog.Errorx("temporary failure delivering from queue, sending delayed dsn", errors.New(errmsg), mlog.Field("backoff", backoff))
+
+		retryUntil := m.LastAttempt.Add((4 + 8 + 16) * time.Hour)
+		queueDSNDelay(qlog, m, remoteMTA, secodeOpt, errmsg, retryUntil)
+	} else {
+		qlog.Errorx("temporary failure delivering from queue", errors.New(errmsg), mlog.Field("backoff", backoff), mlog.Field("nextattempt", m.NextAttempt))
+	}
+}
+
+// Delivery by directly dialing MX hosts for destination domain.
+func deliverDirect(cid int64, qlog *mlog.Log, resolver dns.Resolver, dialer contextDialer, ourHostname dns.Domain, transportName string, m Msg, backoff time.Duration) {
+	hosts, effectiveDomain, permanent, err := gatherHosts(resolver, m, cid, qlog)
+	if err != nil {
+		fail(qlog, m, backoff, permanent, dsn.NameIP{}, "", err.Error())
+		return
+	}
+
+	// Check for MTA-STS policy and enforce it if needed. We have to check the
+	// effective domain (found after following CNAME record(s)): there will certainly
+	// not be an mtasts record for the original recipient domain, because that is not
+	// allowed when a CNAME record is present.
+	var policyFresh bool
+	var policy *mtasts.Policy
+	tlsModeDefault := smtpclient.TLSOpportunistic
+	if !effectiveDomain.IsZero() {
+		cidctx := context.WithValue(mox.Shutdown, mlog.CidKey, cid)
+		policy, policyFresh, err = mtastsdb.Get(cidctx, resolver, effectiveDomain)
+		if err != nil {
+			// No need to refuse to deliver if we have some mtasts error.
+			qlog.Infox("mtasts failed, continuing with strict tls requirement", err, mlog.Field("domain", effectiveDomain))
+			tlsModeDefault = smtpclient.TLSStrictStartTLS
+		}
+		// note: policy can be nil, if a domain does not implement MTA-STS or its the first
+		// time we fetch the policy and if we encountered an error.
+	}
+
+	// We try delivery to each record until we have success or a permanent failure. So
+	// for transient errors, we'll try the next MX record. For MX records pointing to a
+	// dual stack host, we turn a permanent failure due to policy on the first delivery
+	// attempt into a temporary failure and make sure to try the other address family
+	// the next attempt. This should reduce issues due to one of our IPs being on a
+	// block list. We won't try multiple IPs of the same address family. Surprisingly,
+	// RFC 5321 does not specify a clear algorithm, but common practicie is probably
+	// ../rfc/3974:268.
+	var remoteMTA dsn.NameIP
+	var secodeOpt, errmsg string
+	permanent = false
+	mtastsFailure := true
+	// todo: should make distinction between host permanently not accepting the message, and the message not being deliverable permanently. e.g. a mx host may have a size limit, or not accept 8bitmime, while another host in the list does accept the message. same for smtputf8, ../rfc/6531:555
+	for _, h := range hosts {
+		var badTLS, ok bool
+
+		// ../rfc/8461:913
+		if policy != nil && policy.Mode == mtasts.ModeEnforce && !policy.Matches(h.Domain) {
+			var policyHosts []string
+			for _, mx := range policy.MX {
+				policyHosts = append(policyHosts, mx.LogString())
+			}
+			errmsg = fmt.Sprintf("mx host %s does not match enforced mta-sts policy with hosts %s", h.Domain, strings.Join(policyHosts, ","))
+			qlog.Error("mx host does not match enforce mta-sts policy, skipping", mlog.Field("host", h.Domain), mlog.Field("policyhosts", policyHosts))
+			continue
+		}
+
+		qlog.Info("delivering to remote", mlog.Field("remote", h), mlog.Field("queuecid", cid))
+		cid := mox.Cid()
+		nqlog := qlog.WithCid(cid)
+		var remoteIP net.IP
+		tlsMode := tlsModeDefault
+		if policy != nil && policy.Mode == mtasts.ModeEnforce {
+			tlsMode = smtpclient.TLSStrictStartTLS
+		}
+		permanent, badTLS, secodeOpt, remoteIP, errmsg, ok = deliverHost(nqlog, resolver, dialer, cid, ourHostname, transportName, h, &m, tlsMode)
+		if !ok && badTLS && tlsMode == smtpclient.TLSOpportunistic {
+			// In case of failure with opportunistic TLS, try again without TLS. ../rfc/7435:459
+			// todo future: revisit this decision. perhaps it should be a configuration option that defaults to not doing this?
+			nqlog.Info("connecting again for delivery attempt without tls")
+			permanent, badTLS, secodeOpt, remoteIP, errmsg, ok = deliverHost(nqlog, resolver, dialer, cid, ourHostname, transportName, h, &m, smtpclient.TLSSkip)
+		}
+		if ok {
+			nqlog.Info("delivered from queue")
+			if err := queueDelete(context.Background(), m.ID); err != nil {
+				nqlog.Errorx("deleting message from queue after delivery", err)
+			}
+			return
+		}
+		remoteMTA = dsn.NameIP{Name: h.XString(false), IP: remoteIP}
+		if !badTLS {
+			mtastsFailure = false
+		}
+		if permanent {
+			break
+		}
+	}
+	if mtastsFailure && policyFresh {
+		permanent = true
+	}
+
+	fail(qlog, m, backoff, permanent, remoteMTA, secodeOpt, errmsg)
+}
+
+var (
+	errCNAMELoop  = errors.New("cname loop")
+	errCNAMELimit = errors.New("too many cname records")
+	errNoRecord   = errors.New("no dns record")
+	errDNS        = errors.New("dns lookup error")
+	errNoMail     = errors.New("domain does not accept email as indicated with single dot for mx record")
+)
+
+// Gather hosts to try to deliver to. We start with the straight-forward MX record.
+// If that does not exist, we'll look for CNAME of the entire domain (following
+// chains if needed). If a CNAME does not exist, but the domain name has an A or
+// AAAA record, we'll try delivery directly to that host.
+// ../rfc/5321:3824
+func gatherHosts(resolver dns.Resolver, m Msg, cid int64, qlog *mlog.Log) (hosts []dns.IPDomain, effectiveDomain dns.Domain, permanent bool, err error) {
+	if len(m.RecipientDomain.IP) > 0 {
+		return []dns.IPDomain{m.RecipientDomain}, effectiveDomain, false, nil
+	}
+
+	// We start out delivering to the recipient domain. We follow CNAMEs a few times.
+	rcptDomain := m.RecipientDomain.Domain
+	// Domain we are actually delivering to, after following CNAME record(s).
+	effectiveDomain = rcptDomain
+	domainsSeen := map[string]bool{}
+	for i := 0; ; i++ {
+		if domainsSeen[effectiveDomain.ASCII] {
+			return nil, effectiveDomain, true, fmt.Errorf("%w: recipient domain %s: already saw %s", errCNAMELoop, rcptDomain, effectiveDomain)
+		}
+		domainsSeen[effectiveDomain.ASCII] = true
+
+		// note: The Go resolver returns the requested name if the domain has no CNAME record but has a host record.
+		if i == 16 {
+			// We have a maximum number of CNAME records we follow. There is no hard limit for
+			// DNS, and you might think folks wouldn't configure CNAME chains at all, but for
+			// (non-mail) domains, CNAME chains of 10 records have been encountered according
+			// to the internet.
+			return nil, effectiveDomain, true, fmt.Errorf("%w: recipient domain %s, last resolved domain %s", errCNAMELimit, rcptDomain, effectiveDomain)
+		}
+
+		cidctx := context.WithValue(mox.Context, mlog.CidKey, cid)
+		ctx, cancel := context.WithTimeout(cidctx, 30*time.Second)
+		defer cancel()
+		// Note: LookupMX can return an error and still return records: Invalid records are
+		// filtered out and an error returned. We must process any records that are valid.
+		// Only if all are unusable will we return an error. ../rfc/5321:3851
+		mxl, err := resolver.LookupMX(ctx, effectiveDomain.ASCII+".")
+		cancel()
+		if err != nil && len(mxl) == 0 {
+			if !dns.IsNotFound(err) {
+				return nil, effectiveDomain, false, fmt.Errorf("%w: mx lookup for %s: %v", errDNS, effectiveDomain, err)
+			}
+
+			// No MX record. First attempt CNAME lookup. ../rfc/5321:3838 ../rfc/3974:197
+			ctx, cancel = context.WithTimeout(cidctx, 30*time.Second)
+			defer cancel()
+			cname, err := resolver.LookupCNAME(ctx, effectiveDomain.ASCII+".")
+			cancel()
+			if err != nil && !dns.IsNotFound(err) {
+				return nil, effectiveDomain, false, fmt.Errorf("%w: cname lookup for %s: %v", errDNS, effectiveDomain, err)
+			}
+			if err == nil && cname != effectiveDomain.ASCII+"." {
+				d, err := dns.ParseDomain(strings.TrimSuffix(cname, "."))
+				if err != nil {
+					return nil, effectiveDomain, true, fmt.Errorf("%w: parsing cname domain %s: %v", errDNS, effectiveDomain, err)
+				}
+				effectiveDomain = d
+				// Start again with new domain.
+				continue
+			}
+
+			// See if the host exists. If so, attempt delivery directly to host. ../rfc/5321:3842
+			ctx, cancel = context.WithTimeout(cidctx, 30*time.Second)
+			defer cancel()
+			_, err = resolver.LookupHost(ctx, effectiveDomain.ASCII+".")
+			cancel()
+			if dns.IsNotFound(err) {
+				return nil, effectiveDomain, true, fmt.Errorf("%w: recipient domain/host %s", errNoRecord, effectiveDomain)
+			} else if err != nil {
+				return nil, effectiveDomain, false, fmt.Errorf("%w: looking up host %s because of no mx record: %v", errDNS, effectiveDomain, err)
+			}
+			hosts = []dns.IPDomain{{Domain: effectiveDomain}}
+		} else if err != nil {
+			qlog.Infox("partial mx failure, attempting delivery to valid mx records", err)
+		}
+
+		// ../rfc/7505:122
+		if err == nil && len(mxl) == 1 && mxl[0].Host == "." {
+			return nil, effectiveDomain, true, errNoMail
+		}
+
+		// The Go resolver already sorts by preference, randomizing records of same
+		// preference. ../rfc/5321:3885
+		for _, mx := range mxl {
+			host, err := dns.ParseDomain(strings.TrimSuffix(mx.Host, "."))
+			if err != nil {
+				// note: should not happen because Go resolver already filters these out.
+				return nil, effectiveDomain, true, fmt.Errorf("%w: invalid host name in mx record %q: %v", errDNS, mx.Host, err)
+			}
+			hosts = append(hosts, dns.IPDomain{Domain: host})
+		}
+		if len(hosts) > 0 {
+			err = nil
+		}
+		return hosts, effectiveDomain, false, err
+	}
+}
+
+// deliverHost attempts to deliver m to host.
+// deliverHost updated m.DialedIPs, which must be saved in case of failure to deliver.
+func deliverHost(log *mlog.Log, resolver dns.Resolver, dialer contextDialer, cid int64, ourHostname dns.Domain, transportName string, host dns.IPDomain, m *Msg, tlsMode smtpclient.TLSMode) (permanent, badTLS bool, secodeOpt string, remoteIP net.IP, errmsg string, ok bool) {
+	// About attempting delivery to multiple addresses of a host: ../rfc/5321:3898
+
+	start := time.Now()
+	var deliveryResult string
+	defer func() {
+		metricDelivery.WithLabelValues(fmt.Sprintf("%d", m.Attempts), transportName, string(tlsMode), deliveryResult).Observe(float64(time.Since(start)) / float64(time.Second))
+		log.Debug("queue deliverhost result", mlog.Field("host", host), mlog.Field("attempt", m.Attempts), mlog.Field("tlsmode", tlsMode), mlog.Field("permanent", permanent), mlog.Field("badtls", badTLS), mlog.Field("secodeopt", secodeOpt), mlog.Field("errmsg", errmsg), mlog.Field("ok", ok), mlog.Field("duration", time.Since(start)))
+	}()
+
+	f, err := os.Open(m.MessagePath())
+	if err != nil {
+		return false, false, "", nil, fmt.Sprintf("open message file: %s", err), false
+	}
+	msgr := store.FileMsgReader(m.MsgPrefix, f)
+	defer func() {
+		err := msgr.Close()
+		log.Check(err, "closing message after delivery attempt")
+	}()
+
+	cidctx := context.WithValue(mox.Context, mlog.CidKey, cid)
+	ctx, cancel := context.WithTimeout(cidctx, 30*time.Second)
+	defer cancel()
+
+	conn, ip, dualstack, err := dialHost(ctx, log, resolver, dialer, host, 25, m)
+	remoteIP = ip
+	cancel()
+	var result string
+	switch {
+	case err == nil:
+		result = "ok"
+	case errors.Is(err, os.ErrDeadlineExceeded), errors.Is(err, context.DeadlineExceeded):
+		result = "timeout"
+	case errors.Is(err, context.Canceled):
+		result = "canceled"
+	default:
+		result = "error"
+	}
+	metricConnection.WithLabelValues(result).Inc()
+	if err != nil {
+		log.Debugx("connecting to remote smtp", err, mlog.Field("host", host))
+		return false, false, "", ip, fmt.Sprintf("dialing smtp server: %v", err), false
+	}
+
+	var mailFrom string
+	if m.SenderLocalpart != "" || !m.SenderDomain.IsZero() {
+		mailFrom = m.Sender().XString(m.SMTPUTF8)
+	}
+	rcptTo := m.Recipient().XString(m.SMTPUTF8)
+
+	// todo future: get closer to timeouts specified in rfc? ../rfc/5321:3610
+	log = log.Fields(mlog.Field("remoteip", ip))
+	ctx, cancel = context.WithTimeout(cidctx, 30*time.Minute)
+	defer cancel()
+	mox.Connections.Register(conn, "smtpclient", "queue")
+	sc, err := smtpclient.New(ctx, log, conn, tlsMode, ourHostname, host.Domain, nil)
+	defer func() {
+		if sc == nil {
+			conn.Close()
+		} else {
+			sc.Close()
+		}
+		mox.Connections.Unregister(conn)
+	}()
+	if err == nil {
+		has8bit := m.Has8bit
+		smtputf8 := m.SMTPUTF8
+		var msg io.Reader = msgr
+		size := m.Size
+		if m.DSNUTF8 != nil && sc.Supports8BITMIME() && sc.SupportsSMTPUTF8() {
+			has8bit = true
+			smtputf8 = true
+			size = int64(len(m.DSNUTF8))
+			msg = bytes.NewReader(m.DSNUTF8)
+		}
+		err = sc.Deliver(ctx, mailFrom, rcptTo, size, msg, has8bit, smtputf8)
+	}
+	if err != nil {
+		log.Infox("delivery failed", err)
+	}
+	var cerr smtpclient.Error
+	switch {
+	case err == nil:
+		deliveryResult = "ok"
+	case errors.Is(err, os.ErrDeadlineExceeded), errors.Is(err, context.DeadlineExceeded):
+		deliveryResult = "timeout"
+	case errors.Is(err, context.Canceled):
+		deliveryResult = "canceled"
+	case errors.As(err, &cerr):
+		deliveryResult = "temperror"
+		if cerr.Permanent {
+			deliveryResult = "permerror"
+		}
+	default:
+		deliveryResult = "error"
+	}
+	if err == nil {
+		return false, false, "", ip, "", true
+	} else if cerr, ok := err.(smtpclient.Error); ok {
+		// If we are being rejected due to policy reasons on the first
+		// attempt and remote has both IPv4 and IPv6, we'll give it
+		// another try. Our first IP may be in a block list, the address for
+		// the other family perhaps is not.
+		permanent := cerr.Permanent
+		if permanent && m.Attempts == 1 && dualstack && strings.HasPrefix(cerr.Secode, "7.") {
+			permanent = false
+		}
+		return permanent, errors.Is(cerr, smtpclient.ErrTLS), cerr.Secode, ip, cerr.Error(), false
+	} else {
+		return false, errors.Is(cerr, smtpclient.ErrTLS), "", ip, err.Error(), false
+	}
+}

queue/queue.go

@@ -4,9 +4,7 @@
 package queue
 
 import (
-	"bytes"
 	"context"
-	"errors"
 	"fmt"
 	"io"
 	"net"
@@ -17,21 +15,21 @@ import (
 	"strings"
 	"time"
 
+	"golang.org/x/net/proxy"
+
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promauto"
 
 	"github.com/mjl-/bstore"
 
+	"github.com/mjl-/mox/config"
 	"github.com/mjl-/mox/dns"
 	"github.com/mjl-/mox/dsn"
 	"github.com/mjl-/mox/metrics"
 	"github.com/mjl-/mox/mlog"
 	"github.com/mjl-/mox/mox-"
 	"github.com/mjl-/mox/moxio"
-	"github.com/mjl-/mox/mtasts"
-	"github.com/mjl-/mox/mtastsdb"
 	"github.com/mjl-/mox/smtp"
-	"github.com/mjl-/mox/smtpclient"
 	"github.com/mjl-/mox/store"
 )
 
@@ -47,24 +45,36 @@ var (
 			"result", // "ok", "timeout", "canceled", "error"
 		},
 	)
-	metricDeliveryHost = promauto.NewHistogramVec(
+	metricDelivery = promauto.NewHistogramVec(
 		prometheus.HistogramOpts{
 			Name:    "mox_queue_delivery_duration_seconds",
 			Help:    "SMTP client delivery attempt to single host.",
 			Buckets: []float64{0.01, 0.05, 0.100, 0.5, 1, 5, 10, 20, 30, 60, 120},
 		},
 		[]string{
-			"attempt", // Number of attempts.
-			"tlsmode", // strict, opportunistic, skip
-			"result",  // ok, timeout, canceled, temperror, permerror, error
+			"attempt",   // Number of attempts.
+			"transport", // empty for default direct delivery.
+			"tlsmode",   // strict, opportunistic, skip
+			"result",    // ok, timeout, canceled, temperror, permerror, error
 		},
 	)
 )
 
+type contextDialer interface {
+	DialContext(ctx context.Context, network, addr string) (c net.Conn, err error)
+}
+
 // Used to dial remote SMTP servers.
 // Overridden for tests.
-var dial = func(ctx context.Context, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
-	dialer := &net.Dialer{Timeout: timeout, LocalAddr: laddr}
+var dial = func(ctx context.Context, dialer contextDialer, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
+	// If this is a net.Dialer, use its settings and add the timeout and localaddr.
+	// This is the typical case, but SOCKS5 support can use a different dialer.
+	if d, ok := dialer.(*net.Dialer); ok {
+		nd := *d
+		nd.Timeout = timeout
+		nd.LocalAddr = laddr
+		return nd.DialContext(ctx, "tcp", addr)
+	}
 	return dialer.DialContext(ctx, "tcp", addr)
 }
 
@@ -80,7 +90,7 @@ var Localserve bool
 type Msg struct {
 	ID                 int64
 	Queued             time.Time      `bstore:"default now"`
-	SenderAccount      string         // Failures are delivered back to this local account.
+	SenderAccount      string         // Failures are delivered back to this local account. Also used for routing.
 	SenderLocalpart    smtp.Localpart // Should be a local user and domain.
 	SenderDomain       dns.IPDomain
 	RecipientLocalpart smtp.Localpart // Typically a remote user and domain.
@@ -95,7 +105,16 @@ type Msg struct {
 	SMTPUTF8           bool  // Whether message requires use of SMTPUTF8.
 	Size               int64 // Full size of message, combined MsgPrefix with contents of message file.
 	MsgPrefix          []byte
-	DSNUTF8            []byte // If set, this message is a DSN and this is a version using utf-8, for the case the remote MTA supports smtputf8. In this case, Size and MsgPrefix are not relevant.
+
+	// If set, this message is a DSN and this is a version using utf-8, for the case
+	// the remote MTA supports smtputf8. In this case, Size and MsgPrefix are not
+	// relevant.
+	DSNUTF8 []byte
+
+	// If non-empty, the transport to use for this message. Can be set through cli or
+	// admin interface. If empty (the default for a submitted message), regular routing
+	// rules apply.
+	Transport string
 }
 
 // Sender of message as used in MAIL FROM.
@@ -180,17 +199,17 @@ func Count(ctx context.Context) (int, error) {
 // this data is used as the message when delivering the DSN and the remote SMTP
 // server supports SMTPUTF8. If the remote SMTP server does not support SMTPUTF8,
 // the regular non-utf8 message is delivered.
-func Add(ctx context.Context, log *mlog.Log, senderAccount string, mailFrom, rcptTo smtp.Path, has8bit, smtputf8 bool, size int64, msgPrefix []byte, msgFile *os.File, dsnutf8Opt []byte, consumeFile bool) error {
+func Add(ctx context.Context, log *mlog.Log, senderAccount string, mailFrom, rcptTo smtp.Path, has8bit, smtputf8 bool, size int64, msgPrefix []byte, msgFile *os.File, dsnutf8Opt []byte, consumeFile bool) (int64, error) {
 	// todo: Add should accept multiple rcptTo if they are for the same domain. so we can queue them for delivery in one (or just a few) session(s), transferring the data only once. ../rfc/5321:3759
 
 	if Localserve {
 		// Safety measure, shouldn't happen.
-		return fmt.Errorf("no queuing with localserve")
+		return 0, fmt.Errorf("no queuing with localserve")
 	}
 
 	tx, err := DB.Begin(ctx, true)
 	if err != nil {
-		return fmt.Errorf("begin transaction: %w", err)
+		return 0, fmt.Errorf("begin transaction: %w", err)
 	}
 	defer func() {
 		if tx != nil {
@@ -201,10 +220,10 @@ func Add(ctx context.Context, log *mlog.Log, senderAccount string, mailFrom, rcp
 	}()
 
 	now := time.Now()
-	qm := Msg{0, now, senderAccount, mailFrom.Localpart, mailFrom.IPDomain, rcptTo.Localpart, rcptTo.IPDomain, formatIPDomain(rcptTo.IPDomain), 0, nil, now, nil, "", has8bit, smtputf8, size, msgPrefix, dsnutf8Opt}
+	qm := Msg{0, now, senderAccount, mailFrom.Localpart, mailFrom.IPDomain, rcptTo.Localpart, rcptTo.IPDomain, formatIPDomain(rcptTo.IPDomain), 0, nil, now, nil, "", has8bit, smtputf8, size, msgPrefix, dsnutf8Opt, ""}
 
 	if err := tx.Insert(&qm); err != nil {
-		return err
+		return 0, err
 	}
 
 	dst := qm.MessagePath()
@@ -219,27 +238,27 @@ func Add(ctx context.Context, log *mlog.Log, senderAccount string, mailFrom, rcp
 	if consumeFile {
 		if err := os.Rename(msgFile.Name(), dst); err != nil {
 			// Could be due to cross-filesystem rename. Users shouldn't configure their systems that way.
-			return fmt.Errorf("move message into queue dir: %w", err)
+			return 0, fmt.Errorf("move message into queue dir: %w", err)
 		}
 	} else if err := os.Link(msgFile.Name(), dst); err != nil {
 		// Assume file system does not support hardlinks. Copy it instead.
 		if err := writeFile(dst, &moxio.AtReader{R: msgFile}); err != nil {
-			return fmt.Errorf("copying message to new file: %s", err)
+			return 0, fmt.Errorf("copying message to new file: %s", err)
 		}
 	}
 
 	if err := moxio.SyncDir(dstDir); err != nil {
-		return fmt.Errorf("sync directory: %v", err)
+		return 0, fmt.Errorf("sync directory: %v", err)
 	}
 
 	if err := tx.Commit(); err != nil {
-		return fmt.Errorf("commit transaction: %s", err)
+		return 0, fmt.Errorf("commit transaction: %s", err)
 	}
 	tx = nil
 	dst = ""
 
 	queuekick()
-	return nil
+	return qm.ID, nil
 }
 
 // write contents of r to new file dst, for delivering a message.
@@ -284,11 +303,13 @@ func queuekick() {
 	}
 }
 
-// Kick sets the NextAttempt for messages matching all parameters that are nonzero,
-// and kicks the queue, attempting delivery of those messages. If all parameters
-// are zero, all messages are kicked.
+// Kick sets the NextAttempt for messages matching all filter parameters (ID,
+// toDomain, recipient) that are nonzero, and kicks the queue, attempting delivery
+// of those messages. If all parameters are zero, all messages are kicked. If
+// transport is set, the delivery attempts for the matching messages will use the
+// transport. An empty string is the default transport, i.e. direct delivery.
 // Returns number of messages queued for immediate delivery.
-func Kick(ctx context.Context, ID int64, toDomain string, recipient string) (int, error) {
+func Kick(ctx context.Context, ID int64, toDomain, recipient string, transport *string) (int, error) {
 	q := bstore.QueryDB[Msg](ctx, DB)
 	if ID > 0 {
 		q.FilterID(ID)
@@ -301,7 +322,17 @@ func Kick(ctx context.Context, ID int64, toDomain string, recipient string) (int
 			return qm.Recipient().XString(true) == recipient
 		})
 	}
-	n, err := q.UpdateNonzero(Msg{NextAttempt: time.Now()})
+	up := map[string]any{"NextAttempt": time.Now()}
+	if transport != nil {
+		if *transport != "" {
+			_, ok := mox.Conf.Static.Transports[*transport]
+			if !ok {
+				return 0, fmt.Errorf("unknown transport %q", *transport)
+			}
+		}
+		up["Transport"] = *transport
+	}
+	n, err := q.UpdateFields(up)
 	if err != nil {
 		return 0, fmt.Errorf("selecting and updating messages in queue: %v", err)
 	}
@@ -500,343 +531,92 @@ func deliver(resolver dns.Resolver, m Msg) {
 		return
 	}
 
-	fail := func(permanent bool, remoteMTA dsn.NameIP, secodeOpt, errmsg string) {
-		if permanent || m.Attempts >= 8 {
-			qlog.Errorx("permanent failure delivering from queue", errors.New(errmsg))
-			queueDSNFailure(qlog, m, remoteMTA, secodeOpt, errmsg)
-
-			if err := queueDelete(context.Background(), m.ID); err != nil {
-				qlog.Errorx("deleting message from queue after permanent failure", err)
-			}
+	// Find route for transport to use for delivery attempt.
+	var transport config.Transport
+	var transportName string
+	if m.Transport != "" {
+		var ok bool
+		transport, ok = mox.Conf.Static.Transports[m.Transport]
+		if !ok {
+			var remoteMTA dsn.NameIP // Zero value, will not be included in DSN. ../rfc/3464:1027
+			fail(qlog, m, backoff, false, remoteMTA, "", fmt.Sprintf("cannot find transport %q", m.Transport))
 			return
 		}
-
-		qup := bstore.QueryDB[Msg](context.Background(), DB)
-		qup.FilterID(m.ID)
-		if _, err := qup.UpdateNonzero(Msg{LastError: errmsg, DialedIPs: m.DialedIPs}); err != nil {
-			qlog.Errorx("storing delivery error", err, mlog.Field("deliveryerror", errmsg))
-		}
-
-		if m.Attempts == 5 {
-			// We've attempted deliveries at these intervals: 0, 7.5m, 15m, 30m, 1h, 2u.
-			// Let sender know delivery is delayed.
-			qlog.Errorx("temporary failure delivering from queue, sending delayed dsn", errors.New(errmsg), mlog.Field("backoff", backoff))
-
-			retryUntil := m.LastAttempt.Add((4 + 8 + 16) * time.Hour)
-			queueDSNDelay(qlog, m, remoteMTA, secodeOpt, errmsg, retryUntil)
-		} else {
-			qlog.Errorx("temporary failure delivering from queue", errors.New(errmsg), mlog.Field("backoff", backoff), mlog.Field("nextattempt", m.NextAttempt))
-		}
-	}
-
-	hosts, effectiveDomain, permanent, err := gatherHosts(resolver, m, cid, qlog)
-	if err != nil {
-		fail(permanent, dsn.NameIP{}, "", err.Error())
-		return
-	}
-
-	// Check for MTA-STS policy and enforce it if needed. We have to check the
-	// effective domain (found after following CNAME record(s)): there will certainly
-	// not be an mtasts record for the original recipient domain, because that is not
-	// allowed when a CNAME record is present.
-	var policyFresh bool
-	var policy *mtasts.Policy
-	tlsModeDefault := smtpclient.TLSOpportunistic
-	if !effectiveDomain.IsZero() {
-		cidctx := context.WithValue(mox.Shutdown, mlog.CidKey, cid)
-		policy, policyFresh, err = mtastsdb.Get(cidctx, resolver, effectiveDomain)
-		if err != nil {
-			// No need to refuse to deliver if we have some mtasts error.
-			qlog.Infox("mtasts failed, continuing with strict tls requirement", err, mlog.Field("domain", effectiveDomain))
-			tlsModeDefault = smtpclient.TLSStrict
-		}
-		// note: policy can be nil, if a domain does not implement MTA-STS or its the first
-		// time we fetch the policy and if we encountered an error.
-	}
-
-	// We try delivery to each record until we have success or a permanent failure. So
-	// for transient errors, we'll try the next MX record. For MX records pointing to a
-	// dual stack host, we turn a permanent failure due to policy on the first delivery
-	// attempt into a temporary failure and make sure to try the other address family
-	// the next attempt. This should reduce issues due to one of our IPs being on a
-	// block list. We won't try multiple IPs of the same address family. Surprisingly,
-	// RFC 5321 does not specify a clear algorithm, but common practicie is probably
-	// ../rfc/3974:268.
-	var remoteMTA dsn.NameIP
-	var secodeOpt, errmsg string
-	permanent = false
-	mtastsFailure := true
-	// todo: should make distinction between host permanently not accepting the message, and the message not being deliverable permanently. e.g. a mx host may have a size limit, or not accept 8bitmime, while another host in the list does accept the message. same for smtputf8, ../rfc/6531:555
-	for _, h := range hosts {
-		var badTLS, ok bool
-
-		// ../rfc/8461:913
-		if policy != nil && policy.Mode == mtasts.ModeEnforce && !policy.Matches(h.Domain) {
-			var policyHosts []string
-			for _, mx := range policy.MX {
-				policyHosts = append(policyHosts, mx.LogString())
-			}
-			errmsg = fmt.Sprintf("mx host %s does not match enforced mta-sts policy with hosts %s", h.Domain, strings.Join(policyHosts, ","))
-			qlog.Error("mx host does not match enforce mta-sts policy, skipping", mlog.Field("host", h.Domain), mlog.Field("policyhosts", policyHosts))
-			continue
-		}
-
-		qlog.Info("delivering to remote", mlog.Field("remote", h), mlog.Field("queuecid", cid))
-		cid := mox.Cid()
-		nqlog := qlog.WithCid(cid)
-		var remoteIP net.IP
-		tlsMode := tlsModeDefault
-		if policy != nil && policy.Mode == mtasts.ModeEnforce {
-			tlsMode = smtpclient.TLSStrict
-		}
-		permanent, badTLS, secodeOpt, remoteIP, errmsg, ok = deliverHost(nqlog, resolver, cid, h, &m, tlsMode)
-		if !ok && badTLS && tlsMode == smtpclient.TLSOpportunistic {
-			// In case of failure with opportunistic TLS, try again without TLS. ../rfc/7435:459
-			// todo future: revisit this decision. perhaps it should be a configuration option that defaults to not doing this?
-			nqlog.Info("connecting again for delivery attempt without tls")
-			permanent, badTLS, secodeOpt, remoteIP, errmsg, ok = deliverHost(nqlog, resolver, cid, h, &m, smtpclient.TLSSkip)
-		}
-		if ok {
-			nqlog.Info("delivered from queue")
-			if err := queueDelete(context.Background(), m.ID); err != nil {
-				nqlog.Errorx("deleting message from queue after delivery", err)
-			}
-			return
-		}
-		remoteMTA = dsn.NameIP{Name: h.XString(false), IP: remoteIP}
-		if !badTLS {
-			mtastsFailure = false
-		}
-		if permanent {
-			break
-		}
-	}
-	if mtastsFailure && policyFresh {
-		permanent = true
-	}
-
-	fail(permanent, remoteMTA, secodeOpt, errmsg)
-}
-
-var (
-	errCNAMELoop  = errors.New("cname loop")
-	errCNAMELimit = errors.New("too many cname records")
-	errNoRecord   = errors.New("no dns record")
-	errDNS        = errors.New("dns lookup error")
-	errNoMail     = errors.New("domain does not accept email as indicated with single dot for mx record")
-)
-
-// Gather hosts to try to deliver to. We start with the straight-forward MX record.
-// If that does not exist, we'll look for CNAME of the entire domain (following
-// chains if needed). If a CNAME does not exist, but the domain name has an A or
-// AAAA record, we'll try delivery directly to that host.
-// ../rfc/5321:3824
-func gatherHosts(resolver dns.Resolver, m Msg, cid int64, qlog *mlog.Log) (hosts []dns.IPDomain, effectiveDomain dns.Domain, permanent bool, err error) {
-	if len(m.RecipientDomain.IP) > 0 {
-		return []dns.IPDomain{m.RecipientDomain}, effectiveDomain, false, nil
-	}
-
-	// We start out delivering to the recipient domain. We follow CNAMEs a few times.
-	rcptDomain := m.RecipientDomain.Domain
-	// Domain we are actually delivering to, after following CNAME record(s).
-	effectiveDomain = rcptDomain
-	domainsSeen := map[string]bool{}
-	for i := 0; ; i++ {
-		if domainsSeen[effectiveDomain.ASCII] {
-			return nil, effectiveDomain, true, fmt.Errorf("%w: recipient domain %s: already saw %s", errCNAMELoop, rcptDomain, effectiveDomain)
-		}
-		domainsSeen[effectiveDomain.ASCII] = true
-
-		// note: The Go resolver returns the requested name if the domain has no CNAME record but has a host record.
-		if i == 16 {
-			// We have a maximum number of CNAME records we follow. There is no hard limit for
-			// DNS, and you might think folks wouldn't configure CNAME chains at all, but for
-			// (non-mail) domains, CNAME chains of 10 records have been encountered according
-			// to the internet.
-			return nil, effectiveDomain, true, fmt.Errorf("%w: recipient domain %s, last resolved domain %s", errCNAMELimit, rcptDomain, effectiveDomain)
-		}
-
-		cidctx := context.WithValue(mox.Context, mlog.CidKey, cid)
-		ctx, cancel := context.WithTimeout(cidctx, 30*time.Second)
-		defer cancel()
-		// Note: LookupMX can return an error and still return records: Invalid records are
-		// filtered out and an error returned. We must process any records that are valid.
-		// Only if all are unusable will we return an error. ../rfc/5321:3851
-		mxl, err := resolver.LookupMX(ctx, effectiveDomain.ASCII+".")
-		cancel()
-		if err != nil && len(mxl) == 0 {
-			if !dns.IsNotFound(err) {
-				return nil, effectiveDomain, false, fmt.Errorf("%w: mx lookup for %s: %v", errDNS, effectiveDomain, err)
-			}
-
-			// No MX record. First attempt CNAME lookup. ../rfc/5321:3838 ../rfc/3974:197
-			ctx, cancel = context.WithTimeout(cidctx, 30*time.Second)
-			defer cancel()
-			cname, err := resolver.LookupCNAME(ctx, effectiveDomain.ASCII+".")
-			cancel()
-			if err != nil && !dns.IsNotFound(err) {
-				return nil, effectiveDomain, false, fmt.Errorf("%w: cname lookup for %s: %v", errDNS, effectiveDomain, err)
-			}
-			if err == nil && cname != effectiveDomain.ASCII+"." {
-				d, err := dns.ParseDomain(strings.TrimSuffix(cname, "."))
-				if err != nil {
-					return nil, effectiveDomain, true, fmt.Errorf("%w: parsing cname domain %s: %v", errDNS, effectiveDomain, err)
-				}
-				effectiveDomain = d
-				// Start again with new domain.
-				continue
-			}
-
-			// See if the host exists. If so, attempt delivery directly to host. ../rfc/5321:3842
-			ctx, cancel = context.WithTimeout(cidctx, 30*time.Second)
-			defer cancel()
-			_, err = resolver.LookupHost(ctx, effectiveDomain.ASCII+".")
-			cancel()
-			if dns.IsNotFound(err) {
-				return nil, effectiveDomain, true, fmt.Errorf("%w: recipient domain/host %s", errNoRecord, effectiveDomain)
-			} else if err != nil {
-				return nil, effectiveDomain, false, fmt.Errorf("%w: looking up host %s because of no mx record: %v", errDNS, effectiveDomain, err)
-			}
-			hosts = []dns.IPDomain{{Domain: effectiveDomain}}
-		} else if err != nil {
-			qlog.Infox("partial mx failure, attempting delivery to valid mx records", err)
-		}
-
-		// ../rfc/7505:122
-		if err == nil && len(mxl) == 1 && mxl[0].Host == "." {
-			return nil, effectiveDomain, true, errNoMail
-		}
-
-		// The Go resolver already sorts by preference, randomizing records of same
-		// preference. ../rfc/5321:3885
-		for _, mx := range mxl {
-			host, err := dns.ParseDomain(strings.TrimSuffix(mx.Host, "."))
-			if err != nil {
-				// note: should not happen because Go resolver already filters these out.
-				return nil, effectiveDomain, true, fmt.Errorf("%w: invalid host name in mx record %q: %v", errDNS, mx.Host, err)
-			}
-			hosts = append(hosts, dns.IPDomain{Domain: host})
-		}
-		if len(hosts) > 0 {
-			err = nil
-		}
-		return hosts, effectiveDomain, false, err
-	}
-}
-
-// deliverHost attempts to deliver m to host.
-// deliverHost updated m.DialedIPs, which must be saved in case of failure to deliver.
-func deliverHost(log *mlog.Log, resolver dns.Resolver, cid int64, host dns.IPDomain, m *Msg, tlsMode smtpclient.TLSMode) (permanent, badTLS bool, secodeOpt string, remoteIP net.IP, errmsg string, ok bool) {
-	// About attempting delivery to multiple addresses of a host: ../rfc/5321:3898
-
-	start := time.Now()
-	var deliveryResult string
-	defer func() {
-		metricDeliveryHost.WithLabelValues(fmt.Sprintf("%d", m.Attempts), string(tlsMode), deliveryResult).Observe(float64(time.Since(start)) / float64(time.Second))
-		log.Debug("queue deliverhost result", mlog.Field("host", host), mlog.Field("attempt", m.Attempts), mlog.Field("tlsmode", tlsMode), mlog.Field("permanent", permanent), mlog.Field("badtls", badTLS), mlog.Field("secodeopt", secodeOpt), mlog.Field("errmsg", errmsg), mlog.Field("ok", ok), mlog.Field("duration", time.Since(start)))
-	}()
-
-	f, err := os.Open(m.MessagePath())
-	if err != nil {
-		return false, false, "", nil, fmt.Sprintf("open message file: %s", err), false
-	}
-	msgr := store.FileMsgReader(m.MsgPrefix, f)
-	defer func() {
-		err := msgr.Close()
-		log.Check(err, "closing message after delivery attempt")
-	}()
-
-	cidctx := context.WithValue(mox.Context, mlog.CidKey, cid)
-	ctx, cancel := context.WithTimeout(cidctx, 30*time.Second)
-	defer cancel()
-
-	conn, ip, dualstack, err := dialHost(ctx, log, resolver, host, m)
-	remoteIP = ip
-	cancel()
-	var result string
-	switch {
-	case err == nil:
-		result = "ok"
-	case errors.Is(err, os.ErrDeadlineExceeded), errors.Is(err, context.DeadlineExceeded):
-		result = "timeout"
-	case errors.Is(err, context.Canceled):
-		result = "canceled"
-	default:
-		result = "error"
-	}
-	metricConnection.WithLabelValues(result).Inc()
-	if err != nil {
-		log.Debugx("connecting to remote smtp", err, mlog.Field("host", host))
-		return false, false, "", ip, fmt.Sprintf("dialing smtp server: %v", err), false
-	}
-
-	var mailFrom string
-	if m.SenderLocalpart != "" || !m.SenderDomain.IsZero() {
-		mailFrom = m.Sender().XString(m.SMTPUTF8)
-	}
-	rcptTo := m.Recipient().XString(m.SMTPUTF8)
-
-	// todo future: get closer to timeouts specified in rfc? ../rfc/5321:3610
-	log = log.Fields(mlog.Field("remoteip", ip))
-	ctx, cancel = context.WithTimeout(cidctx, 30*time.Minute)
-	defer cancel()
-	mox.Connections.Register(conn, "smtpclient", "queue")
-	sc, err := smtpclient.New(ctx, log, conn, tlsMode, host.String(), "")
-	defer func() {
-		if sc == nil {
-			conn.Close()
-		} else {
-			sc.Close()
-		}
-		mox.Connections.Unregister(conn)
-	}()
-	if err == nil {
-		has8bit := m.Has8bit
-		smtputf8 := m.SMTPUTF8
-		var msg io.Reader = msgr
-		size := m.Size
-		if m.DSNUTF8 != nil && sc.Supports8BITMIME() && sc.SupportsSMTPUTF8() {
-			has8bit = true
-			smtputf8 = true
-			size = int64(len(m.DSNUTF8))
-			msg = bytes.NewReader(m.DSNUTF8)
-		}
-		err = sc.Deliver(ctx, mailFrom, rcptTo, size, msg, has8bit, smtputf8)
-	}
-	if err != nil {
-		log.Infox("delivery failed", err)
-	}
-	var cerr smtpclient.Error
-	switch {
-	case err == nil:
-		deliveryResult = "ok"
-	case errors.Is(err, os.ErrDeadlineExceeded), errors.Is(err, context.DeadlineExceeded):
-		deliveryResult = "timeout"
-	case errors.Is(err, context.Canceled):
-		deliveryResult = "canceled"
-	case errors.As(err, &cerr):
-		deliveryResult = "temperror"
-		if cerr.Permanent {
-			deliveryResult = "permerror"
-		}
-	default:
-		deliveryResult = "error"
-	}
-	if err == nil {
-		return false, false, "", ip, "", true
-	} else if cerr, ok := err.(smtpclient.Error); ok {
-		// If we are being rejected due to policy reasons on the first
-		// attempt and remote has both IPv4 and IPv6, we'll give it
-		// another try. Our first IP may be in a block list, the address for
-		// the other family perhaps is not.
-		permanent := cerr.Permanent
-		if permanent && m.Attempts == 1 && dualstack && strings.HasPrefix(cerr.Secode, "7.") {
-			permanent = false
-		}
-		return permanent, errors.Is(cerr, smtpclient.ErrTLS), cerr.Secode, ip, cerr.Error(), false
+		transportName = m.Transport
 	} else {
-		return false, errors.Is(cerr, smtpclient.ErrTLS), "", ip, err.Error(), false
+		route := findRoute(m.Attempts-1, m)
+		transport = route.ResolvedTransport
+		transportName = route.Transport
 	}
+
+	if transportName != "" {
+		qlog = qlog.Fields(mlog.Field("transport", transportName))
+		qlog.Debug("delivering with transport", mlog.Field("transport", transportName))
+	}
+
+	var dialer contextDialer = &net.Dialer{}
+	if transport.Submissions != nil {
+		deliverSubmit(cid, qlog, resolver, dialer, m, backoff, transportName, transport.Submissions, true, 465)
+	} else if transport.Submission != nil {
+		deliverSubmit(cid, qlog, resolver, dialer, m, backoff, transportName, transport.Submission, false, 587)
+	} else if transport.SMTP != nil {
+		deliverSubmit(cid, qlog, resolver, dialer, m, backoff, transportName, transport.SMTP, false, 25)
+	} else {
+		ourHostname := mox.Conf.Static.HostnameDomain
+		if transport.Socks != nil {
+			socksdialer, err := proxy.SOCKS5("tcp", transport.Socks.Address, nil, &net.Dialer{})
+			if err != nil {
+				fail(qlog, m, backoff, false, dsn.NameIP{}, "", fmt.Sprintf("socks dialer: %v", err))
+				return
+			} else if d, ok := socksdialer.(contextDialer); !ok {
+				fail(qlog, m, backoff, false, dsn.NameIP{}, "", "socks dialer is not a contextdialer")
+				return
+			} else {
+				dialer = d
+			}
+			ourHostname = transport.Socks.Hostname
+		}
+		deliverDirect(cid, qlog, resolver, dialer, ourHostname, transportName, m, backoff)
+	}
+}
+
+func findRoute(attempt int, m Msg) config.Route {
+	routesAccount, routesDomain, routesGlobal := mox.Conf.Routes(m.SenderAccount, m.SenderDomain.Domain)
+	if r, ok := findRouteInList(attempt, m, routesAccount); ok {
+		return r
+	}
+	if r, ok := findRouteInList(attempt, m, routesDomain); ok {
+		return r
+	}
+	if r, ok := findRouteInList(attempt, m, routesGlobal); ok {
+		return r
+	}
+	return config.Route{}
+}
+
+func findRouteInList(attempt int, m Msg, routes []config.Route) (config.Route, bool) {
+	for _, r := range routes {
+		if routeMatch(attempt, m, r) {
+			return r, true
+		}
+	}
+	return config.Route{}, false
+}
+
+func routeMatch(attempt int, m Msg, r config.Route) bool {
+	return attempt >= r.MinimumAttempts && routeMatchDomain(r.FromDomainASCII, m.SenderDomain.Domain) && routeMatchDomain(r.ToDomainASCII, m.RecipientDomain.Domain)
+}
+
+func routeMatchDomain(l []string, d dns.Domain) bool {
+	if len(l) == 0 {
+		return true
+	}
+	for _, e := range l {
+		if d.ASCII == e || strings.HasPrefix(e, ".") && (d.ASCII == e[1:] || strings.HasSuffix(d.ASCII, e)) {
+			return true
+		}
+	}
+	return false
 }
 
 // dialHost dials host for delivering Msg, taking previous attempts into accounts.
@@ -846,7 +626,7 @@ func deliverHost(log *mlog.Log, resolver dns.Resolver, cid int64, host dns.IPDom
 // If we have fully specified local smtp listen IPs, we set those for the outgoing
 // connection. The admin probably configured these same IPs in SPF, but others
 // possibly not.
-func dialHost(ctx context.Context, log *mlog.Log, resolver dns.Resolver, host dns.IPDomain, m *Msg) (conn net.Conn, ip net.IP, dualstack bool, rerr error) {
+func dialHost(ctx context.Context, log *mlog.Log, resolver dns.Resolver, dialer contextDialer, host dns.IPDomain, port int, m *Msg) (conn net.Conn, ip net.IP, dualstack bool, rerr error) {
 	var ips []net.IP
 	if len(host.IP) > 0 {
 		ips = []net.IP{host.IP}
@@ -906,8 +686,8 @@ func dialHost(ctx context.Context, log *mlog.Log, resolver dns.Resolver, host dn
 	var lastErr error
 	var lastIP net.IP
 	for _, ip := range ips {
-		addr := net.JoinHostPort(ip.String(), "25")
-		log.Debug("dialing remote smtp", mlog.Field("addr", addr))
+		addr := net.JoinHostPort(ip.String(), fmt.Sprintf("%d", port))
+		log.Debug("dialing remote host for delivery", mlog.Field("addr", addr))
 		var laddr net.Addr
 		for _, lip := range mox.Conf.Static.SpecifiedSMTPListenIPs {
 			ipIs4 := ip.To4() != nil
@@ -917,7 +697,7 @@ func dialHost(ctx context.Context, log *mlog.Log, resolver dns.Resolver, host dn
 				break
 			}
 		}
-		conn, err := dial(ctx, timeout, addr, laddr)
+		conn, err := dial(ctx, dialer, timeout, addr, laddr)
 		if err == nil {
 			log.Debug("connected for smtp delivery", mlog.Field("host", host), mlog.Field("addr", addr), mlog.Field("laddr", laddr))
 			if m.DialedIPs == nil {

queue/queue_test.go

@@ -3,9 +3,14 @@ package queue
 import (
 	"bufio"
 	"context"
+	"crypto/ed25519"
+	cryptorand "crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
 	"errors"
 	"fmt"
 	"io"
+	"math/big"
 	"net"
 	"os"
 	"reflect"
@@ -80,11 +85,11 @@ func TestQueue(t *testing.T) {
 	}
 
 	path := smtp.Path{Localpart: "mjl", IPDomain: dns.IPDomain{Domain: dns.Domain{ASCII: "mox.example"}}}
-	err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
+	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
 	tcheck(t, err, "add message to queue for delivery")
 
 	mf2 := prepareFile(t)
-	err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, mf2, nil, false)
+	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, mf2, nil, false)
 	tcheck(t, err, "add message to queue for delivery")
 	os.Remove(mf2.Name())
 
@@ -120,11 +125,14 @@ func TestQueue(t *testing.T) {
 
 	// Override dial function. We'll make connecting fail for now.
 	resolver := dns.MockResolver{
-		A:  map[string][]string{"mox.example.": {"127.0.0.1"}},
+		A: map[string][]string{
+			"mox.example.":        {"127.0.0.1"},
+			"submission.example.": {"127.0.0.1"},
+		},
 		MX: map[string][]*net.MX{"mox.example.": {{Host: "mox.example", Pref: 10}}},
 	}
 	dialed := make(chan struct{}, 1)
-	dial = func(ctx context.Context, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
+	dial = func(ctx context.Context, dialer contextDialer, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
 		dialed <- struct{}{}
 		return nil, fmt.Errorf("failure from test")
 	}
@@ -167,26 +175,20 @@ func TestQueue(t *testing.T) {
 		t.Fatalf("message mismatch, got %q, expected %q", string(msgbuf), testmsg)
 	}
 
-	n, err = Kick(ctxbg, msg.ID+1, "", "")
+	n, err = Kick(ctxbg, msg.ID+1, "", "", nil)
 	tcheck(t, err, "kick")
 	if n != 0 {
 		t.Fatalf("kick %d, expected 0", n)
 	}
-	n, err = Kick(ctxbg, msg.ID, "", "")
+	n, err = Kick(ctxbg, msg.ID, "", "", nil)
 	tcheck(t, err, "kick")
 	if n != 1 {
 		t.Fatalf("kicked %d, expected 1", n)
 	}
 
-	// Setting up a pipe. We'll start a fake smtp server on the server-side. And return the
-	// client-side to the invocation dial, for the attempted delivery from the queue.
-	// The delivery should succeed.
-	server, client := net.Pipe()
-	defer server.Close()
-	defer client.Close()
-
 	smtpdone := make(chan struct{})
-	go func() {
+
+	fakeSMTPServer := func(server net.Conn) {
 		// We do a minimal fake smtp server. We cannot import smtpserver.Serve due to cyclic dependencies.
 		fmt.Fprintf(server, "220 mox.example\r\n")
 		br := bufio.NewReader(server)
@@ -205,42 +207,139 @@ func TestQueue(t *testing.T) {
 		fmt.Fprintf(server, "221 ok\r\n")
 
 		smtpdone <- struct{}{}
-	}()
-
-	dial = func(ctx context.Context, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
-		dialed <- struct{}{}
-		return client, nil
 	}
-	launchWork(resolver, map[string]struct{}{})
 
-	timer.Reset(time.Second)
-	select {
-	case <-dialed:
-		select {
-		case <-smtpdone:
-			i := 0
-			for {
-				xmsgs, err := List(ctxbg)
-				tcheck(t, err, "list queue")
-				if len(xmsgs) == 0 {
-					break
-				}
-				i++
-				if i == 10 {
-					t.Fatalf("%d messages in queue, expected 0", len(xmsgs))
-				}
-				time.Sleep(100 * time.Millisecond)
-			}
-		case <-timer.C:
-			t.Fatalf("no deliver within 1s")
+	fakeSubmitServer := func(server net.Conn) {
+		// We do a minimal fake smtp server. We cannot import smtpserver.Serve due to cyclic dependencies.
+		fmt.Fprintf(server, "220 mox.example\r\n")
+		br := bufio.NewReader(server)
+		br.ReadString('\n') // Should be EHLO.
+		fmt.Fprintf(server, "250-localhost\r\n")
+		fmt.Fprintf(server, "250 AUTH PLAIN\r\n")
+		br.ReadString('\n') // Should be AUTH PLAIN
+		fmt.Fprintf(server, "235 2.7.0 auth ok\r\n")
+		br.ReadString('\n') // Should be MAIL FROM.
+		fmt.Fprintf(server, "250 ok\r\n")
+		br.ReadString('\n') // Should be RCPT TO.
+		fmt.Fprintf(server, "250 ok\r\n")
+		br.ReadString('\n') // Should be DATA.
+		fmt.Fprintf(server, "354 continue\r\n")
+		reader := smtp.NewDataReader(br)
+		io.Copy(io.Discard, reader)
+		fmt.Fprintf(server, "250 ok\r\n")
+		br.ReadString('\n') // Should be QUIT.
+		fmt.Fprintf(server, "221 ok\r\n")
+
+		smtpdone <- struct{}{}
+	}
+
+	testDeliver := func(fakeServer func(conn net.Conn)) bool {
+		t.Helper()
+
+		// Setting up a pipe. We'll start a fake smtp server on the server-side. And return the
+		// client-side to the invocation dial, for the attempted delivery from the queue.
+		// The delivery should succeed.
+		server, client := net.Pipe()
+		defer server.Close()
+		defer client.Close()
+
+		var wasNetDialer bool
+		dial = func(ctx context.Context, dialer contextDialer, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
+			_, wasNetDialer = dialer.(*net.Dialer)
+			dialed <- struct{}{}
+			return client, nil
 		}
-	case <-timer.C:
-		t.Fatalf("no dial within 1s")
+
+		waitDeliver := func() {
+			t.Helper()
+			timer.Reset(time.Second)
+			select {
+			case <-dialed:
+				select {
+				case <-smtpdone:
+					i := 0
+					for {
+						xmsgs, err := List(ctxbg)
+						tcheck(t, err, "list queue")
+						if len(xmsgs) == 0 {
+							break
+						}
+						i++
+						if i == 10 {
+							t.Fatalf("%d messages in queue, expected 0", len(xmsgs))
+						}
+						time.Sleep(100 * time.Millisecond)
+					}
+				case <-timer.C:
+					t.Fatalf("no deliver within 1s")
+				}
+			case <-timer.C:
+				t.Fatalf("no dial within 1s")
+			}
+			<-deliveryResult // Deliver sends here.
+		}
+
+		go fakeServer(server)
+		launchWork(resolver, map[string]struct{}{})
+		waitDeliver()
+		return wasNetDialer
+	}
+
+	// Test direct delivery.
+	wasNetDialer := testDeliver(fakeSMTPServer)
+	if !wasNetDialer {
+		t.Fatalf("expected net.Dialer as dialer")
+	}
+
+	// Add a message to be delivered with submit because of its route.
+	topath := smtp.Path{Localpart: "mjl", IPDomain: dns.IPDomain{Domain: dns.Domain{ASCII: "submit.example"}}}
+	_, err = Add(ctxbg, xlog, "mjl", path, topath, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
+	tcheck(t, err, "add message to queue for delivery")
+	wasNetDialer = testDeliver(fakeSubmitServer)
+	if !wasNetDialer {
+		t.Fatalf("expected net.Dialer as dialer")
+	}
+
+	// Add a message to be delivered with submit because of explicitly configured transport, that uses TLS.
+	msgID, err := Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
+	tcheck(t, err, "add message to queue for delivery")
+	transportSubmitTLS := "submittls"
+	n, err = Kick(ctxbg, msgID, "", "", &transportSubmitTLS)
+	tcheck(t, err, "kick queue")
+	if n != 1 {
+		t.Fatalf("kick changed %d messages, expected 1", n)
+	}
+	// Make fake cert, and make it trusted.
+	cert := fakeCert(t, "submission.example", false)
+	mox.Conf.Static.TLS.CertPool = x509.NewCertPool()
+	mox.Conf.Static.TLS.CertPool.AddCert(cert.Leaf)
+	tlsConfig := tls.Config{
+		Certificates: []tls.Certificate{cert},
+	}
+	wasNetDialer = testDeliver(func(conn net.Conn) {
+		conn = tls.Server(conn, &tlsConfig)
+		fakeSubmitServer(conn)
+	})
+	if !wasNetDialer {
+		t.Fatalf("expected net.Dialer as dialer")
+	}
+
+	// Add a message to be delivered with socks.
+	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
+	tcheck(t, err, "add message to queue for delivery")
+	transportSocks := "socks"
+	n, err = Kick(ctxbg, msgID, "", "", &transportSocks)
+	tcheck(t, err, "kick queue")
+	if n != 1 {
+		t.Fatalf("kick changed %d messages, expected 1", n)
+	}
+	wasNetDialer = testDeliver(fakeSMTPServer)
+	if wasNetDialer {
+		t.Fatalf("expected non-net.Dialer as dialer") // SOCKS5 dialer is a private type, we cannot check for it.
 	}
-	<-deliveryResult // Deliver sends here.
 
 	// Add another message that we'll fail to deliver entirely.
-	err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
+	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
 	tcheck(t, err, "add message to queue for delivery")
 
 	msgs, err = List(ctxbg)
@@ -270,7 +369,7 @@ func TestQueue(t *testing.T) {
 	}()
 
 	seq := 0
-	dial = func(ctx context.Context, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
+	dial = func(ctx context.Context, dialer contextDialer, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
 		seq++
 		switch seq {
 		default:
@@ -337,7 +436,7 @@ func TestQueueStart(t *testing.T) {
 		MX: map[string][]*net.MX{"mox.example.": {{Host: "mox.example", Pref: 10}}},
 	}
 	dialed := make(chan struct{}, 1)
-	dial = func(ctx context.Context, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
+	dial = func(ctx context.Context, dialer contextDialer, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
 		dialed <- struct{}{}
 		return nil, fmt.Errorf("failure from test")
 	}
@@ -374,7 +473,7 @@ func TestQueueStart(t *testing.T) {
 	}
 
 	path := smtp.Path{Localpart: "mjl", IPDomain: dns.IPDomain{Domain: dns.Domain{ASCII: "mox.example"}}}
-	err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
+	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), nil, prepareFile(t), nil, true)
 	tcheck(t, err, "add message to queue for delivery")
 	checkDialed(true)
 
@@ -383,7 +482,7 @@ func TestQueueStart(t *testing.T) {
 	checkDialed(false)
 
 	// Kick for real, should see another attempt.
-	n, err := Kick(ctxbg, 0, "mox.example", "")
+	n, err := Kick(ctxbg, 0, "mox.example", "", nil)
 	tcheck(t, err, "kick queue")
 	if n != 1 {
 		t.Fatalf("kick changed %d messages, expected 1", n)
@@ -520,7 +619,7 @@ func TestDialHost(t *testing.T) {
 		},
 	}
 
-	dial = func(ctx context.Context, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
+	dial = func(ctx context.Context, dialer contextDialer, timeout time.Duration, addr string, laddr net.Addr) (net.Conn, error) {
 		return nil, nil // No error, nil connection isn't used.
 	}
 
@@ -529,12 +628,44 @@ func TestDialHost(t *testing.T) {
 	}
 
 	m := Msg{DialedIPs: map[string][]net.IP{}}
-	_, ip, dualstack, err := dialHost(ctxbg, xlog, resolver, ipdomain("dualstack.example"), &m)
+	_, ip, dualstack, err := dialHost(ctxbg, xlog, resolver, nil, ipdomain("dualstack.example"), 25, &m)
 	if err != nil || ip.String() != "10.0.0.1" || !dualstack {
 		t.Fatalf("expected err nil, address 10.0.0.1, dualstack true, got %v %v %v", err, ip, dualstack)
 	}
-	_, ip, dualstack, err = dialHost(ctxbg, xlog, resolver, ipdomain("dualstack.example"), &m)
+	_, ip, dualstack, err = dialHost(ctxbg, xlog, resolver, nil, ipdomain("dualstack.example"), 25, &m)
 	if err != nil || ip.String() != "2001:db8::1" || !dualstack {
 		t.Fatalf("expected err nil, address 2001:db8::1, dualstack true, got %v %v %v", err, ip, dualstack)
 	}
 }
+
+// Just a cert that appears valid.
+func fakeCert(t *testing.T, name string, expired bool) tls.Certificate {
+	notAfter := time.Now()
+	if expired {
+		notAfter = notAfter.Add(-time.Hour)
+	} else {
+		notAfter = notAfter.Add(time.Hour)
+	}
+
+	privKey := ed25519.NewKeyFromSeed(make([]byte, ed25519.SeedSize)) // Fake key, don't use this for real!
+	template := &x509.Certificate{
+		SerialNumber: big.NewInt(1), // Required field...
+		DNSNames:     []string{name},
+		NotBefore:    time.Now().Add(-time.Hour),
+		NotAfter:     notAfter,
+	}
+	localCertBuf, err := x509.CreateCertificate(cryptorand.Reader, template, template, privKey.Public(), privKey)
+	if err != nil {
+		t.Fatalf("making certificate: %s", err)
+	}
+	cert, err := x509.ParseCertificate(localCertBuf)
+	if err != nil {
+		t.Fatalf("parsing generated certificate: %s", err)
+	}
+	c := tls.Certificate{
+		Certificate: [][]byte{localCertBuf},
+		PrivateKey:  privKey,
+		Leaf:        cert,
+	}
+	return c
+}

queue/submit.go

@@ -0,0 +1,191 @@
+package queue
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"time"
+
+	"github.com/mjl-/mox/config"
+	"github.com/mjl-/mox/dns"
+	"github.com/mjl-/mox/dsn"
+	"github.com/mjl-/mox/mlog"
+	"github.com/mjl-/mox/mox-"
+	"github.com/mjl-/mox/sasl"
+	"github.com/mjl-/mox/smtpclient"
+	"github.com/mjl-/mox/store"
+)
+
+// todo: reuse connection? do fewer concurrently (other than with direct delivery).
+
+// deliver via another SMTP server, e.g. relaying to a smart host, possibly
+// with authentication (submission).
+func deliverSubmit(cid int64, qlog *mlog.Log, resolver dns.Resolver, dialer contextDialer, m Msg, backoff time.Duration, transportName string, transport *config.TransportSMTP, dialTLS bool, defaultPort int) {
+	// todo: configurable timeouts
+
+	port := transport.Port
+	if port == 0 {
+		port = defaultPort
+	}
+
+	tlsMode := smtpclient.TLSStrictStartTLS
+	if dialTLS {
+		tlsMode = smtpclient.TLSStrictImmediate
+	} else if transport.STARTTLSInsecureSkipVerify {
+		tlsMode = smtpclient.TLSOpportunistic
+	} else if transport.NoSTARTTLS {
+		tlsMode = smtpclient.TLSSkip
+	}
+	start := time.Now()
+	var deliveryResult string
+	var permanent bool
+	var secodeOpt string
+	var errmsg string
+	var success bool
+	defer func() {
+		metricDelivery.WithLabelValues(fmt.Sprintf("%d", m.Attempts), transportName, string(tlsMode), deliveryResult).Observe(float64(time.Since(start)) / float64(time.Second))
+		qlog.Debug("queue deliversubmit result", mlog.Field("host", transport.DNSHost), mlog.Field("port", port), mlog.Field("attempt", m.Attempts), mlog.Field("permanent", permanent), mlog.Field("secodeopt", secodeOpt), mlog.Field("errmsg", errmsg), mlog.Field("ok", success), mlog.Field("duration", time.Since(start)))
+	}()
+
+	dialctx, dialcancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer dialcancel()
+	addr := net.JoinHostPort(transport.Host, fmt.Sprintf("%d", port))
+	conn, _, _, err := dialHost(dialctx, qlog, resolver, dialer, dns.IPDomain{Domain: transport.DNSHost}, port, &m)
+	var result string
+	switch {
+	case err == nil:
+		result = "ok"
+	case errors.Is(err, os.ErrDeadlineExceeded), errors.Is(err, context.DeadlineExceeded):
+		result = "timeout"
+	case errors.Is(err, context.Canceled):
+		result = "canceled"
+	default:
+		result = "error"
+	}
+	metricConnection.WithLabelValues(result).Inc()
+	if err != nil {
+		if conn != nil {
+			err := conn.Close()
+			qlog.Check(err, "closing connection")
+		}
+		qlog.Errorx("dialing for submission", err, mlog.Field("remote", addr))
+		errmsg = fmt.Sprintf("transport %s: dialing %s for submission: %v", transportName, addr, err)
+		fail(qlog, m, backoff, false, dsn.NameIP{}, "", errmsg)
+		return
+	}
+	dialcancel()
+
+	var auth []sasl.Client
+	if transport.Auth != nil {
+		a := transport.Auth
+		for _, mech := range a.EffectiveMechanisms {
+			switch mech {
+			case "PLAIN":
+				auth = append(auth, sasl.NewClientPlain(a.Username, a.Password))
+			case "CRAM-MD5":
+				auth = append(auth, sasl.NewClientCRAMMD5(a.Username, a.Password))
+			case "SCRAM-SHA-1":
+				auth = append(auth, sasl.NewClientSCRAMSHA1(a.Username, a.Password))
+			case "SCRAM-SHA-256":
+				auth = append(auth, sasl.NewClientSCRAMSHA256(a.Username, a.Password))
+			default:
+				// Should not happen.
+				qlog.Error("missing smtp authentication mechanisms implementation", mlog.Field("mechanism", mech))
+				errmsg = fmt.Sprintf("transport %s: authentication mechanisms %q not implemented", transportName, mech)
+				fail(qlog, m, backoff, false, dsn.NameIP{}, "", errmsg)
+				return
+			}
+		}
+	}
+	clientctx, clientcancel := context.WithTimeout(context.Background(), 60*time.Second)
+	defer clientcancel()
+	client, err := smtpclient.New(clientctx, qlog, conn, tlsMode, mox.Conf.Static.HostnameDomain, transport.DNSHost, auth)
+	if err != nil {
+		smtperr, ok := err.(smtpclient.Error)
+		var remoteMTA dsn.NameIP
+		if ok {
+			remoteMTA.Name = transport.Host
+		}
+		qlog.Errorx("establishing smtp session for submission", err, mlog.Field("remote", addr))
+		errmsg = fmt.Sprintf("transport %s: establishing smtp session with %s for submission: %v", transportName, addr, err)
+		secodeOpt = smtperr.Secode
+		fail(qlog, m, backoff, false, remoteMTA, secodeOpt, errmsg)
+		return
+	}
+	defer func() {
+		err := client.Close()
+		qlog.Check(err, "closing smtp client after delivery")
+	}()
+	clientcancel()
+
+	var msgr io.ReadCloser
+	var size int64
+	var req8bit, reqsmtputf8 bool
+	if len(m.DSNUTF8) > 0 && client.SupportsSMTPUTF8() {
+		msgr = io.NopCloser(bytes.NewReader(m.DSNUTF8))
+		reqsmtputf8 = true
+		size = int64(len(m.DSNUTF8))
+	} else {
+		req8bit = m.Has8bit // todo: not require this, but just try to submit?
+		size = m.Size
+
+		p := m.MessagePath()
+		f, err := os.Open(p)
+		if err != nil {
+			qlog.Errorx("opening message for delivery", err, mlog.Field("remote", addr), mlog.Field("path", p))
+			errmsg = fmt.Sprintf("transport %s: opening message file for submission: %v", transportName, err)
+			fail(qlog, m, backoff, false, dsn.NameIP{}, "", errmsg)
+			return
+		}
+		msgr = store.FileMsgReader(m.MsgPrefix, f)
+		defer func() {
+			err := msgr.Close()
+			qlog.Check(err, "closing message after delivery attempt")
+		}()
+	}
+
+	deliverctx, delivercancel := context.WithTimeout(context.Background(), time.Duration(60+size/(1024*1024))*time.Second)
+	defer delivercancel()
+	err = client.Deliver(deliverctx, m.Sender().String(), m.Recipient().String(), size, msgr, req8bit, reqsmtputf8)
+	if err != nil {
+		qlog.Infox("delivery failed", err)
+	}
+	var cerr smtpclient.Error
+	switch {
+	case err == nil:
+		deliveryResult = "ok"
+		success = true
+	case errors.Is(err, os.ErrDeadlineExceeded), errors.Is(err, context.DeadlineExceeded):
+		deliveryResult = "timeout"
+	case errors.Is(err, context.Canceled):
+		deliveryResult = "canceled"
+	case errors.As(err, &cerr):
+		deliveryResult = "temperror"
+		if cerr.Permanent {
+			deliveryResult = "permerror"
+		}
+	default:
+		deliveryResult = "error"
+	}
+	if err != nil {
+		smtperr, ok := err.(smtpclient.Error)
+		var remoteMTA dsn.NameIP
+		if ok {
+			remoteMTA.Name = transport.Host
+		}
+		qlog.Errorx("submitting email", err, mlog.Field("remote", addr))
+		permanent = smtperr.Permanent
+		secodeOpt = smtperr.Secode
+		errmsg = fmt.Sprintf("transport %s: submitting email to %s: %v", transportName, addr, err)
+		fail(qlog, m, backoff, permanent, remoteMTA, secodeOpt, errmsg)
+		return
+	}
+	qlog.Info("delivered from queue with transport")
+	if err := queueDelete(context.Background(), m.ID); err != nil {
+		qlog.Errorx("deleting message from queue after delivery", err)
+	}
+}

quickstart_test.go

@@ -5,18 +5,18 @@
 package main
 
 import (
-	"bytes"
 	"crypto/tls"
-	"encoding/base64"
 	"fmt"
 	"os"
 	"strings"
 	"testing"
 	"time"
 
+	"github.com/mjl-/mox/dns"
 	"github.com/mjl-/mox/imapclient"
 	"github.com/mjl-/mox/mlog"
 	"github.com/mjl-/mox/mox-"
+	"github.com/mjl-/mox/sasl"
 	"github.com/mjl-/mox/smtpclient"
 )
 
@@ -32,10 +32,10 @@ func tcheck(t *testing.T, err error, msg string) {
 func TestDeliver(t *testing.T) {
 	mlog.Logfmt = true
 
-	// smtpclient uses the hostname for outgoing connections.
-	var err error
-	mox.Conf.Static.HostnameDomain.ASCII, err = os.Hostname()
+	hostname, err := os.Hostname()
 	tcheck(t, err, "hostname")
+	ourHostname, err := dns.ParseDomain(hostname)
+	tcheck(t, err, "parse hostname")
 
 	// Deliver submits a message over submissions, and checks with imap idle if the
 	// message is received by the destination mail server.
@@ -120,9 +120,8 @@ Subject: test message
 This is the message.
 `, mailfrom, rcptto)
 		msg = strings.ReplaceAll(msg, "\n", "\r\n")
-		auth := bytes.Join([][]byte{nil, []byte(mailfrom), []byte(password)}, []byte{0})
-		authLine := fmt.Sprintf("AUTH PLAIN %s", base64.StdEncoding.EncodeToString(auth))
-		c, err := smtpclient.New(mox.Context, xlog, conn, smtpclient.TLSSkip, desthost, authLine)
+		auth := []sasl.Client{sasl.NewClientPlain(mailfrom, password)}
+		c, err := smtpclient.New(mox.Context, xlog, conn, smtpclient.TLSSkip, ourHostname, dns.Domain{ASCII: desthost}, auth)
 		tcheck(t, err, "smtp hello")
 		err = c.Deliver(mox.Context, mailfrom, rcptto, int64(len(msg)), strings.NewReader(msg), false, false)
 		tcheck(t, err, "deliver with smtp")

sasl/sasl.go

@@ -0,0 +1,176 @@
+// Package SASL implements Simple Authentication and Security Layer, RFC 4422.
+package sasl
+
+import (
+	"crypto/md5"
+	"crypto/sha1"
+	"crypto/sha256"
+	"fmt"
+	"hash"
+	"strings"
+
+	"github.com/mjl-/mox/scram"
+)
+
+// Client is a SASL client
+type Client interface {
+	// Name as used in SMTP AUTH, e.g. PLAIN, CRAM-MD5, SCRAM-SHA-256.
+	// cleartextCredentials indicates if credentials are exchanged in clear text, which influences whether they are logged.
+	Info() (name string, cleartextCredentials bool)
+
+	// Next is called for each step of the SASL communication. The first call has a nil
+	// fromServer and serves to get a possible "initial response" from the client. If
+	// the client sends its final message it indicates so with last. Returning an error
+	// aborts the authentication attempt.
+	// For the first toServer ("initial response"), a nil toServer indicates there is
+	// no data, which is different from a non-nil zero-length toServer.
+	Next(fromServer []byte) (toServer []byte, last bool, err error)
+}
+
+type clientPlain struct {
+	Username, Password string
+	step               int
+}
+
+var _ Client = (*clientPlain)(nil)
+
+// NewClientPlain returns a client for SASL PLAIN authentication.
+func NewClientPlain(username, password string) Client {
+	return &clientPlain{username, password, 0}
+}
+
+func (a *clientPlain) Info() (name string, hasCleartextCredentials bool) {
+	return "PLAIN", true
+}
+
+func (a *clientPlain) Next(fromServer []byte) (toServer []byte, last bool, rerr error) {
+	defer func() { a.step++ }()
+	switch a.step {
+	case 0:
+		return []byte(fmt.Sprintf("\u0000%s\u0000%s", a.Username, a.Password)), true, nil
+	default:
+		return nil, false, fmt.Errorf("invalid step %d", a.step)
+	}
+}
+
+type clientCRAMMD5 struct {
+	Username, Password string
+	step               int
+}
+
+var _ Client = (*clientCRAMMD5)(nil)
+
+// NewClientCRAMMD5 returns a client for SASL CRAM-MD5 authentication.
+func NewClientCRAMMD5(username, password string) Client {
+	return &clientCRAMMD5{username, password, 0}
+}
+
+func (a *clientCRAMMD5) Info() (name string, hasCleartextCredentials bool) {
+	return "CRAM-MD5", false
+}
+
+func (a *clientCRAMMD5) Next(fromServer []byte) (toServer []byte, last bool, rerr error) {
+	defer func() { a.step++ }()
+	switch a.step {
+	case 0:
+		return nil, false, nil
+	case 1:
+		// Validate the challenge.
+		// ../rfc/2195:82
+		s := string(fromServer)
+		if !strings.HasPrefix(s, "<") || !strings.HasSuffix(s, ">") {
+			return nil, false, fmt.Errorf("invalid challenge, missing angle brackets")
+		}
+		t := strings.SplitN(s, ".", 2)
+		if len(t) != 2 || t[0] == "" {
+			return nil, false, fmt.Errorf("invalid challenge, missing dot or random digits")
+		}
+		t = strings.Split(t[1], "@")
+		if len(t) == 1 || t[0] == "" || t[len(t)-1] == "" {
+			return nil, false, fmt.Errorf("invalid challenge, empty timestamp or empty hostname")
+		}
+
+		// ../rfc/2195:138
+		key := []byte(a.Password)
+		if len(key) > 64 {
+			t := md5.Sum(key)
+			key = t[:]
+		}
+		ipad := make([]byte, md5.BlockSize)
+		opad := make([]byte, md5.BlockSize)
+		copy(ipad, key)
+		copy(opad, key)
+		for i := range ipad {
+			ipad[i] ^= 0x36
+			opad[i] ^= 0x5c
+		}
+		ipadh := md5.New()
+		ipadh.Write(ipad)
+		ipadh.Write([]byte(fromServer))
+
+		opadh := md5.New()
+		opadh.Write(opad)
+		opadh.Write(ipadh.Sum(nil))
+
+		// ../rfc/2195:88
+		return []byte(fmt.Sprintf("%s %x", a.Username, opadh.Sum(nil))), true, nil
+
+	default:
+		return nil, false, fmt.Errorf("invalid step %d", a.step)
+	}
+}
+
+type clientSCRAMSHA struct {
+	Username, Password string
+
+	name  string
+	step  int
+	scram *scram.Client
+}
+
+var _ Client = (*clientSCRAMSHA)(nil)
+
+// NewClientSCRAMSHA1 returns a client for SASL SCRAM-SHA-1 authentication.
+func NewClientSCRAMSHA1(username, password string) Client {
+	return &clientSCRAMSHA{username, password, "SCRAM-SHA-1", 0, nil}
+}
+
+// NewClientSCRAMSHA256 returns a client for SASL SCRAM-SHA-256 authentication.
+func NewClientSCRAMSHA256(username, password string) Client {
+	return &clientSCRAMSHA{username, password, "SCRAM-SHA-256", 0, nil}
+}
+
+func (a *clientSCRAMSHA) Info() (name string, hasCleartextCredentials bool) {
+	return a.name, false
+}
+
+func (a *clientSCRAMSHA) Next(fromServer []byte) (toServer []byte, last bool, rerr error) {
+	defer func() { a.step++ }()
+	switch a.step {
+	case 0:
+		var h func() hash.Hash
+		switch a.name {
+		case "SCRAM-SHA-1":
+			h = sha1.New
+		case "SCRAM-SHA-256":
+			h = sha256.New
+		default:
+			return nil, false, fmt.Errorf("invalid SCRAM-SHA variant %q", a.name)
+		}
+
+		a.scram = scram.NewClient(h, a.Username, "")
+		toserver, err := a.scram.ClientFirst()
+		return []byte(toserver), false, err
+
+	case 1:
+		clientFinal, err := a.scram.ServerFirst(fromServer, a.Password)
+		return []byte(clientFinal), false, err
+
+	case 2:
+		err := a.scram.ServerFinal(fromServer)
+		return nil, true, err
+
+	default:
+		return nil, false, fmt.Errorf("invalid step %d", a.step)
+	}
+}

sendmail.go

@@ -3,8 +3,6 @@ package main
 import (
 	"bufio"
 	"context"
-	"crypto/tls"
-	"encoding/base64"
 	"fmt"
 	"io"
 	"log"
@@ -17,12 +15,37 @@ import (
 
 	"github.com/mjl-/sconf"
 
+	"github.com/mjl-/mox/dns"
 	"github.com/mjl-/mox/mlog"
-	"github.com/mjl-/mox/mox-"
+	"github.com/mjl-/mox/sasl"
 	"github.com/mjl-/mox/smtp"
 	"github.com/mjl-/mox/smtpclient"
 )
 
+var submitconf struct {
+	LocalHostname      string `sconf-doc:"Hosts don't always have an FQDN, set it explicitly, for EHLO."`
+	Host               string `sconf-doc:"Host to dial for delivery, e.g. mail.<domain>."`
+	Port               int    `sconf-doc:"Port to dial for delivery, e.g. 465 for submissions, 587 for submission, or perhaps 25 for smtp."`
+	TLS                bool   `sconf-doc:"Connect with TLS. Usually for connections to port 465."`
+	STARTTLS           bool   `sconf-doc:"After starting in plain text, use STARTTLS to enable TLS. For port 587 and 25."`
+	Username           string `sconf-doc:"For SMTP authentication."`
+	Password           string `sconf-doc:"For password-based SMTP authentication, e.g. SCRAM-SHA-256, SCRAM-SHA-1, CRAM-MD5, PLAIN."`
+	AuthMethod         string `sconf-doc:"If set, only attempt this authentication mechanism. E.g. SCRAM-SHA-256. If not set, any mutually supported algorithm can be used, in order of most to least secure."`
+	From               string `sconf-doc:"Address for MAIL FROM in SMTP and From-header in message."`
+	DefaultDestination string `sconf:"optional" sconf-doc:"Used when specified address does not contain an @ and may be a local user (eg root)."`
+}
+
+func cmdConfigDescribeSendmail(c *cmd) {
+	c.params = ">/etc/moxsubmit.conf"
+	c.help = `Describe configuration for mox when invoked as sendmail.`
+	if len(c.Parse()) != 0 {
+		c.Usage()
+	}
+
+	err := sconf.Describe(os.Stdout, submitconf)
+	xcheckf(err, "describe config")
+}
+
 func cmdSendmail(c *cmd) {
 	c.params = "[-Fname] [ignoredflags] [-t] [<message]"
 	c.help = `Sendmail is a drop-in replacement for /usr/sbin/sendmail to deliver emails sent by unix processes like cron.
@@ -213,26 +236,49 @@ binary should be setgid that group:
 		os.Exit(1)
 	}
 
-	var conn net.Conn
 	addr := net.JoinHostPort(submitconf.Host, fmt.Sprintf("%d", submitconf.Port))
 	d := net.Dialer{Timeout: 30 * time.Second}
-	if submitconf.TLS {
-		conn, err = tls.DialWithDialer(&d, "tcp", addr, nil)
-	} else {
-		conn, err = d.Dial("tcp", addr)
-	}
+	conn, err := d.Dial("tcp", addr)
 	xcheckf(err, "dial submit server")
 
+	var auth []sasl.Client
+	switch submitconf.AuthMethod {
+	case "SCRAM-SHA-256":
+		auth = []sasl.Client{sasl.NewClientSCRAMSHA256(submitconf.Username, submitconf.Password)}
+	case "SCRAM-SHA-1":
+		auth = []sasl.Client{sasl.NewClientSCRAMSHA1(submitconf.Username, submitconf.Password)}
+	case "CRAM-MD5":
+		auth = []sasl.Client{sasl.NewClientCRAMMD5(submitconf.Username, submitconf.Password)}
+	case "PLAIN":
+		auth = []sasl.Client{sasl.NewClientPlain(submitconf.Username, submitconf.Password)}
+	default:
+		auth = []sasl.Client{
+			sasl.NewClientSCRAMSHA256(submitconf.Username, submitconf.Password),
+			sasl.NewClientSCRAMSHA1(submitconf.Username, submitconf.Password),
+			sasl.NewClientCRAMMD5(submitconf.Username, submitconf.Password),
+			sasl.NewClientPlain(submitconf.Username, submitconf.Password),
+		}
+	}
+
 	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
 	defer cancel()
-	tlsMode := smtpclient.TLSStrict
-	if !submitconf.STARTTLS {
-		tlsMode = smtpclient.TLSSkip
+	tlsMode := smtpclient.TLSSkip
+	if submitconf.TLS {
+		tlsMode = smtpclient.TLSStrictImmediate
+	} else if submitconf.STARTTLS {
+		tlsMode = smtpclient.TLSStrictStartTLS
 	}
-	// todo: should have more auth options, scram-sha-256 at least, perhaps cram-md5 for compatibility as well.
-	authLine := fmt.Sprintf("AUTH PLAIN %s", base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("\u0000%s\u0000%s", submitconf.Username, submitconf.Password))))
-	mox.Conf.Static.HostnameDomain.ASCII = submitconf.LocalHostname
-	client, err := smtpclient.New(ctx, mlog.New("sendmail"), conn, tlsMode, submitconf.Host, authLine)
+
+	ourHostname, err := dns.ParseDomain(submitconf.LocalHostname)
+	xcheckf(err, "parsing our local hostname")
+
+	var remoteHostname dns.Domain
+	if net.ParseIP(submitconf.Host) != nil {
+		remoteHostname, err = dns.ParseDomain(submitconf.Host)
+		xcheckf(err, "parsing remote hostname")
+	}
+
+	client, err := smtpclient.New(ctx, mlog.New("sendmail"), conn, tlsMode, ourHostname, remoteHostname, auth)
 	xcheckf(err, "open smtp session")
 
 	err = client.Deliver(ctx, submitconf.From, recipient, int64(len(msg)), strings.NewReader(msg), true, false)

serve.go

@@ -80,7 +80,7 @@ func monitorDNSBL(log *mlog.Log) {
 		time.Sleep(sleep)
 		sleep = 3 * time.Hour
 
-		ips, err := mox.IPs(mox.Context)
+		ips, err := mox.IPs(mox.Context, false)
 		if err != nil {
 			log.Errorx("listing ips for dnsbl monitor", err)
 			continue

smtpclient/client.go

@@ -5,6 +5,7 @@ import (
 	"bufio"
 	"context"
 	"crypto/tls"
+	"encoding/base64"
 	"errors"
 	"fmt"
 	"io"
@@ -16,10 +17,12 @@ import (
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promauto"
 
+	"github.com/mjl-/mox/dns"
 	"github.com/mjl-/mox/metrics"
 	"github.com/mjl-/mox/mlog"
 	"github.com/mjl-/mox/mox-"
 	"github.com/mjl-/mox/moxio"
+	"github.com/mjl-/mox/sasl"
 	"github.com/mjl-/mox/smtp"
 )
 
@@ -55,8 +58,11 @@ var (
 type TLSMode string
 
 const (
-	// TLS with validated certificate is required: matching name, not expired, trusted by CA.
-	TLSStrict TLSMode = "strict"
+	// TLS with STARTTLS for MX SMTP servers, with validated certificate is required: matching name, not expired, trusted by CA.
+	TLSStrictStartTLS TLSMode = "strictstarttls"
+
+	// TLS immediately ("implicit TLS"), with validated certificate is required: matching name, not expired, trusted by CA.
+	TLSStrictImmediate TLSMode = "strictimmediate"
 
 	// Use TLS if remote claims to support it, but do not validate the certificate
 	// (not trusted by CA, different host name or expired certificate is accepted).
@@ -89,13 +95,14 @@ type Client struct {
 	botched  bool // If set, protocol is out of sync and no further commands can be sent.
 	needRset bool // If set, a new delivery requires an RSET command.
 
-	extEcodes     bool // Remote server supports sending extended error codes.
-	extStartTLS   bool // Remote server supports STARTTLS.
-	ext8bitmime   bool
-	extSize       bool  // Remote server supports SIZE parameter.
-	maxSize       int64 // Max size of email message.
-	extPipelining bool  // Remote server supports command pipelining.
-	extSMTPUTF8   bool  // Remote server supports SMTPUTF8 extension.
+	extEcodes         bool // Remote server supports sending extended error codes.
+	extStartTLS       bool // Remote server supports STARTTLS.
+	ext8bitmime       bool
+	extSize           bool     // Remote server supports SIZE parameter.
+	maxSize           int64    // Max size of email message.
+	extPipelining     bool     // Remote server supports command pipelining.
+	extSMTPUTF8       bool     // Remote server supports SMTPUTF8 extension.
+	extAuthMechanisms []string // Supported authentication mechanisms.
 }
 
 // Error represents a failure to deliver a message.
@@ -146,11 +153,11 @@ func (e Error) Error() string {
 // New initializes an SMTP session on the given connection, returning a client that
 // can be used to deliver messages.
 //
-// New reads the server greeting, identifies itself with a HELO or EHLO command,
-// initializes TLS if remote supports it and optionally authenticates. If
-// successful, a client is returned on which eventually Close must be called.
-// Otherwise an error is returned and the caller is responsible for closing the
-// connection.
+// New optionally starts TLS (for submission), reads the server greeting,
+// identifies itself with a HELO or EHLO command, initializes TLS with STARTTLS if
+// remote supports it and optionally authenticates. If successful, a client is
+// returned on which eventually Close must be called. Otherwise an error is
+// returned and the caller is responsible for closing the connection.
 //
 // Connecting to the correct host is outside the scope of the client. The queue
 // managing outgoing messages decides which host to deliver to, taking multiple MX
@@ -159,18 +166,17 @@ func (e Error) Error() string {
 //
 // tlsMode indicates if TLS is required, optional or should not be used. A
 // certificate is only validated (trusted, match remoteHostname and not expired)
-// for tls mode "required". By default, SMTP does not verify TLS for interopability
-// reasons, but MTA-STS or DANE can require it. If opportunistic TLS is used, and a
-// TLS error is encountered, the caller may want to try again (on a new connection)
-// without TLS.
+// for the strict tls modes. By default, SMTP does not verify TLS for
+// interopability reasons, but MTA-STS or DANE can require it. If opportunistic TLS
+// is used, and a TLS error is encountered, the caller may want to try again (on a
+// new connection) without TLS.
 //
-// If auth is non-empty, it is executed as a command after SMTP greeting/EHLO
-// initialization, before starting delivery. For authenticating to a submission
-// service with AUTH PLAIN, only meant for testing.
-func New(ctx context.Context, log *mlog.Log, conn net.Conn, tlsMode TLSMode, remoteHostname, auth string) (*Client, error) {
+// If auth is non-empty, authentication will be done with the first algorithm
+// supported by the server. If none of the algorithms are supported, an error is
+// returned.
+func New(ctx context.Context, log *mlog.Log, conn net.Conn, tlsMode TLSMode, ourHostname, remoteHostname dns.Domain, auth []sasl.Client) (*Client, error) {
 	c := &Client{
 		origConn: conn,
-		conn:     conn,
 		lastlog:  time.Now(),
 		cmds:     []string{"(none)"},
 	}
@@ -182,6 +188,24 @@ func New(ctx context.Context, log *mlog.Log, conn net.Conn, tlsMode TLSMode, rem
 		c.lastlog = now
 		return l
 	})
+
+	if tlsMode == TLSStrictImmediate {
+		tlsconfig := tls.Config{
+			ServerName: remoteHostname.ASCII,
+			RootCAs:    mox.Conf.Static.TLS.CertPool,
+			MinVersion: tls.VersionTLS12, // ../rfc/8996:31 ../rfc/8997:66
+		}
+		tlsconn := tls.Client(conn, &tlsconfig)
+		if err := tlsconn.HandshakeContext(ctx); err != nil {
+			return nil, err
+		}
+		c.conn = tlsconn
+		tlsversion, ciphersuite := mox.TLSInfo(tlsconn)
+		c.log.Debug("tls client handshake done", mlog.Field("tls", tlsversion), mlog.Field("ciphersuite", ciphersuite), mlog.Field("servername", remoteHostname))
+	} else {
+		c.conn = conn
+	}
+
 	// We don't wrap reads in a timeoutReader for fear of an optional TLS wrapper doing
 	// reads without the client asking for it. Such reads could result in a timeout
 	// error.
@@ -192,7 +216,7 @@ func New(ctx context.Context, log *mlog.Log, conn net.Conn, tlsMode TLSMode, rem
 	c.tw = moxio.NewTraceWriter(c.log, "LC: ", timeoutWriter{c.conn, 30 * time.Second, c.log})
 	c.w = bufio.NewWriter(c.tw)
 
-	if err := c.hello(ctx, tlsMode, remoteHostname, auth); err != nil {
+	if err := c.hello(ctx, tlsMode, ourHostname, remoteHostname, auth); err != nil {
 		return nil, err
 	}
 	return c, nil
@@ -324,16 +348,18 @@ func (c *Client) readecode(ecodes bool) (code int, secode, lastLine string, text
 		}
 		code = co
 		if last {
-			cmd := ""
-			if len(c.cmds) > 0 {
-				cmd = c.cmds[0]
-				// We only keep the last, so we're not creating new slices all the time.
-				if len(c.cmds) > 1 {
-					c.cmds = c.cmds[1:]
+			if code != smtp.C334ContinueAuth {
+				cmd := ""
+				if len(c.cmds) > 0 {
+					cmd = c.cmds[0]
+					// We only keep the last, so we're not creating new slices all the time.
+					if len(c.cmds) > 1 {
+						c.cmds = c.cmds[1:]
+					}
 				}
+				metricCommands.WithLabelValues(cmd, fmt.Sprintf("%d", co), sec).Observe(float64(time.Since(c.cmdStart)) / float64(time.Second))
+				c.log.Debug("smtpclient command result", mlog.Field("cmd", cmd), mlog.Field("code", co), mlog.Field("secode", sec), mlog.Field("duration", time.Since(c.cmdStart)))
 			}
-			metricCommands.WithLabelValues(cmd, fmt.Sprintf("%d", co), sec).Observe(float64(time.Since(c.cmdStart)) / float64(time.Second))
-			c.log.Debug("smtpclient command result", mlog.Field("cmd", cmd), mlog.Field("code", co), mlog.Field("secode", sec), mlog.Field("duration", time.Since(c.cmdStart)))
 			return co, sec, line, texts, nil
 		}
 	}
@@ -437,7 +463,7 @@ func (c *Client) recover(rerr *error) {
 	*rerr = cerr
 }
 
-func (c *Client) hello(ctx context.Context, tlsMode TLSMode, remoteHostname, auth string) (rerr error) {
+func (c *Client) hello(ctx context.Context, tlsMode TLSMode, ourHostname, remoteHostname dns.Domain, auth []sasl.Client) (rerr error) {
 	defer c.recover(&rerr)
 
 	// perform EHLO handshake, falling back to HELO if server does not appear to
@@ -448,7 +474,7 @@ func (c *Client) hello(ctx context.Context, tlsMode TLSMode, remoteHostname, aut
 		c.cmds[0] = "ehlo"
 		c.cmdStart = time.Now()
 		// Syntax: ../rfc/5321:1827
-		c.xwritelinef("EHLO %s", mox.Conf.Static.HostnameDomain.ASCII)
+		c.xwritelinef("EHLO %s", ourHostname.ASCII)
 		code, _, lastLine, remains := c.xreadecode(false)
 		switch code {
 		// ../rfc/5321:997
@@ -460,7 +486,7 @@ func (c *Client) hello(ctx context.Context, tlsMode TLSMode, remoteHostname, aut
 			// ../rfc/5321:996
 			c.cmds[0] = "helo"
 			c.cmdStart = time.Now()
-			c.xwritelinef("HELO %s", mox.Conf.Static.HostnameDomain.ASCII)
+			c.xwritelinef("HELO %s", ourHostname.ASCII)
 			code, _, lastLine, _ = c.xreadecode(false)
 			if code != smtp.C250Completed {
 				c.xerrorf(code/100 == 5, code, "", lastLine, "%w: expected 250 to HELO, got %d", ErrStatus, code)
@@ -491,6 +517,8 @@ func (c *Client) hello(ctx context.Context, tlsMode TLSMode, remoteHostname, aut
 					if v, err := strconv.ParseInt(s[len("SIZE "):], 10, 64); err == nil {
 						c.maxSize = v
 					}
+				} else if strings.HasPrefix(s, "AUTH ") {
+					c.extAuthMechanisms = strings.Split(s[len("AUTH "):], " ")
 				}
 			}
 		}
@@ -507,8 +535,8 @@ func (c *Client) hello(ctx context.Context, tlsMode TLSMode, remoteHostname, aut
 	// Write EHLO, falling back to HELO if server doesn't appear to support it.
 	hello(true)
 
-	// Attempt TLS if remote understands STARTTLS or if caller requires it.
-	if c.extStartTLS && tlsMode != TLSSkip || tlsMode == TLSStrict {
+	// Attempt TLS if remote understands STARTTLS and we aren't doing immediate TLS or if caller requires it.
+	if c.extStartTLS && (tlsMode != TLSSkip && tlsMode != TLSStrictImmediate) || tlsMode == TLSStrictStartTLS {
 		c.log.Debug("starting tls client", mlog.Field("tlsmode", tlsMode), mlog.Field("servername", remoteHostname))
 		c.cmds[0] = "starttls"
 		c.cmdStart = time.Now()
@@ -531,13 +559,13 @@ func (c *Client) hello(ctx context.Context, tlsMode TLSMode, remoteHostname, aut
 			}
 		}
 
-		// For TLSStrict, the Go TLS library performs the checks needed for MTA-STS.
+		// For TLSStrictStartTLS, the Go TLS library performs the checks needed for MTA-STS.
 		// ../rfc/8461:646
 		// todo: possibly accept older TLS versions for TLSOpportunistic?
 		tlsConfig := &tls.Config{
-			ServerName:         remoteHostname,
+			ServerName:         remoteHostname.ASCII,
 			RootCAs:            mox.Conf.Static.TLS.CertPool,
-			InsecureSkipVerify: tlsMode != TLSStrict,
+			InsecureSkipVerify: tlsMode != TLSStrictStartTLS,
 			MinVersion:         tls.VersionTLS12, // ../rfc/8996:31 ../rfc/8997:66
 		}
 		nconn := tls.Client(conn, tlsConfig)
@@ -556,25 +584,103 @@ func (c *Client) hello(ctx context.Context, tlsMode TLSMode, remoteHostname, aut
 		c.w = bufio.NewWriter(c.tw)
 
 		tlsversion, ciphersuite := mox.TLSInfo(nconn)
-		c.log.Debug("tls client handshake done", mlog.Field("tls", tlsversion), mlog.Field("ciphersuite", ciphersuite), mlog.Field("servername", remoteHostname), mlog.Field("insecureskipverify", tlsConfig.InsecureSkipVerify))
+		c.log.Debug("starttls client handshake done", mlog.Field("tls", tlsversion), mlog.Field("ciphersuite", ciphersuite), mlog.Field("servername", remoteHostname), mlog.Field("insecureskipverify", tlsConfig.InsecureSkipVerify))
 
 		hello(false)
 	}
 
-	if auth != "" {
-		// No metrics, only used for tests.
-		c.cmds[0] = "auth"
-		c.cmdStart = time.Now()
-		defer c.xtrace(mlog.LevelTraceauth)()
-		c.xwriteline(auth)
-		c.xtrace(mlog.LevelTrace) // Restore.
-		code, secode, lastLine, _ := c.xread()
-		if code != smtp.C235AuthSuccess {
-			c.xerrorf(code/100 == 5, code, secode, lastLine, "%w: auth: got %d, expected 2xx", ErrStatus, code)
+	if len(auth) > 0 {
+		return c.auth(auth)
+	}
+	return
+}
+
+// ../rfc/4954:139
+func (c *Client) auth(auth []sasl.Client) (rerr error) {
+	defer c.recover(&rerr)
+
+	c.cmds[0] = "auth"
+	c.cmdStart = time.Now()
+
+	var a sasl.Client
+	var name string
+	var cleartextCreds bool
+	for _, x := range auth {
+		name, cleartextCreds = x.Info()
+		for _, s := range c.extAuthMechanisms {
+			if s == name {
+				a = x
+				break
+			}
 		}
 	}
+	if a == nil {
+		c.xerrorf(true, 0, "", "", "no matching authentication mechanisms, server supports %s", strings.Join(c.extAuthMechanisms, ", "))
+	}
 
-	return
+	abort := func() (int, string, string) {
+		// Abort authentication. ../rfc/4954:193
+		c.xwriteline("*")
+
+		// Server must respond with 501. // ../rfc/4954:195
+		code, secode, lastline, _ := c.xread()
+		if code != smtp.C501BadParamSyntax {
+			c.botched = true
+		}
+		return code, secode, lastline
+	}
+
+	toserver, last, err := a.Next(nil)
+	if err != nil {
+		c.xerrorf(false, 0, "", "", "initial step in auth mechanism %s: %w", name, err)
+	}
+	if cleartextCreds {
+		defer c.xtrace(mlog.LevelTraceauth)()
+	}
+	if toserver == nil {
+		c.xwriteline("AUTH " + name)
+	} else if len(toserver) == 0 {
+		c.xwriteline("AUTH " + name + " =") // ../rfc/4954:214
+	} else {
+		c.xwriteline("AUTH " + name + " " + base64.StdEncoding.EncodeToString(toserver))
+	}
+	for {
+		if cleartextCreds && last {
+			c.xtrace(mlog.LevelTrace) // Restore.
+		}
+
+		code, secode, lastLine, texts := c.xreadecode(last)
+		if code == smtp.C235AuthSuccess {
+			if !last {
+				c.xerrorf(false, code, secode, lastLine, "server completed authentication earlier than client expected")
+			}
+			return nil
+		} else if code == smtp.C334ContinueAuth {
+			if last {
+				c.xerrorf(false, code, secode, lastLine, "server requested unexpected continuation of authentication")
+			}
+			if len(texts) != 1 {
+				abort()
+				c.xerrorf(false, code, secode, lastLine, "server responded with multiline contination")
+			}
+			fromserver, err := base64.StdEncoding.DecodeString(texts[0])
+			if err != nil {
+				abort()
+				c.xerrorf(false, code, secode, lastLine, "malformed base64 data in authentication continuation response")
+			}
+			toserver, last, err = a.Next(fromserver)
+			if err != nil {
+				// For failing SCRAM, the client stops due to message about invalid proof. The
+				// server still sends an authentication result (it probably should send 501
+				// instead).
+				xcode, xsecode, lastline := abort()
+				c.xerrorf(false, xcode, xsecode, lastline, "client aborted authentication: %w", err)
+			}
+			c.xwriteline(base64.StdEncoding.EncodeToString(toserver))
+		} else {
+			c.xerrorf(code/100 == 5, code, secode, lastLine, "unexpected response during authentication, expected 334 continue or 235 auth success")
+		}
+	}
 }
 
 // Supports8BITMIME returns whether the SMTP server supports the 8BITMIME

smtpclient/client_test.go

@@ -5,10 +5,14 @@ import (
 	"context"
 	"crypto/ed25519"
 	cryptorand "crypto/rand"
+	"crypto/sha1"
+	"crypto/sha256"
 	"crypto/tls"
 	"crypto/x509"
+	"encoding/base64"
 	"errors"
 	"fmt"
+	"hash"
 	"io"
 	"math/big"
 	"net"
@@ -17,11 +21,17 @@ import (
 	"testing"
 	"time"
 
+	"github.com/mjl-/mox/dns"
 	"github.com/mjl-/mox/mlog"
 	"github.com/mjl-/mox/mox-"
+	"github.com/mjl-/mox/sasl"
+	"github.com/mjl-/mox/scram"
 	"github.com/mjl-/mox/smtp"
 )
 
+var zerohost dns.Domain
+var localhost = dns.Domain{ASCII: "localhost"}
+
 func TestClient(t *testing.T) {
 	ctx := context.Background()
 	log := mlog.New("smtpclient")
@@ -36,9 +46,10 @@ func TestClient(t *testing.T) {
 		ehlo         bool
 
 		tlsMode      TLSMode
-		tlsHostname  string
+		tlsHostname  dns.Domain
 		need8bitmime bool
 		needsmtputf8 bool
+		auths        []string // Allowed mechanisms.
 
 		nodeliver bool // For server, whether client will attempt a delivery.
 	}
@@ -51,7 +62,7 @@ func TestClient(t *testing.T) {
 		Certificates: []tls.Certificate{cert},
 	}
 
-	test := func(msg string, opts options, expClientErr, expDeliverErr, expServerErr error) {
+	test := func(msg string, opts options, auths []sasl.Client, expClientErr, expDeliverErr, expServerErr error) {
 		t.Helper()
 
 		if opts.tlsMode == "" {
@@ -80,7 +91,7 @@ func TestClient(t *testing.T) {
 			}
 
 			br := bufio.NewReader(serverConn)
-			readline := func(prefix string) {
+			readline := func(prefix string) string {
 				s, err := br.ReadString('\n')
 				if err != nil {
 					fail("expected command: %v", err)
@@ -88,6 +99,8 @@ func TestClient(t *testing.T) {
 				if !strings.HasPrefix(strings.ToLower(s), strings.ToLower(prefix)) {
 					fail("expected command %q, got: %s", prefix, s)
 				}
+				s = s[len(prefix):]
+				return strings.TrimSuffix(s, "\r\n")
 			}
 			writeline := func(s string) {
 				fmt.Fprintf(serverConn, "%s\r\n", s)
@@ -133,6 +146,9 @@ func TestClient(t *testing.T) {
 				if opts.smtputf8 {
 					writeline("250-SMTPUTF8")
 				}
+				if opts.auths != nil {
+					writeline("250-AUTH " + strings.Join(opts.auths, " "))
+				}
 				writeline("250 UNKNOWN") // To be ignored.
 			}
 
@@ -157,6 +173,61 @@ func TestClient(t *testing.T) {
 				hello()
 			}
 
+			if opts.auths != nil {
+				more := readline("AUTH ")
+				t := strings.SplitN(more, " ", 2)
+				switch t[0] {
+				case "PLAIN":
+					writeline("235 2.7.0 auth ok")
+				case "CRAM-MD5":
+					writeline("334 " + base64.StdEncoding.EncodeToString([]byte("<123.1234@host>")))
+					readline("") // Proof
+					writeline("235 2.7.0 auth ok")
+				case "SCRAM-SHA-1", "SCRAM-SHA-256":
+					// Cannot fake/hardcode scram interactions.
+					var h func() hash.Hash
+					salt := scram.MakeRandom()
+					var iterations int
+					if t[0] == "SCRAM-SHA-1" {
+						h = sha1.New
+						iterations = 2 * 4096
+					} else {
+						h = sha256.New
+						iterations = 4096
+					}
+					saltedPassword := scram.SaltPassword(h, "test", salt, iterations)
+
+					clientFirst, err := base64.StdEncoding.DecodeString(t[1])
+					if err != nil {
+						fail("bad base64: %w", err)
+					}
+					s, err := scram.NewServer(h, clientFirst)
+					if err != nil {
+						fail("scram new server: %w", err)
+					}
+					serverFirst, err := s.ServerFirst(iterations, salt)
+					if err != nil {
+						fail("scram server first: %w", err)
+					}
+					writeline("334 " + base64.StdEncoding.EncodeToString([]byte(serverFirst)))
+
+					xclientFinal := readline("")
+					clientFinal, err := base64.StdEncoding.DecodeString(xclientFinal)
+					if err != nil {
+						fail("bad base64: %w", err)
+					}
+					serverFinal, err := s.Finish([]byte(clientFinal), saltedPassword)
+					if err != nil {
+						fail("scram finish: %w", err)
+					}
+					writeline("334 " + base64.StdEncoding.EncodeToString([]byte(serverFinal)))
+					readline("")
+					writeline("235 2.7.0 auth ok")
+				default:
+					writeline("501 unknown mechanism")
+				}
+			}
+
 			if expClientErr == nil && !opts.nodeliver {
 				readline("MAIL FROM:")
 				writeline("250 ok")
@@ -200,7 +271,7 @@ func TestClient(t *testing.T) {
 				result <- fmt.Errorf("client: %w", fmt.Errorf(format, args...))
 				panic("stop")
 			}
-			c, err := New(ctx, log, clientConn, opts.tlsMode, opts.tlsHostname, "")
+			c, err := New(ctx, log, clientConn, opts.tlsMode, localhost, opts.tlsHostname, auths)
 			if (err == nil) != (expClientErr == nil) || err != nil && !errors.As(err, reflect.New(reflect.ValueOf(expClientErr).Type()).Interface()) && !errors.Is(err, expClientErr) {
 				fail("new client: got err %v, expected %#v", err, expClientErr)
 			}
@@ -257,19 +328,24 @@ test
 		starttls:     true,
 		ehlo:         true,
 
-		tlsMode:      TLSStrict,
-		tlsHostname:  "mox.example",
+		tlsMode:      TLSStrictStartTLS,
+		tlsHostname:  dns.Domain{ASCII: "mox.example"},
 		need8bitmime: true,
 		needsmtputf8: true,
 	}
 
-	test(msg, options{}, nil, nil, nil)
-	test(msg, allopts, nil, nil, nil)
-	test(msg, options{ehlo: true, eightbitmime: true}, nil, nil, nil)
-	test(msg, options{ehlo: true, eightbitmime: false, need8bitmime: true, nodeliver: true}, nil, Err8bitmimeUnsupported, nil)
-	test(msg, options{ehlo: true, smtputf8: false, needsmtputf8: true, nodeliver: true}, nil, ErrSMTPUTF8Unsupported, nil)
-	test(msg, options{ehlo: true, starttls: true, tlsMode: TLSStrict, tlsHostname: "mismatch.example", nodeliver: true}, ErrTLS, nil, &net.OpError{}) // Server TLS handshake is a net.OpError with "remote error" as text.
-	test(msg, options{ehlo: true, maxSize: len(msg) - 1, nodeliver: true}, nil, ErrSize, nil)
+	test(msg, options{}, nil, nil, nil, nil)
+	test(msg, allopts, nil, nil, nil, nil)
+	test(msg, options{ehlo: true, eightbitmime: true}, nil, nil, nil, nil)
+	test(msg, options{ehlo: true, eightbitmime: false, need8bitmime: true, nodeliver: true}, nil, nil, Err8bitmimeUnsupported, nil)
+	test(msg, options{ehlo: true, smtputf8: false, needsmtputf8: true, nodeliver: true}, nil, nil, ErrSMTPUTF8Unsupported, nil)
+	test(msg, options{ehlo: true, starttls: true, tlsMode: TLSStrictStartTLS, tlsHostname: dns.Domain{ASCII: "mismatch.example"}, nodeliver: true}, nil, ErrTLS, nil, &net.OpError{}) // Server TLS handshake is a net.OpError with "remote error" as text.
+	test(msg, options{ehlo: true, maxSize: len(msg) - 1, nodeliver: true}, nil, nil, ErrSize, nil)
+	test(msg, options{ehlo: true, auths: []string{"PLAIN"}}, []sasl.Client{sasl.NewClientPlain("test", "test")}, nil, nil, nil)
+	test(msg, options{ehlo: true, auths: []string{"CRAM-MD5"}}, []sasl.Client{sasl.NewClientCRAMMD5("test", "test")}, nil, nil, nil)
+	test(msg, options{ehlo: true, auths: []string{"SCRAM-SHA-1"}}, []sasl.Client{sasl.NewClientSCRAMSHA1("test", "test")}, nil, nil, nil)
+	test(msg, options{ehlo: true, auths: []string{"SCRAM-SHA-256"}}, []sasl.Client{sasl.NewClientSCRAMSHA256("test", "test")}, nil, nil, nil)
+	// todo: add tests for failing authentication, also at various stages in SCRAM
 
 	// Set an expired certificate. For non-strict TLS, we should still accept it.
 	// ../rfc/7435:424
@@ -279,14 +355,14 @@ test
 	tlsConfig = tls.Config{
 		Certificates: []tls.Certificate{cert},
 	}
-	test(msg, options{ehlo: true, starttls: true}, nil, nil, nil)
+	test(msg, options{ehlo: true, starttls: true}, nil, nil, nil, nil)
 
 	// Again with empty cert pool so it isn't trusted in any way.
 	mox.Conf.Static.TLS.CertPool = x509.NewCertPool()
 	tlsConfig = tls.Config{
 		Certificates: []tls.Certificate{cert},
 	}
-	test(msg, options{ehlo: true, starttls: true}, nil, nil, nil)
+	test(msg, options{ehlo: true, starttls: true}, nil, nil, nil, nil)
 }
 
 func TestErrors(t *testing.T) {
@@ -297,7 +373,7 @@ func TestErrors(t *testing.T) {
 	run(t, func(s xserver) {
 		s.writeline("bogus") // Invalid, should be "220 <hostname>".
 	}, func(conn net.Conn) {
-		_, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		_, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		var xerr Error
 		if err == nil || !errors.Is(err, ErrProtocol) || !errors.As(err, &xerr) || xerr.Permanent {
 			panic(fmt.Errorf("got %#v, expected ErrProtocol without Permanent", err))
@@ -308,7 +384,7 @@ func TestErrors(t *testing.T) {
 	run(t, func(s xserver) {
 		s.conn.Close()
 	}, func(conn net.Conn) {
-		_, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		_, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		var xerr Error
 		if err == nil || !errors.Is(err, io.ErrUnexpectedEOF) || !errors.As(err, &xerr) || xerr.Permanent {
 			panic(fmt.Errorf("got %#v (%v), expected ErrUnexpectedEOF without Permanent", err, err))
@@ -319,7 +395,7 @@ func TestErrors(t *testing.T) {
 	run(t, func(s xserver) {
 		s.writeline("521 not accepting connections")
 	}, func(conn net.Conn) {
-		_, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		_, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		var xerr Error
 		if err == nil || !errors.Is(err, ErrStatus) || !errors.As(err, &xerr) || !xerr.Permanent {
 			panic(fmt.Errorf("got %#v, expected ErrStatus with Permanent", err))
@@ -330,7 +406,7 @@ func TestErrors(t *testing.T) {
 	run(t, func(s xserver) {
 		s.writeline("2200 mox.example") // Invalid, too many digits.
 	}, func(conn net.Conn) {
-		_, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		_, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		var xerr Error
 		if err == nil || !errors.Is(err, ErrProtocol) || !errors.As(err, &xerr) || xerr.Permanent {
 			panic(fmt.Errorf("got %#v, expected ErrProtocol without Permanent", err))
@@ -344,7 +420,7 @@ func TestErrors(t *testing.T) {
 		s.writeline("250-mox.example")
 		s.writeline("500 different code") // Invalid.
 	}, func(conn net.Conn) {
-		_, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		_, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		var xerr Error
 		if err == nil || !errors.Is(err, ErrProtocol) || !errors.As(err, &xerr) || xerr.Permanent {
 			panic(fmt.Errorf("got %#v, expected ErrProtocol without Permanent", err))
@@ -360,7 +436,7 @@ func TestErrors(t *testing.T) {
 		s.readline("MAIL FROM:")
 		s.writeline("550 5.7.0 not allowed")
 	}, func(conn net.Conn) {
-		c, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		c, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		if err != nil {
 			panic(err)
 		}
@@ -380,7 +456,7 @@ func TestErrors(t *testing.T) {
 		s.readline("MAIL FROM:")
 		s.writeline("451 bad sender")
 	}, func(conn net.Conn) {
-		c, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		c, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		if err != nil {
 			panic(err)
 		}
@@ -402,7 +478,7 @@ func TestErrors(t *testing.T) {
 		s.readline("RCPT TO:")
 		s.writeline("451")
 	}, func(conn net.Conn) {
-		c, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		c, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		if err != nil {
 			panic(err)
 		}
@@ -426,7 +502,7 @@ func TestErrors(t *testing.T) {
 		s.readline("DATA")
 		s.writeline("550 no!")
 	}, func(conn net.Conn) {
-		c, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		c, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		if err != nil {
 			panic(err)
 		}
@@ -446,7 +522,7 @@ func TestErrors(t *testing.T) {
 		s.readline("STARTTLS")
 		s.writeline("502 command not implemented")
 	}, func(conn net.Conn) {
-		_, err := New(ctx, log, conn, TLSStrict, "mox.example", "")
+		_, err := New(ctx, log, conn, TLSStrictStartTLS, localhost, dns.Domain{ASCII: "mox.example"}, nil)
 		var xerr Error
 		if err == nil || !errors.Is(err, ErrTLS) || !errors.As(err, &xerr) || !xerr.Permanent {
 			panic(fmt.Errorf("got %#v, expected ErrTLS with Permanent", err))
@@ -462,7 +538,7 @@ func TestErrors(t *testing.T) {
 		s.readline("MAIL FROM:")
 		s.writeline("451 enough")
 	}, func(conn net.Conn) {
-		c, err := New(ctx, log, conn, TLSSkip, "mox.example", "")
+		c, err := New(ctx, log, conn, TLSSkip, localhost, dns.Domain{ASCII: "mox.example"}, nil)
 		if err != nil {
 			panic(err)
 		}
@@ -492,7 +568,7 @@ func TestErrors(t *testing.T) {
 		s.readline("DATA")
 		s.writeline("550 not now")
 	}, func(conn net.Conn) {
-		c, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		c, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		if err != nil {
 			panic(err)
 		}
@@ -522,7 +598,7 @@ func TestErrors(t *testing.T) {
 		s.readline("MAIL FROM:")
 		s.writeline("550 ok")
 	}, func(conn net.Conn) {
-		c, err := New(ctx, log, conn, TLSOpportunistic, "", "")
+		c, err := New(ctx, log, conn, TLSOpportunistic, localhost, zerohost, nil)
 		if err != nil {
 			panic(err)
 		}

smtpserver/dsn.go

@@ -46,7 +46,7 @@ func queueDSN(ctx context.Context, c *conn, rcptTo smtp.Path, m dsn.Message) err
 	// ../rfc/3464:433
 	const has8bit = false
 	const smtputf8 = false
-	if err := queue.Add(ctx, c.log, "", smtp.Path{}, rcptTo, has8bit, smtputf8, int64(len(buf)), nil, f, bufUTF8, true); err != nil {
+	if _, err := queue.Add(ctx, c.log, "", smtp.Path{}, rcptTo, has8bit, smtputf8, int64(len(buf)), nil, f, bufUTF8, true); err != nil {
 		return err
 	}
 	err = f.Close()

smtpserver/server.go

@@ -799,7 +799,7 @@ func (c *conn) cmdHello(p *parser, ehlo bool) {
 	if c.submission {
 		// ../rfc/4954:123
 		if c.tls || !c.requireTLSForAuth {
-			c.bwritelinef("250-AUTH PLAIN SCRAM-SHA-256 SCRAM-SHA-1 CRAM-MD5")
+			c.bwritelinef("250-AUTH SCRAM-SHA-256 SCRAM-SHA-1 CRAM-MD5 PLAIN")
 		} else {
 			c.bwritelinef("250-AUTH ")
 		}
@@ -1860,7 +1860,7 @@ func (c *conn) submit(ctx context.Context, recvHdrFor func(string) string, msgWr
 			}
 
 			msgSize := int64(len(xmsgPrefix)) + msgWriter.Size
-			if err := queue.Add(ctx, c.log, c.account.Name, *c.mailFrom, rcptAcc.rcptTo, msgWriter.Has8bit, c.smtputf8, msgSize, xmsgPrefix, dataFile, nil, i == len(c.recipients)-1); err != nil {
+			if _, err := queue.Add(ctx, c.log, c.account.Name, *c.mailFrom, rcptAcc.rcptTo, msgWriter.Has8bit, c.smtputf8, msgSize, xmsgPrefix, dataFile, nil, i == len(c.recipients)-1); err != nil {
 				// Aborting the transaction is not great. But continuing and generating DSNs will
 				// probably result in errors as well...
 				metricSubmission.WithLabelValues("queueerror").Inc()

smtpserver/server_test.go

@@ -32,6 +32,7 @@ import (
 	"github.com/mjl-/mox/mlog"
 	"github.com/mjl-/mox/mox-"
 	"github.com/mjl-/mox/queue"
+	"github.com/mjl-/mox/sasl"
 	"github.com/mjl-/mox/smtp"
 	"github.com/mjl-/mox/smtpclient"
 	"github.com/mjl-/mox/store"
@@ -79,6 +80,7 @@ type testserver struct {
 	comm       *store.Comm
 	cid        int64
 	resolver   dns.Resolver
+	auth       []sasl.Client
 	user, pass string
 	submission bool
 	dnsbls     []dns.Domain
@@ -135,12 +137,16 @@ func (ts *testserver) run(fn func(helloErr error, client *smtpclient.Client)) {
 		close(serverdone)
 	}()
 
-	var authLine string
-	if ts.user != "" {
-		authLine = fmt.Sprintf("AUTH PLAIN %s", base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("\u0000%s\u0000%s", ts.user, ts.pass))))
+	var auth []sasl.Client
+	if len(ts.auth) > 0 {
+		auth = ts.auth
+	} else if ts.user != "" {
+		auth = append(auth, sasl.NewClientPlain(ts.user, ts.pass))
 	}
 
-	client, err := smtpclient.New(ctxbg, xlog.WithCid(ts.cid-1), clientConn, ts.tlsmode, "mox.example", authLine)
+	ourHostname := mox.Conf.Static.HostnameDomain
+	remoteHostname := dns.Domain{ASCII: "mox.example"}
+	client, err := smtpclient.New(ctxbg, xlog.WithCid(ts.cid-1), clientConn, ts.tlsmode, ourHostname, remoteHostname, auth)
 	if err != nil {
 		clientConn.Close()
 	} else {
@@ -192,10 +198,13 @@ func TestSubmission(t *testing.T) {
 	}
 	mox.Conf.Dynamic.Domains["mox.example"] = dom
 
-	testAuth := func(user, pass string, expErr *smtpclient.Error) {
+	testAuth := func(authfn func(user, pass string) sasl.Client, user, pass string, expErr *smtpclient.Error) {
 		t.Helper()
-		ts.user = user
-		ts.pass = pass
+		if authfn != nil {
+			ts.auth = []sasl.Client{authfn(user, pass)}
+		} else {
+			ts.auth = nil
+		}
 		ts.run(func(err error, client *smtpclient.Client) {
 			t.Helper()
 			mailFrom := "mjl@mox.example"
@@ -205,16 +214,24 @@ func TestSubmission(t *testing.T) {
 			}
 			var cerr smtpclient.Error
 			if expErr == nil && err != nil || expErr != nil && (err == nil || !errors.As(err, &cerr) || cerr.Secode != expErr.Secode) {
-				t.Fatalf("got err %#v, expected %#v", err, expErr)
+				t.Fatalf("got err %#v (%q), expected %#v", err, err, expErr)
 			}
 		})
 	}
 
 	ts.submission = true
-	testAuth("", "", &smtpclient.Error{Permanent: true, Code: smtp.C530SecurityRequired, Secode: smtp.SePol7Other0})
-	testAuth("mjl@mox.example", "test", &smtpclient.Error{Secode: smtp.SePol7AuthBadCreds8})         // Bad (short) password.
-	testAuth("mjl@mox.example", "testtesttest", &smtpclient.Error{Secode: smtp.SePol7AuthBadCreds8}) // Bad password.
-	testAuth("mjl@mox.example", "testtest", nil)
+	testAuth(nil, "", "", &smtpclient.Error{Permanent: true, Code: smtp.C530SecurityRequired, Secode: smtp.SePol7Other0})
+	authfns := []func(user, pass string) sasl.Client{
+		sasl.NewClientPlain,
+		sasl.NewClientCRAMMD5,
+		sasl.NewClientSCRAMSHA1,
+		sasl.NewClientSCRAMSHA256,
+	}
+	for _, fn := range authfns {
+		testAuth(fn, "mjl@mox.example", "test", &smtpclient.Error{Secode: smtp.SePol7AuthBadCreds8})         // Bad (short) password.
+		testAuth(fn, "mjl@mox.example", "testtesttest", &smtpclient.Error{Secode: smtp.SePol7AuthBadCreds8}) // Bad password.
+		testAuth(fn, "mjl@mox.example", "testtest", nil)
+	}
 }
 
 // Test delivery from external MTA.

testdata/queue/domains.conf

@@ -5,3 +5,9 @@ Accounts:
 		Domain: mox.example
 		Destinations:
 			mjl@mox.example: nil
+
+Routes:
+	-
+		ToDomain:
+			- submit.example
+		Transport: submit

testdata/queue/mox.conf

@@ -7,3 +7,30 @@ Listeners:
 Postmaster:
 	Account: mjl
 	Mailbox: postmaster
+Transports:
+	submit:
+		Submission:
+			# Dial of host is intercepted in tests.
+			Host: submission.example
+			NoSTARTTLS: true
+			Auth:
+				Username: test
+				Password: test1234
+				Mechanisms:
+					- PLAIN
+	submittls:
+		Submissions:
+			# Dial of host is intercepted in tests.
+			Host: submission.example
+			Auth:
+				Username: test
+				Password: test1234
+				Mechanisms:
+					- PLAIN
+	socks:
+		Socks:
+			# Address is replaced during tests.
+			Address: localhost:1234
+			RemoteIPs:
+				- 127.0.0.1
+			RemoteHostname: localhost

vendor/golang.org/x/net/internal/socks/client.go

@@ -0,0 +1,168 @@
+// Copyright 2018 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package socks
+
+import (
+	"context"
+	"errors"
+	"io"
+	"net"
+	"strconv"
+	"time"
+)
+
+var (
+	noDeadline   = time.Time{}
+	aLongTimeAgo = time.Unix(1, 0)
+)
+
+func (d *Dialer) connect(ctx context.Context, c net.Conn, address string) (_ net.Addr, ctxErr error) {
+	host, port, err := splitHostPort(address)
+	if err != nil {
+		return nil, err
+	}
+	if deadline, ok := ctx.Deadline(); ok && !deadline.IsZero() {
+		c.SetDeadline(deadline)
+		defer c.SetDeadline(noDeadline)
+	}
+	if ctx != context.Background() {
+		errCh := make(chan error, 1)
+		done := make(chan struct{})
+		defer func() {
+			close(done)
+			if ctxErr == nil {
+				ctxErr = <-errCh
+			}
+		}()
+		go func() {
+			select {
+			case <-ctx.Done():
+				c.SetDeadline(aLongTimeAgo)
+				errCh <- ctx.Err()
+			case <-done:
+				errCh <- nil
+			}
+		}()
+	}
+
+	b := make([]byte, 0, 6+len(host)) // the size here is just an estimate
+	b = append(b, Version5)
+	if len(d.AuthMethods) == 0 || d.Authenticate == nil {
+		b = append(b, 1, byte(AuthMethodNotRequired))
+	} else {
+		ams := d.AuthMethods
+		if len(ams) > 255 {
+			return nil, errors.New("too many authentication methods")
+		}
+		b = append(b, byte(len(ams)))
+		for _, am := range ams {
+			b = append(b, byte(am))
+		}
+	}
+	if _, ctxErr = c.Write(b); ctxErr != nil {
+		return
+	}
+
+	if _, ctxErr = io.ReadFull(c, b[:2]); ctxErr != nil {
+		return
+	}
+	if b[0] != Version5 {
+		return nil, errors.New("unexpected protocol version " + strconv.Itoa(int(b[0])))
+	}
+	am := AuthMethod(b[1])
+	if am == AuthMethodNoAcceptableMethods {
+		return nil, errors.New("no acceptable authentication methods")
+	}
+	if d.Authenticate != nil {
+		if ctxErr = d.Authenticate(ctx, c, am); ctxErr != nil {
+			return
+		}
+	}
+
+	b = b[:0]
+	b = append(b, Version5, byte(d.cmd), 0)
+	if ip := net.ParseIP(host); ip != nil {
+		if ip4 := ip.To4(); ip4 != nil {
+			b = append(b, AddrTypeIPv4)
+			b = append(b, ip4...)
+		} else if ip6 := ip.To16(); ip6 != nil {
+			b = append(b, AddrTypeIPv6)
+			b = append(b, ip6...)
+		} else {
+			return nil, errors.New("unknown address type")
+		}
+	} else {
+		if len(host) > 255 {
+			return nil, errors.New("FQDN too long")
+		}
+		b = append(b, AddrTypeFQDN)
+		b = append(b, byte(len(host)))
+		b = append(b, host...)
+	}
+	b = append(b, byte(port>>8), byte(port))
+	if _, ctxErr = c.Write(b); ctxErr != nil {
+		return
+	}
+
+	if _, ctxErr = io.ReadFull(c, b[:4]); ctxErr != nil {
+		return
+	}
+	if b[0] != Version5 {
+		return nil, errors.New("unexpected protocol version " + strconv.Itoa(int(b[0])))
+	}
+	if cmdErr := Reply(b[1]); cmdErr != StatusSucceeded {
+		return nil, errors.New("unknown error " + cmdErr.String())
+	}
+	if b[2] != 0 {
+		return nil, errors.New("non-zero reserved field")
+	}
+	l := 2
+	var a Addr
+	switch b[3] {
+	case AddrTypeIPv4:
+		l += net.IPv4len
+		a.IP = make(net.IP, net.IPv4len)
+	case AddrTypeIPv6:
+		l += net.IPv6len
+		a.IP = make(net.IP, net.IPv6len)
+	case AddrTypeFQDN:
+		if _, err := io.ReadFull(c, b[:1]); err != nil {
+			return nil, err
+		}
+		l += int(b[0])
+	default:
+		return nil, errors.New("unknown address type " + strconv.Itoa(int(b[3])))
+	}
+	if cap(b) < l {
+		b = make([]byte, l)
+	} else {
+		b = b[:l]
+	}
+	if _, ctxErr = io.ReadFull(c, b); ctxErr != nil {
+		return
+	}
+	if a.IP != nil {
+		copy(a.IP, b)
+	} else {
+		a.Name = string(b[:len(b)-2])
+	}
+	a.Port = int(b[len(b)-2])<<8 | int(b[len(b)-1])
+	return &a, nil
+}
+
+func splitHostPort(address string) (string, int, error) {
+	host, port, err := net.SplitHostPort(address)
+	if err != nil {
+		return "", 0, err
+	}
+	portnum, err := strconv.Atoi(port)
+	if err != nil {
+		return "", 0, err
+	}
+	if 1 > portnum || portnum > 0xffff {
+		return "", 0, errors.New("port number out of range " + port)
+	}
+	return host, portnum, nil
+}

vendor/golang.org/x/net/internal/socks/socks.go

@@ -0,0 +1,317 @@
+// Copyright 2018 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// Package socks provides a SOCKS version 5 client implementation.
+//
+// SOCKS protocol version 5 is defined in RFC 1928.
+// Username/Password authentication for SOCKS version 5 is defined in
+// RFC 1929.
+package socks
+
+import (
+	"context"
+	"errors"
+	"io"
+	"net"
+	"strconv"
+)
+
+// A Command represents a SOCKS command.
+type Command int
+
+func (cmd Command) String() string {
+	switch cmd {
+	case CmdConnect:
+		return "socks connect"
+	case cmdBind:
+		return "socks bind"
+	default:
+		return "socks " + strconv.Itoa(int(cmd))
+	}
+}
+
+// An AuthMethod represents a SOCKS authentication method.
+type AuthMethod int
+
+// A Reply represents a SOCKS command reply code.
+type Reply int
+
+func (code Reply) String() string {
+	switch code {
+	case StatusSucceeded:
+		return "succeeded"
+	case 0x01:
+		return "general SOCKS server failure"
+	case 0x02:
+		return "connection not allowed by ruleset"
+	case 0x03:
+		return "network unreachable"
+	case 0x04:
+		return "host unreachable"
+	case 0x05:
+		return "connection refused"
+	case 0x06:
+		return "TTL expired"
+	case 0x07:
+		return "command not supported"
+	case 0x08:
+		return "address type not supported"
+	default:
+		return "unknown code: " + strconv.Itoa(int(code))
+	}
+}
+
+// Wire protocol constants.
+const (
+	Version5 = 0x05
+
+	AddrTypeIPv4 = 0x01
+	AddrTypeFQDN = 0x03
+	AddrTypeIPv6 = 0x04
+
+	CmdConnect Command = 0x01 // establishes an active-open forward proxy connection
+	cmdBind    Command = 0x02 // establishes a passive-open forward proxy connection
+
+	AuthMethodNotRequired         AuthMethod = 0x00 // no authentication required
+	AuthMethodUsernamePassword    AuthMethod = 0x02 // use username/password
+	AuthMethodNoAcceptableMethods AuthMethod = 0xff // no acceptable authentication methods
+
+	StatusSucceeded Reply = 0x00
+)
+
+// An Addr represents a SOCKS-specific address.
+// Either Name or IP is used exclusively.
+type Addr struct {
+	Name string // fully-qualified domain name
+	IP   net.IP
+	Port int
+}
+
+func (a *Addr) Network() string { return "socks" }
+
+func (a *Addr) String() string {
+	if a == nil {
+		return "<nil>"
+	}
+	port := strconv.Itoa(a.Port)
+	if a.IP == nil {
+		return net.JoinHostPort(a.Name, port)
+	}
+	return net.JoinHostPort(a.IP.String(), port)
+}
+
+// A Conn represents a forward proxy connection.
+type Conn struct {
+	net.Conn
+
+	boundAddr net.Addr
+}
+
+// BoundAddr returns the address assigned by the proxy server for
+// connecting to the command target address from the proxy server.
+func (c *Conn) BoundAddr() net.Addr {
+	if c == nil {
+		return nil
+	}
+	return c.boundAddr
+}
+
+// A Dialer holds SOCKS-specific options.
+type Dialer struct {
+	cmd          Command // either CmdConnect or cmdBind
+	proxyNetwork string  // network between a proxy server and a client
+	proxyAddress string  // proxy server address
+
+	// ProxyDial specifies the optional dial function for
+	// establishing the transport connection.
+	ProxyDial func(context.Context, string, string) (net.Conn, error)
+
+	// AuthMethods specifies the list of request authentication
+	// methods.
+	// If empty, SOCKS client requests only AuthMethodNotRequired.
+	AuthMethods []AuthMethod
+
+	// Authenticate specifies the optional authentication
+	// function. It must be non-nil when AuthMethods is not empty.
+	// It must return an error when the authentication is failed.
+	Authenticate func(context.Context, io.ReadWriter, AuthMethod) error
+}
+
+// DialContext connects to the provided address on the provided
+// network.
+//
+// The returned error value may be a net.OpError. When the Op field of
+// net.OpError contains "socks", the Source field contains a proxy
+// server address and the Addr field contains a command target
+// address.
+//
+// See func Dial of the net package of standard library for a
+// description of the network and address parameters.
+func (d *Dialer) DialContext(ctx context.Context, network, address string) (net.Conn, error) {
+	if err := d.validateTarget(network, address); err != nil {
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: err}
+	}
+	if ctx == nil {
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: errors.New("nil context")}
+	}
+	var err error
+	var c net.Conn
+	if d.ProxyDial != nil {
+		c, err = d.ProxyDial(ctx, d.proxyNetwork, d.proxyAddress)
+	} else {
+		var dd net.Dialer
+		c, err = dd.DialContext(ctx, d.proxyNetwork, d.proxyAddress)
+	}
+	if err != nil {
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: err}
+	}
+	a, err := d.connect(ctx, c, address)
+	if err != nil {
+		c.Close()
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: err}
+	}
+	return &Conn{Conn: c, boundAddr: a}, nil
+}
+
+// DialWithConn initiates a connection from SOCKS server to the target
+// network and address using the connection c that is already
+// connected to the SOCKS server.
+//
+// It returns the connection's local address assigned by the SOCKS
+// server.
+func (d *Dialer) DialWithConn(ctx context.Context, c net.Conn, network, address string) (net.Addr, error) {
+	if err := d.validateTarget(network, address); err != nil {
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: err}
+	}
+	if ctx == nil {
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: errors.New("nil context")}
+	}
+	a, err := d.connect(ctx, c, address)
+	if err != nil {
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: err}
+	}
+	return a, nil
+}
+
+// Dial connects to the provided address on the provided network.
+//
+// Unlike DialContext, it returns a raw transport connection instead
+// of a forward proxy connection.
+//
+// Deprecated: Use DialContext or DialWithConn instead.
+func (d *Dialer) Dial(network, address string) (net.Conn, error) {
+	if err := d.validateTarget(network, address); err != nil {
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: err}
+	}
+	var err error
+	var c net.Conn
+	if d.ProxyDial != nil {
+		c, err = d.ProxyDial(context.Background(), d.proxyNetwork, d.proxyAddress)
+	} else {
+		c, err = net.Dial(d.proxyNetwork, d.proxyAddress)
+	}
+	if err != nil {
+		proxy, dst, _ := d.pathAddrs(address)
+		return nil, &net.OpError{Op: d.cmd.String(), Net: network, Source: proxy, Addr: dst, Err: err}
+	}
+	if _, err := d.DialWithConn(context.Background(), c, network, address); err != nil {
+		c.Close()
+		return nil, err
+	}
+	return c, nil
+}
+
+func (d *Dialer) validateTarget(network, address string) error {
+	switch network {
+	case "tcp", "tcp6", "tcp4":
+	default:
+		return errors.New("network not implemented")
+	}
+	switch d.cmd {
+	case CmdConnect, cmdBind:
+	default:
+		return errors.New("command not implemented")
+	}
+	return nil
+}
+
+func (d *Dialer) pathAddrs(address string) (proxy, dst net.Addr, err error) {
+	for i, s := range []string{d.proxyAddress, address} {
+		host, port, err := splitHostPort(s)
+		if err != nil {
+			return nil, nil, err
+		}
+		a := &Addr{Port: port}
+		a.IP = net.ParseIP(host)
+		if a.IP == nil {
+			a.Name = host
+		}
+		if i == 0 {
+			proxy = a
+		} else {
+			dst = a
+		}
+	}
+	return
+}
+
+// NewDialer returns a new Dialer that dials through the provided
+// proxy server's network and address.
+func NewDialer(network, address string) *Dialer {
+	return &Dialer{proxyNetwork: network, proxyAddress: address, cmd: CmdConnect}
+}
+
+const (
+	authUsernamePasswordVersion = 0x01
+	authStatusSucceeded         = 0x00
+)
+
+// UsernamePassword are the credentials for the username/password
+// authentication method.
+type UsernamePassword struct {
+	Username string
+	Password string
+}
+
+// Authenticate authenticates a pair of username and password with the
+// proxy server.
+func (up *UsernamePassword) Authenticate(ctx context.Context, rw io.ReadWriter, auth AuthMethod) error {
+	switch auth {
+	case AuthMethodNotRequired:
+		return nil
+	case AuthMethodUsernamePassword:
+		if len(up.Username) == 0 || len(up.Username) > 255 || len(up.Password) == 0 || len(up.Password) > 255 {
+			return errors.New("invalid username/password")
+		}
+		b := []byte{authUsernamePasswordVersion}
+		b = append(b, byte(len(up.Username)))
+		b = append(b, up.Username...)
+		b = append(b, byte(len(up.Password)))
+		b = append(b, up.Password...)
+		// TODO(mikio): handle IO deadlines and cancelation if
+		// necessary
+		if _, err := rw.Write(b); err != nil {
+			return err
+		}
+		if _, err := io.ReadFull(rw, b[:2]); err != nil {
+			return err
+		}
+		if b[0] != authUsernamePasswordVersion {
+			return errors.New("invalid username/password version")
+		}
+		if b[1] != authStatusSucceeded {
+			return errors.New("username/password authentication failed")
+		}
+		return nil
+	}
+	return errors.New("unsupported authentication method " + strconv.Itoa(int(auth)))
+}

vendor/golang.org/x/net/proxy/dial.go

@@ -0,0 +1,54 @@
+// Copyright 2019 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package proxy
+
+import (
+	"context"
+	"net"
+)
+
+// A ContextDialer dials using a context.
+type ContextDialer interface {
+	DialContext(ctx context.Context, network, address string) (net.Conn, error)
+}
+
+// Dial works like DialContext on net.Dialer but using a dialer returned by FromEnvironment.
+//
+// The passed ctx is only used for returning the Conn, not the lifetime of the Conn.
+//
+// Custom dialers (registered via RegisterDialerType) that do not implement ContextDialer
+// can leak a goroutine for as long as it takes the underlying Dialer implementation to timeout.
+//
+// A Conn returned from a successful Dial after the context has been cancelled will be immediately closed.
+func Dial(ctx context.Context, network, address string) (net.Conn, error) {
+	d := FromEnvironment()
+	if xd, ok := d.(ContextDialer); ok {
+		return xd.DialContext(ctx, network, address)
+	}
+	return dialContext(ctx, d, network, address)
+}
+
+// WARNING: this can leak a goroutine for as long as the underlying Dialer implementation takes to timeout
+// A Conn returned from a successful Dial after the context has been cancelled will be immediately closed.
+func dialContext(ctx context.Context, d Dialer, network, address string) (net.Conn, error) {
+	var (
+		conn net.Conn
+		done = make(chan struct{}, 1)
+		err  error
+	)
+	go func() {
+		conn, err = d.Dial(network, address)
+		close(done)
+		if conn != nil && ctx.Err() != nil {
+			conn.Close()
+		}
+	}()
+	select {
+	case <-ctx.Done():
+		err = ctx.Err()
+	case <-done:
+	}
+	return conn, err
+}

vendor/golang.org/x/net/proxy/direct.go

@@ -0,0 +1,31 @@
+// Copyright 2011 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package proxy
+
+import (
+	"context"
+	"net"
+)
+
+type direct struct{}
+
+// Direct implements Dialer by making network connections directly using net.Dial or net.DialContext.
+var Direct = direct{}
+
+var (
+	_ Dialer        = Direct
+	_ ContextDialer = Direct
+)
+
+// Dial directly invokes net.Dial with the supplied parameters.
+func (direct) Dial(network, addr string) (net.Conn, error) {
+	return net.Dial(network, addr)
+}
+
+// DialContext instantiates a net.Dialer and invokes its DialContext receiver with the supplied parameters.
+func (direct) DialContext(ctx context.Context, network, addr string) (net.Conn, error) {
+	var d net.Dialer
+	return d.DialContext(ctx, network, addr)
+}

vendor/golang.org/x/net/proxy/per_host.go

@@ -0,0 +1,155 @@
+// Copyright 2011 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package proxy
+
+import (
+	"context"
+	"net"
+	"strings"
+)
+
+// A PerHost directs connections to a default Dialer unless the host name
+// requested matches one of a number of exceptions.
+type PerHost struct {
+	def, bypass Dialer
+
+	bypassNetworks []*net.IPNet
+	bypassIPs      []net.IP
+	bypassZones    []string
+	bypassHosts    []string
+}
+
+// NewPerHost returns a PerHost Dialer that directs connections to either
+// defaultDialer or bypass, depending on whether the connection matches one of
+// the configured rules.
+func NewPerHost(defaultDialer, bypass Dialer) *PerHost {
+	return &PerHost{
+		def:    defaultDialer,
+		bypass: bypass,
+	}
+}
+
+// Dial connects to the address addr on the given network through either
+// defaultDialer or bypass.
+func (p *PerHost) Dial(network, addr string) (c net.Conn, err error) {
+	host, _, err := net.SplitHostPort(addr)
+	if err != nil {
+		return nil, err
+	}
+
+	return p.dialerForRequest(host).Dial(network, addr)
+}
+
+// DialContext connects to the address addr on the given network through either
+// defaultDialer or bypass.
+func (p *PerHost) DialContext(ctx context.Context, network, addr string) (c net.Conn, err error) {
+	host, _, err := net.SplitHostPort(addr)
+	if err != nil {
+		return nil, err
+	}
+	d := p.dialerForRequest(host)
+	if x, ok := d.(ContextDialer); ok {
+		return x.DialContext(ctx, network, addr)
+	}
+	return dialContext(ctx, d, network, addr)
+}
+
+func (p *PerHost) dialerForRequest(host string) Dialer {
+	if ip := net.ParseIP(host); ip != nil {
+		for _, net := range p.bypassNetworks {
+			if net.Contains(ip) {
+				return p.bypass
+			}
+		}
+		for _, bypassIP := range p.bypassIPs {
+			if bypassIP.Equal(ip) {
+				return p.bypass
+			}
+		}
+		return p.def
+	}
+
+	for _, zone := range p.bypassZones {
+		if strings.HasSuffix(host, zone) {
+			return p.bypass
+		}
+		if host == zone[1:] {
+			// For a zone ".example.com", we match "example.com"
+			// too.
+			return p.bypass
+		}
+	}
+	for _, bypassHost := range p.bypassHosts {
+		if bypassHost == host {
+			return p.bypass
+		}
+	}
+	return p.def
+}
+
+// AddFromString parses a string that contains comma-separated values
+// specifying hosts that should use the bypass proxy. Each value is either an
+// IP address, a CIDR range, a zone (*.example.com) or a host name
+// (localhost). A best effort is made to parse the string and errors are
+// ignored.
+func (p *PerHost) AddFromString(s string) {
+	hosts := strings.Split(s, ",")
+	for _, host := range hosts {
+		host = strings.TrimSpace(host)
+		if len(host) == 0 {
+			continue
+		}
+		if strings.Contains(host, "/") {
+			// We assume that it's a CIDR address like 127.0.0.0/8
+			if _, net, err := net.ParseCIDR(host); err == nil {
+				p.AddNetwork(net)
+			}
+			continue
+		}
+		if ip := net.ParseIP(host); ip != nil {
+			p.AddIP(ip)
+			continue
+		}
+		if strings.HasPrefix(host, "*.") {
+			p.AddZone(host[1:])
+			continue
+		}
+		p.AddHost(host)
+	}
+}
+
+// AddIP specifies an IP address that will use the bypass proxy. Note that
+// this will only take effect if a literal IP address is dialed. A connection
+// to a named host will never match an IP.
+func (p *PerHost) AddIP(ip net.IP) {
+	p.bypassIPs = append(p.bypassIPs, ip)
+}
+
+// AddNetwork specifies an IP range that will use the bypass proxy. Note that
+// this will only take effect if a literal IP address is dialed. A connection
+// to a named host will never match.
+func (p *PerHost) AddNetwork(net *net.IPNet) {
+	p.bypassNetworks = append(p.bypassNetworks, net)
+}
+
+// AddZone specifies a DNS suffix that will use the bypass proxy. A zone of
+// "example.com" matches "example.com" and all of its subdomains.
+func (p *PerHost) AddZone(zone string) {
+	if strings.HasSuffix(zone, ".") {
+		zone = zone[:len(zone)-1]
+	}
+	if !strings.HasPrefix(zone, ".") {
+		zone = "." + zone
+	}
+	p.bypassZones = append(p.bypassZones, zone)
+}
+
+// AddHost specifies a host name that will use the bypass proxy.
+func (p *PerHost) AddHost(host string) {
+	if strings.HasSuffix(host, ".") {
+		host = host[:len(host)-1]
+	}
+	p.bypassHosts = append(p.bypassHosts, host)
+}

vendor/golang.org/x/net/proxy/proxy.go

@@ -0,0 +1,149 @@
+// Copyright 2011 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// Package proxy provides support for a variety of protocols to proxy network
+// data.
+package proxy // import "golang.org/x/net/proxy"
+
+import (
+	"errors"
+	"net"
+	"net/url"
+	"os"
+	"sync"
+)
+
+// A Dialer is a means to establish a connection.
+// Custom dialers should also implement ContextDialer.
+type Dialer interface {
+	// Dial connects to the given address via the proxy.
+	Dial(network, addr string) (c net.Conn, err error)
+}
+
+// Auth contains authentication parameters that specific Dialers may require.
+type Auth struct {
+	User, Password string
+}
+
+// FromEnvironment returns the dialer specified by the proxy-related
+// variables in the environment and makes underlying connections
+// directly.
+func FromEnvironment() Dialer {
+	return FromEnvironmentUsing(Direct)
+}
+
+// FromEnvironmentUsing returns the dialer specify by the proxy-related
+// variables in the environment and makes underlying connections
+// using the provided forwarding Dialer (for instance, a *net.Dialer
+// with desired configuration).
+func FromEnvironmentUsing(forward Dialer) Dialer {
+	allProxy := allProxyEnv.Get()
+	if len(allProxy) == 0 {
+		return forward
+	}
+
+	proxyURL, err := url.Parse(allProxy)
+	if err != nil {
+		return forward
+	}
+	proxy, err := FromURL(proxyURL, forward)
+	if err != nil {
+		return forward
+	}
+
+	noProxy := noProxyEnv.Get()
+	if len(noProxy) == 0 {
+		return proxy
+	}
+
+	perHost := NewPerHost(proxy, forward)
+	perHost.AddFromString(noProxy)
+	return perHost
+}
+
+// proxySchemes is a map from URL schemes to a function that creates a Dialer
+// from a URL with such a scheme.
+var proxySchemes map[string]func(*url.URL, Dialer) (Dialer, error)
+
+// RegisterDialerType takes a URL scheme and a function to generate Dialers from
+// a URL with that scheme and a forwarding Dialer. Registered schemes are used
+// by FromURL.
+func RegisterDialerType(scheme string, f func(*url.URL, Dialer) (Dialer, error)) {
+	if proxySchemes == nil {
+		proxySchemes = make(map[string]func(*url.URL, Dialer) (Dialer, error))
+	}
+	proxySchemes[scheme] = f
+}
+
+// FromURL returns a Dialer given a URL specification and an underlying
+// Dialer for it to make network requests.
+func FromURL(u *url.URL, forward Dialer) (Dialer, error) {
+	var auth *Auth
+	if u.User != nil {
+		auth = new(Auth)
+		auth.User = u.User.Username()
+		if p, ok := u.User.Password(); ok {
+			auth.Password = p
+		}
+	}
+
+	switch u.Scheme {
+	case "socks5", "socks5h":
+		addr := u.Hostname()
+		port := u.Port()
+		if port == "" {
+			port = "1080"
+		}
+		return SOCKS5("tcp", net.JoinHostPort(addr, port), auth, forward)
+	}
+
+	// If the scheme doesn't match any of the built-in schemes, see if it
+	// was registered by another package.
+	if proxySchemes != nil {
+		if f, ok := proxySchemes[u.Scheme]; ok {
+			return f(u, forward)
+		}
+	}
+
+	return nil, errors.New("proxy: unknown scheme: " + u.Scheme)
+}
+
+var (
+	allProxyEnv = &envOnce{
+		names: []string{"ALL_PROXY", "all_proxy"},
+	}
+	noProxyEnv = &envOnce{
+		names: []string{"NO_PROXY", "no_proxy"},
+	}
+)
+
+// envOnce looks up an environment variable (optionally by multiple
+// names) once. It mitigates expensive lookups on some platforms
+// (e.g. Windows).
+// (Borrowed from net/http/transport.go)
+type envOnce struct {
+	names []string
+	once  sync.Once
+	val   string
+}
+
+func (e *envOnce) Get() string {
+	e.once.Do(e.init)
+	return e.val
+}
+
+func (e *envOnce) init() {
+	for _, n := range e.names {
+		e.val = os.Getenv(n)
+		if e.val != "" {
+			return
+		}
+	}
+}
+
+// reset is used by tests
+func (e *envOnce) reset() {
+	e.once = sync.Once{}
+	e.val = ""
+}

vendor/golang.org/x/net/proxy/socks5.go

@@ -0,0 +1,42 @@
+// Copyright 2011 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package proxy
+
+import (
+	"context"
+	"net"
+
+	"golang.org/x/net/internal/socks"
+)
+
+// SOCKS5 returns a Dialer that makes SOCKSv5 connections to the given
+// address with an optional username and password.
+// See RFC 1928 and RFC 1929.
+func SOCKS5(network, address string, auth *Auth, forward Dialer) (Dialer, error) {
+	d := socks.NewDialer(network, address)
+	if forward != nil {
+		if f, ok := forward.(ContextDialer); ok {
+			d.ProxyDial = func(ctx context.Context, network string, address string) (net.Conn, error) {
+				return f.DialContext(ctx, network, address)
+			}
+		} else {
+			d.ProxyDial = func(ctx context.Context, network string, address string) (net.Conn, error) {
+				return dialContext(ctx, forward, network, address)
+			}
+		}
+	}
+	if auth != nil {
+		up := socks.UsernamePassword{
+			Username: auth.User,
+			Password: auth.Password,
+		}
+		d.AuthMethods = []socks.AuthMethod{
+			socks.AuthMethodNotRequired,
+			socks.AuthMethodUsernamePassword,
+		}
+		d.Authenticate = up.Authenticate
+	}
+	return d, nil
+}

vendor/modules.txt

@@ -71,6 +71,8 @@ golang.org/x/mod/semver
 golang.org/x/net/html
 golang.org/x/net/html/atom
 golang.org/x/net/idna
+golang.org/x/net/internal/socks
+golang.org/x/net/proxy
 golang.org/x/net/websocket
 # golang.org/x/sys v0.7.0
 ## explicit; go 1.17