serv/mox
1
0
Fork 0
mirror of https://github.com/mjl-/mox.git synced 2024-12-26 08:23:48 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

better handling of outgoing tls reports to recipient domains vs hosts

Browse source 
based on discussion on uta mailing list. it seems the intention of the tlsrpt
is to only send reports to recipient domains. but i was able to interpret the
tlsrpt rfc as sending reports to mx hosts too ("policy domain", and because it
makes sense given how DANE works per MX host, not recipient domain). this
change makes the behaviour of outgoing reports to recipient domains work more
in line with expectations most folks may have about tls reporting (i.e. also
include per-mx host tlsa policies in the report). this also keeps reports to mx
hosts working, and makes them more useful by including the recipient domains of
affected deliveries.
This commit is contained in:
Mechiel Lukkien 2023-11-20 11:31:46 +01:00
parent e5f77a0411
commit 73a2a09711
No known key found for this signature in database
8 changed files with 466 additions and 125 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
config/config.go
View file

@ -64,11 +64,13 @@ type Static struct {
ParsedLocalpart smtp.Localpart `sconf:"-"`
} `sconf:"optional" sconf-doc:"Destination for per-host TLS reports (TLSRPT). TLS reports can be per recipient domain (for MTA-STS), or per MX host (for DANE). The per-domain TLS reporting configuration is in domains.conf. This is the TLS reporting configuration for this host. If absent, no host-based TLSRPT address is configured, and no host TLSRPT DNS record is suggested."`
InitialMailboxes InitialMailboxes `sconf:"optional" sconf-doc:"Mailboxes to create for new accounts. Inbox is always created. Mailboxes can be given a 'special-use' role, which are understood by most mail clients. If absent/empty, the following mailboxes are created: Sent, Archive, Trash, Drafts and Junk."`
DefaultMailboxes []string `sconf:"optional" sconf-doc:"Deprecated in favor of InitialMailboxes. Mailboxes to create when adding an account. Inbox is always created. If no mailboxes are specified, the following are automatically created: Sent, Archive, Trash, Drafts and Junk."`
Transports map[string]Transport `sconf:"optional" sconf-doc:"Transport are mechanisms for delivering messages. Transports can be referenced from Routes in accounts, domains and the global configuration. There is always an implicit/fallback delivery transport doing direct delivery with SMTP from the outgoing message queue. Transports are typically only configured when using smarthosts, i.e. when delivering through another SMTP server. Zero or one transport methods must be set in a transport, never multiple. When using an external party to send email for a domain, keep in mind you may have to add their IP address to your domain's SPF record, and possibly additional DKIM records."`
NoOutgoingDMARCReports bool `sconf:"optional" sconf-doc:"Do not send DMARC reports (aggregate only). By default, aggregate reports on DMARC evaluations are sent to domains if their DMARC policy requests them. Reports are sent at whole hours, with a minimum of 1 hour and maximum of 24 hours, rounded up so a whole number of intervals cover 24 hours, aligned at whole days in UTC."`
NoOutgoingTLSReports bool `sconf:"optional" sconf-doc:"Do not send TLS reports. By default, reports about successful and failed SMTP STARTTLS connections are sent to domains if their TLSRPT DNS record requests them. Reports covering a 24 hour UTC interval are sent daily."`
InitialMailboxes InitialMailboxes `sconf:"optional" sconf-doc:"Mailboxes to create for new accounts. Inbox is always created. Mailboxes can be given a 'special-use' role, which are understood by most mail clients. If absent/empty, the following mailboxes are created: Sent, Archive, Trash, Drafts and Junk."`
DefaultMailboxes []string `sconf:"optional" sconf-doc:"Deprecated in favor of InitialMailboxes. Mailboxes to create when adding an account. Inbox is always created. If no mailboxes are specified, the following are automatically created: Sent, Archive, Trash, Drafts and Junk."`
Transports map[string]Transport `sconf:"optional" sconf-doc:"Transport are mechanisms for delivering messages. Transports can be referenced from Routes in accounts, domains and the global configuration. There is always an implicit/fallback delivery transport doing direct delivery with SMTP from the outgoing message queue. Transports are typically only configured when using smarthosts, i.e. when delivering through another SMTP server. Zero or one transport methods must be set in a transport, never multiple. When using an external party to send email for a domain, keep in mind you may have to add their IP address to your domain's SPF record, and possibly additional DKIM records."`
// Awkward naming of fields to get intended default behaviour for zero values.
NoOutgoingDMARCReports bool `sconf:"optional" sconf-doc:"Do not send DMARC reports (aggregate only). By default, aggregate reports on DMARC evaluations are sent to domains if their DMARC policy requests them. Reports are sent at whole hours, with a minimum of 1 hour and maximum of 24 hours, rounded up so a whole number of intervals cover 24 hours, aligned at whole days in UTC. Reports are sent from the postmaster@<mailhostname> address."`
NoOutgoingTLSReports bool `sconf:"optional" sconf-doc:"Do not send TLS reports. By default, reports about failed SMTP STARTTLS connections and related MTA-STS/DANE policies are sent to domains if their TLSRPT DNS record requests them. Reports covering a 24 hour UTC interval are sent daily. Reports are sent from the postmaster address of the configured domain the mailhostname is in. If there is no such domain, or it does not have DKIM configured, no reports are sent."`
OutgoingTLSReportsForAllSuccess bool `sconf:"optional" sconf-doc:"Also send TLS reports if there were no SMTP STARTTLS connection failures. By default, reports are only sent when at least one failure occurred. If a report is sent, it does always include the successful connection counts as well."`
// All IPs that were explicitly listen on for external SMTP. Only set when there
// are no unspecified external SMTP listeners and there is at most one for IPv4 and

18
config/doc.go
View file

@ -570,14 +570,24 @@ describe-static" and "mox config describe-domains":
# DMARC evaluations are sent to domains if their DMARC policy requests them.
# Reports are sent at whole hours, with a minimum of 1 hour and maximum of 24
# hours, rounded up so a whole number of intervals cover 24 hours, aligned at
# whole days in UTC. (optional)
# whole days in UTC. Reports are sent from the postmaster@<mailhostname> address.
# (optional)
NoOutgoingDMARCReports: false
# Do not send TLS reports. By default, reports about successful and failed SMTP
# STARTTLS connections are sent to domains if their TLSRPT DNS record requests
# them. Reports covering a 24 hour UTC interval are sent daily. (optional)
# Do not send TLS reports. By default, reports about failed SMTP STARTTLS
# connections and related MTA-STS/DANE policies are sent to domains if their
# TLSRPT DNS record requests them. Reports covering a 24 hour UTC interval are
# sent daily. Reports are sent from the postmaster address of the configured
# domain the mailhostname is in. If there is no such domain, or it does not have
# DKIM configured, no reports are sent. (optional)
NoOutgoingTLSReports: false
# Also send TLS reports if there were no SMTP STARTTLS connection failures. By
# default, reports are only sent when at least one failure occurred. If a report
# is sent, it does always include the successful connection counts as well.
# (optional)
OutgoingTLSReportsForAllSuccess: false
# domains.conf
# NOTE: This config file is in 'sconf' format. Indent with tabs. Comments must be

52
tlsrptdb/result.go
View file

@ -19,26 +19,24 @@ import (
type TLSResult struct {
ID int64
// Domain with TLSRPT DNS record, with addresses that will receive reports. Either
// a recipient domain (for MTA-STS policies) or an (MX) host (for DANE policies).
// Unicode.
// Domain potentially with TLSRPT DNS record, with addresses that will receive
// reports. Either a recipient domain (for MTA-STS policies) or an (MX) host (for
// DANE policies). Unicode.
PolicyDomain string `bstore:"unique PolicyDomain+DayUTC+RecipientDomain,nonzero"`
// DayUTC is of the form yyyymmdd.
DayUTC string `bstore:"nonzero"`
// We send per 24h UTC-aligned days. ../rfc/8460:474
// Reports are sent per policy domain. When delivering a message to a recipient
// domain, we can get multiple TLSResults, typically one for MTA-STS, and one or
// more for DANE (one for each MX target, or actually TLSA base domain). We track
// recipient domain so we can display successes/failures for delivery of messages
// to a recipient domain in the admin pages. Unicode.
// Reports are sent per recipient domain and per MX host. For reports to a
// recipient domain, we type send a result for MTA-STS and one or more MX host
// (DANE) results. Unicode.
RecipientDomain string `bstore:"index,nonzero"`
Created time.Time `bstore:"default now"`
Updated time.Time `bstore:"default now"`
IsHost bool // Result is for host (e.g. DANE), not recipient domain (e.g. MTA-STS).
IsHost bool // Result is for MX host (DANE), not recipient domain (MTA-STS).
// Whether to send a report. TLS results for delivering messages with TLS reports
// will be recorded, but will not cause a report to be sent.
@ -47,6 +45,17 @@ type TLSResult struct {
// but presumably that's to prevent mail servers sending a report every day once
// they start.
// Set after sending to recipient domain, before sending results to policy domain
// (after which the record is removed).
SentToRecipientDomain bool
// Reporting addresses from the recipient domain TLSRPT record, not necessarily
// those we sent to (e.g. due to failure). Used to leave results to MX target
// (DANE) policy domains out that were already sent in the report to the recipient
// domain, so we don't report twice.
RecipientDomainReportingAddresses []string
// Set after sending report to policy domain.
SentToPolicyDomain bool
// Results is updated for each TLS attempt.
Results []tlsrpt.Result
}
@ -149,7 +158,7 @@ func Results(ctx context.Context) ([]TLSResult, error) {
return bstore.QueryDB[TLSResult](ctx, db).SortAsc("PolicyDomain", "DayUTC", "RecipientDomain").List()
}
// ResultsPolicyDomain returns all TLSResults for a policy domain, potentially for
// ResultsDomain returns all TLSResults for a policy domain, potentially for
// multiple days.
func ResultsPolicyDomain(ctx context.Context, policyDomain dns.Domain) ([]TLSResult, error) {
db, err := resultDB(ctx)
@ -160,6 +169,17 @@ func ResultsPolicyDomain(ctx context.Context, policyDomain dns.Domain) ([]TLSRes
return bstore.QueryDB[TLSResult](ctx, db).FilterNonzero(TLSResult{PolicyDomain: policyDomain.Name()}).SortAsc("DayUTC", "RecipientDomain").List()
}
// ResultsRecipientDomain returns all TLSResults for a recipient domain,
// potentially for multiple days.
func ResultsRecipientDomain(ctx context.Context, recipientDomain dns.Domain) ([]TLSResult, error) {
db, err := resultDB(ctx)
if err != nil {
return nil, err
}
return bstore.QueryDB[TLSResult](ctx, db).FilterNonzero(TLSResult{RecipientDomain: recipientDomain.Name()}).SortAsc("DayUTC", "PolicyDomain").List()
}
// RemoveResultsPolicyDomain removes all TLSResults for the policy domain on the
// day from the database.
func RemoveResultsPolicyDomain(ctx context.Context, policyDomain dns.Domain, dayUTC string) error {
@ -172,6 +192,18 @@ func RemoveResultsPolicyDomain(ctx context.Context, policyDomain dns.Domain, day
return err
}
// RemoveResultsRecipientDomain removes all TLSResults for the recipient domain on
// the day from the database.
func RemoveResultsRecipientDomain(ctx context.Context, recipientDomain dns.Domain, dayUTC string) error {
db, err := resultDB(ctx)
if err != nil {
return err
}
_, err = bstore.QueryDB[TLSResult](ctx, db).FilterNonzero(TLSResult{RecipientDomain: recipientDomain.Name(), DayUTC: dayUTC}).Delete()
return err
}
// SuppressAdd adds an address to the suppress list.
func SuppressAdd(ctx context.Context, ba *TLSRPTSuppressAddress) error {
db, err := resultDB(ctx)

241
tlsrptsend/send.go
View file

@ -29,6 +29,8 @@ import (
"sync"
"time"
"golang.org/x/exp/slices"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
@ -162,27 +164,31 @@ func sendReports(ctx context.Context, log *mlog.Log, resolver dns.Resolver, db *
policyDomain string
dayUTC string
}
destDomains := map[key]bool{}
// Gather all policy domains we plan to send to.
rcptDoms := map[key]bool{} // Results where recipient domain is equal to policy domain, regardless of IsHost.
nonRcptDoms := map[key]bool{} // MX domains (without those that are also recipient domains).
var nsend int
q := bstore.QueryDB[tlsrptdb.TLSResult](ctx, db)
q.FilterLessEqual("DayUTC", dayUTC)
q.SortAsc("PolicyDomain", "DayUTC", "RecipientDomain") // Sort for testability.
err := q.ForEach(func(e tlsrptdb.TLSResult) error {
k := key{e.PolicyDomain, dayUTC}
if e.SendReport && !destDomains[k] {
doms := rcptDoms
if e.PolicyDomain != e.RecipientDomain {
doms = nonRcptDoms
}
k := key{e.PolicyDomain, e.DayUTC}
if e.SendReport && !doms[k] {
nsend++
}
destDomains[k] = destDomains[k] || e.SendReport
doms[k] = doms[k] || e.SendReport
return nil
})
if err != nil {
return fmt.Errorf("looking for domains to send tls reports to: %v", err)
}
// Send report to each domain. We stretch sending over 4 hours, but only if there
// are quite a few message. ../rfc/8460:479
// Stretch sending reports over max 4 hours, but only if there are quite a few
// messages. ../rfc/8460:479
between := 4 * time.Hour
if nsend > 0 {
between = between / time.Duration(nsend)
@ -194,61 +200,86 @@ func sendReports(ctx context.Context, log *mlog.Log, resolver dns.Resolver, db *
var wg sync.WaitGroup
var n int
for k, send := range destDomains {
// Cleanup results for domain that doesn't need to get a report (e.g. for TLS
// connections that were the result of delivering TLSRPT messages).
if !send {
removeResults(ctx, log, db, k.policyDomain, k.dayUTC)
continue
}
if n > 0 {
ok := sleepBetween(ctx, between)
if !ok {
return nil
remove := map[key]struct{}{}
var removeMutex sync.Mutex
sendDomains := func(isRcptDom bool, doms map[key]bool) {
for k, send := range doms {
if !send {
removeMutex.Lock()
remove[k] = struct{}{}
removeMutex.Unlock()
continue
}
}
n++
// In goroutine, so our timing stays independent of how fast we process.
wg.Add(1)
go func(policyDomain string, dayUTC string) {
defer func() {
// In case of panic don't take the whole program down.
x := recover()
if x != nil {
log.Error("unhandled panic in tlsrptsend sendReports", mlog.Field("panic", x))
debug.PrintStack()
metrics.PanicInc(metrics.Tlsrptdb)
if n > 0 {
ok := sleepBetween(ctx, between)
if !ok {
return
}
}()
defer wg.Done()
rlog := log.WithCid(mox.Cid()).Fields(mlog.Field("policydomain", policyDomain), mlog.Field("daytutc", dayUTC))
rlog.Info("sending tls report")
if _, err := sendReportDomain(ctx, rlog, resolver, db, endTimeUTC, policyDomain, dayUTC); err != nil {
rlog.Errorx("sending tls report to domain", err)
metricReportError.Inc()
}
}(k.policyDomain, k.dayUTC)
n++
// In goroutine, so our timing stays independent of how fast we process.
wg.Add(1)
go func(k key) {
defer func() {
// In case of panic don't take the whole program down.
x := recover()
if x != nil {
log.Error("unhandled panic in tlsrptsend sendReports", mlog.Field("panic", x))
debug.PrintStack()
metrics.PanicInc(metrics.Tlsrptdb)
}
}()
defer wg.Done()
rlog := log.WithCid(mox.Cid()).Fields(mlog.Field("policydomain", k.policyDomain), mlog.Field("daytutc", k.dayUTC), mlog.Field("isrcptdom", isRcptDom))
rlog.Info("looking to send tls report for domain")
cleanup, err := sendReportDomain(ctx, rlog, resolver, db, endTimeUTC, isRcptDom, k.policyDomain, k.dayUTC)
if err != nil {
rlog.Errorx("sending tls report to domain", err)
metricReportError.Inc()
}
if cleanup {
removeMutex.Lock()
defer removeMutex.Unlock()
remove[k] = struct{}{}
}
}(k)
}
}
// We send to recipient domains first. That will store the reporting addresses for
// the recipient domains, which are used when sending to nonRcptDoms to potentially
// skip sending a duplicate report.
sendDomains(true, rcptDoms)
wg.Wait()
sendDomains(false, nonRcptDoms)
wg.Wait()
return nil
}
// Remove all records that have been processed.
err = db.Write(ctx, func(tx *bstore.Tx) error {
for k := range remove {
q := bstore.QueryTx[tlsrptdb.TLSResult](tx)
q.FilterNonzero(tlsrptdb.TLSResult{PolicyDomain: k.policyDomain, DayUTC: k.dayUTC})
_, err := q.Delete()
if err != nil {
return err
}
}
return nil
})
log.Check(err, "cleaning up tls results in database")
func removeResults(ctx context.Context, log *mlog.Log, db *bstore.DB, policyDomain string, dayUTC string) {
q := bstore.QueryDB[tlsrptdb.TLSResult](ctx, db)
q.FilterNonzero(tlsrptdb.TLSResult{PolicyDomain: policyDomain, DayUTC: dayUTC})
_, err := q.Delete()
log.Check(err, "removing tls results from database")
return nil
}
// replaceable for testing.
var queueAdd = queue.Add
func sendReportDomain(ctx context.Context, log *mlog.Log, resolver dns.Resolver, db *bstore.DB, endUTC time.Time, policyDomain, dayUTC string) (cleanup bool, rerr error) {
func sendReportDomain(ctx context.Context, log *mlog.Log, resolver dns.Resolver, db *bstore.DB, endUTC time.Time, isRcptDom bool, policyDomain, dayUTC string) (cleanup bool, rerr error) {
polDom, err := dns.ParseDomain(policyDomain)
if err != nil {
return false, fmt.Errorf("parsing policy domain for sending tls reports: %v", err)
@ -287,9 +318,8 @@ func sendReportDomain(ctx context.Context, log *mlog.Log, resolver dns.Resolver,
defer func() {
if !cleanup || tempError {
cleanup = false
log.Debug("not cleaning up results after attempting to send tls report")
} else {
removeResults(ctx, log, db, policyDomain, dayUTC)
}
}()
@ -305,6 +335,7 @@ func sendReportDomain(ctx context.Context, log *mlog.Log, resolver dns.Resolver,
}
var recipients []message.NameAddress
var recipientStrs []string
for _, l := range record.RUAs {
for _, s := range l {
@ -321,6 +352,7 @@ func sendReportDomain(ctx context.Context, log *mlog.Log, resolver dns.Resolver,
continue
}
recipients = append(recipients, message.NameAddress{Address: addr})
recipientStrs = append(recipientStrs, string(s))
} else if u.Scheme == "https" {
// Although "report" is ambiguous and could mean both only the JSON data or an
// entire message (including DKIM-Signature) with the JSON data, it appears the
@ -346,10 +378,12 @@ func sendReportDomain(ctx context.Context, log *mlog.Log, resolver dns.Resolver,
return true, nil
}
log.Info("sending tlsrpt report")
q := bstore.QueryDB[tlsrptdb.TLSResult](ctx, db)
q.FilterNonzero(tlsrptdb.TLSResult{PolicyDomain: policyDomain, DayUTC: dayUTC})
if isRcptDom {
q.FilterNonzero(tlsrptdb.TLSResult{RecipientDomain: policyDomain, DayUTC: dayUTC})
} else {
q.FilterNonzero(tlsrptdb.TLSResult{PolicyDomain: policyDomain, DayUTC: dayUTC})
}
tlsResults, err := q.List()
if err != nil {
return true, fmt.Errorf("get tls results from database: %v", err)
@ -360,6 +394,13 @@ func sendReportDomain(ctx context.Context, log *mlog.Log, resolver dns.Resolver,
return true, fmt.Errorf("no tls results found")
}
// Stop if we already sent a report for this destination.
for _, r := range tlsResults {
if r.PolicyDomain == r.RecipientDomain && (isRcptDom && r.SentToRecipientDomain || !isRcptDom && r.SentToPolicyDomain) {
return true, nil
}
}
beginUTC := endUTC.Add(-24 * time.Hour)
report := tlsrpt.Report{
@ -370,14 +411,67 @@ func sendReportDomain(ctx context.Context, log *mlog.Log, resolver dns.Resolver,
},
ContactInfo: "postmaster@" + fromDom.ASCII,
// todo spec: ../rfc/8460:968 ../rfc/8460:1772 ../rfc/8460:691 subject header assumes a report-id in the form of a msg-id, but example and report-id json field explanation allows free-form report-id's (assuming we're talking about the same report-id here).
ReportID: endUTC.Format("20060102") + "." + polDom.ASCII + "@" + fromDom.ASCII,
ReportID: endUTC.Add(-12*time.Hour).Format("20060102") + "." + polDom.ASCII + "@" + fromDom.ASCII,
}
rcptDomAddresses := map[string][]string{}
for _, tlsResult := range tlsResults {
rcptDomAddresses[tlsResult.RecipientDomain] = tlsResult.RecipientDomainReportingAddresses
}
// Merge all results into this report.
for _, tlsResult := range tlsResults {
// If we are sending to a recipient domain, we include all relevant policy domains,
// so possibly multiple MX hosts (with DANE policies). That means we may be sending
// multiple "no-policy-found" results (1 for sts and 0 or more for mx hosts). An
// explicit no-sts or no-tlsa would make these less ambiguous, but the
// policy-domain's will make clear which is the MX and which is the recipient
// domain. Only for recipient domains with an MX target equal to the recipient host
// could it be confusing.
// If we are sending to MX targets (that aren't recipient domains), we mention the
// affected recipient domains as policy-domain while keeping the original policy
// domain (MX target) in the "mx-host" field. This behaviour isn't in the RFC, but
// seems useful to give MX operators insight into the recipient domains affected.
// We also won't include results for a recipient domain if its TLSRPT policy has
// the same reporting addresses as the MX target TLSRPT policy.
for i, tlsResult := range tlsResults {
if !isRcptDom {
if slices.Equal(rcptDomAddresses[tlsResult.RecipientDomain], recipientStrs) {
continue
}
for j, r := range tlsResult.Results {
if tlsResult.IsHost {
tlsResults[i].Results[j].Policy.MXHost = []string{r.Policy.Domain}
}
tlsResults[i].Results[j].Policy.Domain = tlsResult.RecipientDomain
}
}
report.Merge(tlsResult.Results...)
}
// We may not have any results left, i.e. when this is an MX target and we already
// sent all results in the report to the recipient domain with identical reporting
// addresses.
if len(report.Policies) == 0 {
return true, nil
}
if !mox.Conf.Static.OutgoingTLSReportsForAllSuccess {
var haveFailure bool
// Check there is at least one failure. If not, we don't send a report.
for _, r := range report.Policies {
if r.Summary.TotalFailureSessionCount > 0 || len(r.FailureDetails) > 0 {
haveFailure = true
break
}
}
if !haveFailure {
return true, nil
}
}
log.Info("sending tls report")
reportFile, err := store.CreateMessageTemp("tlsreportout")
if err != nil {
return false, fmt.Errorf("creating temporary file for outgoing tls report: %v", err)
@ -440,6 +534,41 @@ Period: %s - %s UTC
}
msgSize := int64(len(msgPrefix)) + msgInfo.Size()
// Already mark the report as sent. If it won't succeed below, it probably won't
// succeed on a later retry either. And if we would fail to mark a report as sent
// after sending it, we may sent duplicates or even get in some kind of sending
// loop.
err = db.Write(ctx, func(tx *bstore.Tx) error {
if isRcptDom {
q := bstore.QueryTx[tlsrptdb.TLSResult](tx)
q.FilterNonzero(tlsrptdb.TLSResult{DayUTC: dayUTC, RecipientDomain: policyDomain})
_, err := q.UpdateNonzero(tlsrptdb.TLSResult{SentToRecipientDomain: true})
if err != nil {
return fmt.Errorf("already marking tls results as sent for recipient domain: %v", err)
}
// Also set reporting addresses for the recipient domain results.
q = bstore.QueryTx[tlsrptdb.TLSResult](tx)
q.FilterNonzero(tlsrptdb.TLSResult{DayUTC: dayUTC, RecipientDomain: policyDomain})
_, err = q.UpdateNonzero(tlsrptdb.TLSResult{RecipientDomainReportingAddresses: recipientStrs})
if err != nil {
return fmt.Errorf("storing recipient domain reporting addresses: %v", err)
}
} else {
q := bstore.QueryTx[tlsrptdb.TLSResult](tx)
q.FilterNonzero(tlsrptdb.TLSResult{DayUTC: dayUTC, PolicyDomain: policyDomain})
_, err := q.UpdateNonzero(tlsrptdb.TLSResult{SentToPolicyDomain: true})
if err != nil {
return fmt.Errorf("already marking tls results as sent for policy domain: %v", err)
}
}
return nil
})
if err != nil {
return false, fmt.Errorf("marking tls results as sent: %v", err)
}
var queued bool
for _, rcpt := range recipients {
// If recipient is on suppression list, we won't queue the reporting message.
q := bstore.QueryDB[tlsrptdb.TLSRPTSuppressAddress](ctx, db)
@ -466,10 +595,12 @@ Period: %s - %s UTC
err = queueAdd(ctx, log, &qm, msgf)
if err != nil {
tempError = true
tempError = !queued
log.Errorx("queueing message with tls report", err)
metricReportError.Inc()
} else {
queued = true
tempError = false
log.Debug("tls report queued", mlog.Field("recipient", rcpt))
metricReport.Inc()
}

182
tlsrptsend/send_test.go
View file

@ -11,6 +11,8 @@ import (
"testing"
"time"
"github.com/mjl-/bstore"
"github.com/mjl-/mox/dns"
"github.com/mjl-/mox/mlog"
"github.com/mjl-/mox/mox-"
@ -238,7 +240,7 @@ func TestSendReports(t *testing.T) {
End: endUTC.Add(-time.Second),
},
ContactInfo: "postmaster@mox.example",
ReportID: endUTC.Format("20060102") + ".sender.example@mox.example",
ReportID: endUTC.Add(-12*time.Hour).Format("20060102") + ".sender.example@mox.example",
Policies: []tlsrpt.Result{
{
Policy: tlsrpt.ResultPolicy{
@ -262,6 +264,29 @@ func TestSendReports(t *testing.T) {
},
},
},
// Includes reports about MX target, for DANE policies.
{
Policy: tlsrpt.ResultPolicy{
Type: tlsrpt.TLSA,
Domain: "mailhost.sender.example",
String: []string{"... tlsa record ..."},
},
Summary: tlsrpt.Summary{
TotalSuccessfulSessionCount: 10,
TotalFailureSessionCount: 1,
},
FailureDetails: []tlsrpt.FailureDetails{
{
ResultType: tlsrpt.ResultValidationFailure,
SendingMTAIP: "1.2.3.4",
ReceivingMXHostname: "mailhost.sender.example",
ReceivingMXHelo: "mailhost.sender.example",
ReceivingIP: "4.3.2.1",
FailedSessionCount: 1,
FailureReasonCode: "dns-extended-error-7-signature-expired",
},
},
},
},
}
report2 := tlsrpt.Report{
@ -271,17 +296,20 @@ func TestSendReports(t *testing.T) {
End: endUTC.Add(-time.Second),
},
ContactInfo: "postmaster@mox.example",
ReportID: endUTC.Format("20060102") + ".mailhost.sender.example@mox.example",
ReportID: endUTC.Add(-12*time.Hour).Format("20060102") + ".mailhost.sender.example@mox.example",
Policies: []tlsrpt.Result{
// The MX target policies are per-recipient domain, so the MX operator can see the
// affected recipient domains.
{
Policy: tlsrpt.ResultPolicy{
Type: tlsrpt.TLSA,
Domain: "mailhost.sender.example",
Domain: "sender.example", // Recipient domain.
String: []string{"... tlsa record ..."},
MXHost: []string{"mailhost.sender.example"}, // Original policy domain.
},
Summary: tlsrpt.Summary{
TotalSuccessfulSessionCount: 20,
TotalFailureSessionCount: 2,
TotalSuccessfulSessionCount: 10,
TotalFailureSessionCount: 1,
},
FailureDetails: []tlsrpt.FailureDetails{
{
@ -290,7 +318,66 @@ func TestSendReports(t *testing.T) {
ReceivingMXHostname: "mailhost.sender.example",
ReceivingMXHelo: "mailhost.sender.example",
ReceivingIP: "4.3.2.1",
FailedSessionCount: 2,
FailedSessionCount: 1,
FailureReasonCode: "dns-extended-error-7-signature-expired",
},
},
},
{
Policy: tlsrpt.ResultPolicy{
Type: tlsrpt.TLSA,
Domain: "sharedsender.example", // Recipient domain.
String: []string{"... tlsa record ..."},
MXHost: []string{"mailhost.sender.example"}, // Original policy domain.
},
Summary: tlsrpt.Summary{
TotalSuccessfulSessionCount: 10,
TotalFailureSessionCount: 1,
},
FailureDetails: []tlsrpt.FailureDetails{
{
ResultType: tlsrpt.ResultValidationFailure,
SendingMTAIP: "1.2.3.4",
ReceivingMXHostname: "mailhost.sender.example",
ReceivingMXHelo: "mailhost.sender.example",
ReceivingIP: "4.3.2.1",
FailedSessionCount: 1,
FailureReasonCode: "dns-extended-error-7-signature-expired",
},
},
},
},
}
report3 := tlsrpt.Report{
OrganizationName: "mox.example",
DateRange: tlsrpt.TLSRPTDateRange{
Start: endUTC.Add(-24 * time.Hour),
End: endUTC.Add(-time.Second),
},
ContactInfo: "postmaster@mox.example",
ReportID: endUTC.Add(-12*time.Hour).Format("20060102") + ".mailhost.sender.example@mox.example",
Policies: []tlsrpt.Result{
// The MX target policies are per-recipient domain, so the MX operator can see the
// affected recipient domains.
{
Policy: tlsrpt.ResultPolicy{
Type: tlsrpt.TLSA,
Domain: "sharedsender.example", // Recipient domain.
String: []string{"... tlsa record ..."},
MXHost: []string{"mailhost.sender.example"}, // Original policy domain.
},
Summary: tlsrpt.Summary{
TotalSuccessfulSessionCount: 10,
TotalFailureSessionCount: 1,
},
FailureDetails: []tlsrpt.FailureDetails{
{
ResultType: tlsrpt.ResultValidationFailure,
SendingMTAIP: "1.2.3.4",
ReceivingMXHostname: "mailhost.sender.example",
ReceivingMXHelo: "mailhost.sender.example",
ReceivingIP: "4.3.2.1",
FailedSessionCount: 1,
FailureReasonCode: "dns-extended-error-7-signature-expired",
},
},
@ -309,8 +396,8 @@ func TestSendReports(t *testing.T) {
sleepBetween = func(ctx context.Context, d time.Duration) (ok bool) { return true }
test := func(results []tlsrptdb.TLSResult, expReports map[string]tlsrpt.Report) {
// t.Helper()
test := func(results []tlsrptdb.TLSResult, expReports map[string][]tlsrpt.Report) {
t.Helper()
mox.Shutdown, mox.ShutdownCancel = context.WithCancel(ctxbg)
@ -319,7 +406,7 @@ func TestSendReports(t *testing.T) {
tcheckf(t, err, "inserting tlsresult")
}
haveReports := map[string]tlsrpt.Report{}
haveReports := map[string][]tlsrpt.Report{}
var mutex sync.Mutex
@ -340,15 +427,7 @@ func TestSendReports(t *testing.T) {
tcheckf(t, err, "parsing generated report message")
addr := qm.Recipient().String()
if _, ok := haveReports[addr]; ok {
t.Fatalf("report for address %s already seen", addr)
} else if expReport, ok := expReports[addr]; !ok {
t.Fatalf("unexpected report for address %s", addr)
} else {
tcompare(t, *report, expReport)
}
haveReports[addr] = *report
haveReports[addr] = append(haveReports[addr], *report)
return nil
}
@ -362,33 +441,76 @@ func TestSendReports(t *testing.T) {
tcompare(t, haveReports, expReports)
// Second loop. Evaluations cleaned, should not result in report messages.
haveReports = map[string]tlsrpt.Report{}
haveReports = map[string][]tlsrpt.Report{}
step <- 0
<-wait
tcompare(t, haveReports, map[string]tlsrpt.Report{})
tcompare(t, haveReports, map[string][]tlsrpt.Report{})
// Caus Start to stop.
mox.ShutdownCancel()
step <- time.Minute
leftover, err := bstore.QueryDB[tlsrptdb.TLSResult](ctxbg, db).List()
tcheckf(t, err, "querying database")
if len(leftover) != 0 {
t.Fatalf("leftover results in database after sending reports: %v", leftover)
}
_, err = bstore.QueryDB[tlsrptdb.TLSResult](ctxbg, db).Delete()
tcheckf(t, err, "cleaning from database")
}
// Multiple results, some are combined into a single report, another result
// generates a separate report to multiple rua's, and the last don't send a report.
expReports := map[string]tlsrpt.Report{
"tls-reports@sender.example": report1,
"tls-reports1@mailhost.sender.example": report2,
"tls-reports2@mailhost.sender.example": report2,
"tls-reports3@mailhost.sender.example": report2,
}
test(tlsResults, expReports)
test(tlsResults, map[string][]tlsrpt.Report{
"tls-reports@sender.example": {report1},
"tls-reports1@mailhost.sender.example": {report2},
"tls-reports2@mailhost.sender.example": {report2},
"tls-reports3@mailhost.sender.example": {report2},
})
// If MX target has same reporting addresses as recipient domain, only recipient
// domain should get a report.
resolver.TXT["_smtp._tls.mailhost.sender.example."] = []string{"v=TLSRPTv1; rua=mailto:tls-reports@sender.example"}
test(tlsResults[:2], map[string][]tlsrpt.Report{
"tls-reports@sender.example": {report1},
})
resolver.TXT["_smtp._tls.sharedsender.example."] = []string{"v=TLSRPTv1; rua=mailto:tls-reports@sender.example"}
test(tlsResults, map[string][]tlsrpt.Report{
"tls-reports@sender.example": {report1, report3},
})
// Suppressed addresses don't get a report.
resolver.TXT["_smtp._tls.mailhost.sender.example."] = []string{"v=TLSRPTv1; rua=mailto:tls-reports1@mailhost.sender.example,mailto:tls-reports2@mailhost.sender.example; rua=mailto:tls-reports3@mailhost.sender.example"}
db.Insert(ctxbg,
&tlsrptdb.TLSRPTSuppressAddress{ReportingAddress: "tls-reports@sender.example", Until: time.Now().Add(-time.Minute)}, // Expired, so ignored.
&tlsrptdb.TLSRPTSuppressAddress{ReportingAddress: "tls-reports1@mailhost.sender.example", Until: time.Now().Add(time.Minute)}, // Still valid.
&tlsrptdb.TLSRPTSuppressAddress{ReportingAddress: "tls-reports3@mailhost.sender.example", Until: time.Now().Add(31 * 24 * time.Hour)}, // Still valid.
)
test(tlsResults, map[string]tlsrpt.Report{
"tls-reports@sender.example": report1,
"tls-reports2@mailhost.sender.example": report2,
test(tlsResults, map[string][]tlsrpt.Report{
"tls-reports@sender.example": {report1},
"tls-reports2@mailhost.sender.example": {report2},
})
// Make reports success-only, ensuring we don't get a report anymore.
for i := range tlsResults {
for j := range tlsResults[i].Results {
tlsResults[i].Results[j].Summary.TotalFailureSessionCount = 0
tlsResults[i].Results[j].FailureDetails = nil
}
}
test(tlsResults, map[string][]tlsrpt.Report{})
// But when we want to send report for all-successful connections, we get reports again.
mox.Conf.Static.OutgoingTLSReportsForAllSuccess = true
for _, report := range []*tlsrpt.Report{&report1, &report2} {
for i := range report.Policies {
report.Policies[i].Summary.TotalFailureSessionCount = 0
report.Policies[i].FailureDetails = nil
}
}
test(tlsResults, map[string][]tlsrpt.Report{
"tls-reports@sender.example": {report1},
"tls-reports2@mailhost.sender.example": {report2},
})
}

18
webadmin/admin.go
View file

@ -2069,10 +2069,15 @@ func (Admin) TLSRPTResults(ctx context.Context) []tlsrptdb.TLSResult {
}
// TLSRPTResultsPolicyDomain returns the TLS results for a domain.
func (Admin) TLSRPTResultsPolicyDomain(ctx context.Context, policyDomain string) (dns.Domain, []tlsrptdb.TLSResult) {
func (Admin) TLSRPTResultsDomain(ctx context.Context, isRcptDom bool, policyDomain string) (dns.Domain, []tlsrptdb.TLSResult) {
dom, err := dns.ParseDomain(policyDomain)
xcheckf(ctx, err, "parsing domain")
if isRcptDom {
results, err := tlsrptdb.ResultsRecipientDomain(ctx, dom)
xcheckf(ctx, err, "get result for recipient domain")
return dom, results
}
results, err := tlsrptdb.ResultsPolicyDomain(ctx, dom)
xcheckf(ctx, err, "get result for policy domain")
return dom, results
@ -2101,12 +2106,17 @@ func (Admin) LookupTLSRPTRecord(ctx context.Context, domain string) (record *TLS
// TLSRPTRemoveResults removes the TLS results for a domain for the given day. If
// day is empty, all results are removed.
func (Admin) TLSRPTRemoveResults(ctx context.Context, domain string, day string) {
func (Admin) TLSRPTRemoveResults(ctx context.Context, isRcptDom bool, domain string, day string) {
dom, err := dns.ParseDomain(domain)
xcheckf(ctx, err, "parsing domain")
err = tlsrptdb.RemoveResultsPolicyDomain(ctx, dom, day)
xcheckf(ctx, err, "removing tls results")
if isRcptDom {
err = tlsrptdb.RemoveResultsRecipientDomain(ctx, dom, day)
xcheckf(ctx, err, "removing tls results")
} else {
err = tlsrptdb.RemoveResultsPolicyDomain(ctx, dom, day)
xcheckf(ctx, err, "removing tls results")
}
}
// TLSRPTSuppressAdd adds a reporting address to the suppress list. Outgoing

26
webadmin/admin.html
View file

@ -1599,7 +1599,7 @@ const tlsrptResults = async () => {
crumblink('TLSRPT', '#tlsrpt'),
'Results',
),
dom.p('Messages are delivered with SMTP with TLS using STARTTLS if supported and/or required by the recipient domain\'s mail server. TLS connections may fail for various reasons, such as mismatching certificate host name, expired certificates or TLS protocol version/cipher suite incompatibilities. Statistics about successful connections and failed connections are tracked. Results can be tracked for recipient domains (for MTA-STS policies), and per MX host (for DANE). A domain/host can publish a TLSRPT DNS record with addresses that should receive TLS reports. Reports are sent every 24 hours. Not all results are enough reason to send a report, but if a report is sent all results are included.'),
dom.p('Messages are delivered with SMTP with TLS using STARTTLS if supported and/or required by the recipient domain\'s mail server. TLS connections may fail for various reasons, such as mismatching certificate host name, expired certificates or TLS protocol version/cipher suite incompatibilities. Statistics about successful connections and failed connections are tracked. Results can be tracked for recipient domains (for MTA-STS policies), and per MX host (for DANE). A domain/host can publish a TLSRPT DNS record with addresses that should receive TLS reports. Reports are sent every 24 hours. Not all results are enough reason to send a report, but if a report is sent all results are included. By default, reports are only sent if a report contains a connection failure. Sending reports about all-successful connections can be configured. Reports sent to recipient domains include the results for its MX hosts, and reports for an MX host reference the recipient domains.'),
dom('table.hover',
dom.thead(
dom.tr(
@ -1611,7 +1611,7 @@ const tlsrptResults = async () => {
dom.th('Success', attr({title: 'Total number of successful connections.'})),
dom.th('Failure', attr({title: 'Total number of failed connection attempts.'})),
dom.th('Failure details', attr({title: 'Total number of details about failures.'})),
dom.th('Send report', attr({title: 'Whether the current results will cause a report to be sent. A report is only sent if the domain has a TLSRPT with reporting addresses configured.'})),
dom.th('Send report', attr({title: 'Whether the current results may cause a report to be sent. To prevent report loops, reports are not sent for TLS connections used to deliver TLS or DMARC reports. Whether a report is eventually sent depends on more factors, such as whether the policy domain has a TLSRPT policy with reporting addresses, and whether TLS connection failures were registered (depending on configuration).'})),
),
),
dom.tbody(
@ -1642,7 +1642,7 @@ const tlsrptResults = async () => {
return dom.tr(
dom.td(r.DayUTC),
dom.td(r.RecipientDomain),
dom.td(dom.a(attr({href: '#tlsrpt/results/'+r.PolicyDomain}), r.PolicyDomain)),
dom.td(dom.a(attr({href: '#tlsrpt/results/'+ (r.RecipientDomain === r.PolicyDomain ? 'rcptdom/' : 'host/') + r.PolicyDomain}), r.PolicyDomain)),
dom.td(r.IsHost ? '✓' : ''),
dom.td(policyTypes.join(', ')),
dom.td(style({textAlign: 'right'}), ''+success),
@ -1752,8 +1752,8 @@ const tlsrptResults = async () => {
)
}
const tlsrptResultsPolicyDomain = async (domain) => {
const [d, tlsresults] = await api.TLSRPTResultsPolicyDomain(domain)
const tlsrptResultsPolicyDomain = async (isrcptdom, domain) => {
const [d, tlsresults] = await api.TLSRPTResultsDomain(isrcptdom, domain)
const recordPromise = api.LookupTLSRPTRecord(domain)
let recordBox
@ -1763,14 +1763,14 @@ const tlsrptResultsPolicyDomain = async (domain) => {
crumblink('Mox Admin', '#'),
crumblink('TLSRPT', '#tlsrpt'),
crumblink('Results', '#tlsrpt/results'),
'Policy domain '+domainString(d),
(isrcptdom ? 'Recipient domain ' : 'Host ') + domainString(d),
),
dom.div(
dom.button('Remove results', async function click(e) {
e.preventDefault()
e.target.disabled = true
try {
await api.TLSRPTRemoveResults(domain, '')
await api.TLSRPTRemoveResults(isrcptdom, domain, '')
window.location.reload() // todo: only clear the table?
} catch (err) {
console.log({err})
@ -1781,14 +1781,14 @@ const tlsrptResultsPolicyDomain = async (domain) => {
}),
),
dom.br(),
dom.div('Fetching TLSRPT DNS record for policy domain...'),
dom.div('Fetching TLSRPT DNS record...'),
recordBox=dom.div(),
dom.br(),
dom.p('Below are the results per day and recipient domain that may be sent in a report.'),
dom.p('Below are the results per day and ' + (isrcptdom ? 'policy' : 'recipient') + ' domain that may be sent in a report.'),
tlsresults.map(tlsresult => [
dom.h2(tlsresult.DayUTC, ' - ', dom.span(attr({title: 'Recipient domain, as used in SMTP MAIL TO, usually based on message To/Cc/Bcc.'}), tlsresult.RecipientDomain)),
dom.h2(tlsresult.DayUTC, ' - ', dom.span(attr({title: 'Recipient domain, as used in SMTP MAIL TO, usually based on message To/Cc/Bcc.'}), isrcptdom ? tlsresult.PolicyDomain : tlsresult.RecipientDomain)),
dom.p(
'Send report (if TLSRPT exists and has address): '+(tlsresult.SendReport ? 'Yes' : 'No'),
'Send report (if TLSRPT policy exists and has address): '+(tlsresult.SendReport ? 'Yes' : 'No'),
dom.br(),
'Report about (MX) host (instead of recipient domain): '+(tlsresult.IsHost ? 'Yes' : 'No'),
),
@ -2811,8 +2811,8 @@ const init = async () => {
await domainTLSRPTID(t[2], parseInt(t[3]))
} else if (h === 'tlsrpt/results') {
await tlsrptResults()
} else if (t[0] == 'tlsrpt' && t[1] == 'results' && t.length === 3) {
await tlsrptResultsPolicyDomain(t[2])
} else if (t[0] == 'tlsrpt' && t[1] == 'results' && (t[2] === 'rcptdom' || t[2] == 'host') && t.length === 4) {
await tlsrptResultsPolicyDomain(t[2] === 'rcptdom', t[3])
} else if (h === 'dmarc') {
await dmarcIndex()
} else if (h === 'dmarc/reports') {

42
webadmin/adminapi.json
View file

@ -914,9 +914,15 @@
]
},
{
"Name": "TLSRPTResultsPolicyDomain",
"Name": "TLSRPTResultsDomain",
"Docs": "TLSRPTResultsPolicyDomain returns the TLS results for a domain.",
"Params": [
{
"Name": "isRcptDom",
"Typewords": [
"bool"
]
},
{
"Name": "policyDomain",
"Typewords": [
@ -977,6 +983,12 @@
"Name": "TLSRPTRemoveResults",
"Docs": "TLSRPTRemoveResults removes the TLS results for a domain for the given day. If\nday is empty, all results are removed.",
"Params": [
{
"Name": "isRcptDom",
"Typewords": [
"bool"
]
},
{
"Name": "domain",
"Typewords": [
@ -4012,7 +4024,7 @@
},
{
"Name": "PolicyDomain",
"Docs": "Domain with TLSRPT DNS record, with addresses that will receive reports. Either a recipient domain (for MTA-STS policies) or an (MX) host (for DANE policies). Unicode.",
"Docs": "Domain potentially with TLSRPT DNS record, with addresses that will receive reports. Either a recipient domain (for MTA-STS policies) or an (MX) host (for DANE policies). Unicode.",
"Typewords": [
"string"
]
@ -4026,7 +4038,7 @@
},
{
"Name": "RecipientDomain",
"Docs": "Reports are sent per policy domain. When delivering a message to a recipient domain, we can get multiple TLSResults, typically one for MTA-STS, and one or more for DANE (one for each MX target, or actually TLSA base domain). We track recipient domain so we can display successes/failures for delivery of messages to a recipient domain in the admin pages. Unicode.",
"Docs": "Reports are sent per recipient domain and per MX host. For reports to a recipient domain, we type send a result for MTA-STS and one or more MX host (DANE) results. Unicode.",
"Typewords": [
"string"
]
@ -4047,7 +4059,7 @@
},
{
"Name": "IsHost",
"Docs": "Result is for host (e.g. DANE), not recipient domain (e.g. MTA-STS).",
"Docs": "Result is for MX host (DANE), not recipient domain (MTA-STS).",
"Typewords": [
"bool"
]
@ -4059,6 +4071,28 @@
"bool"
]
},
{
"Name": "SentToRecipientDomain",
"Docs": "Set after sending to recipient domain, before sending results to policy domain (after which the record is removed).",
"Typewords": [
"bool"
]
},
{
"Name": "RecipientDomainReportingAddresses",
"Docs": "Reporting addresses from the recipient domain TLSRPT record, not necessarily those we sent to (e.g. due to failure). Used to leave results to MX target (DANE) policy domains out that were already sent in the report to the recipient domain, so we don't report twice.",
"Typewords": [
"[]",
"string"
]
},
{
"Name": "SentToPolicyDomain",
"Docs": "Set after sending report to policy domain.",
"Typewords": [
"bool"
]
},
{
"Name": "Results",
"Docs": "Results is updated for each TLS attempt.",