add a webapi and webhooks for a simple http/json-based api
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.
this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.
unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...
matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.
a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.
messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.
suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.
submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".
to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.
admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.
new config options have been introduced. they are editable through the admin
and/or account web interfaces.
the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.
gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.
for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
// Package webhook has data types used for webhooks about incoming and outgoing deliveries.
//
// See package webapi for details about the webapi and webhooks.
//
// Types [Incoming] and [Outgoing] represent the JSON bodies sent in the webhooks.
// New fields may be added in the future, unrecognized fields should be ignored
// when parsing for forward compatibility.
package webhook
import (
"time"
)
// OutgoingEvent is an activity for an outgoing delivery. Either generated by the
// queue, or through an incoming DSN (delivery status notification) message.
type OutgoingEvent string
// note: outgoing hook events are in ../queue/hooks.go, ../mox-/config.go, ../queue.go and ../webapi/gendoc.sh. keep in sync.
// todo: in future have more events: for spam complaints, perhaps mdn's.
const (
// Message was accepted by a next-hop server. This does not necessarily mean the
// message has been delivered in the mailbox of the user.
EventDelivered OutgoingEvent = "delivered"
// Outbound delivery was suppressed because the recipient address is on the
// suppression list of the account, or a simplified/base variant of the address is.
EventSuppressed OutgoingEvent = "suppressed"
// A delivery attempt failed but delivery will be retried again later.
EventDelayed OutgoingEvent = "delayed"
// Delivery of the message failed and will not be tried again. Also see the
// "Suppressing" field of [Outgoing].
EventFailed OutgoingEvent = "failed"
// Message was relayed into a system that does not generate DSNs. Should only
// happen when explicitly requested.
EventRelayed OutgoingEvent = "relayed"
// Message was accepted and is being delivered to multiple recipients (e.g. the
// address was an alias/list), which may generate more DSNs.
EventExpanded OutgoingEvent = "expanded"
// Message was removed from the queue, e.g. canceled by admin/user.
EventCanceled OutgoingEvent = "canceled"
// An incoming message was received that was either a DSN with an unknown event
// type ("action"), or an incoming non-DSN-message was received for the unique
// per-outgoing-message address used for sending.
EventUnrecognized OutgoingEvent = "unrecognized"
)
// Outgoing is the payload sent to webhook URLs for events about outgoing deliveries.
type Outgoing struct {
Version int // Format of hook, currently 0.
Event OutgoingEvent // Type of outgoing delivery event.
DSN bool // If this event was triggered by a delivery status notification message (DSN).
Suppressing bool // If true, this failure caused the address to be added to the suppression list.
QueueMsgID int64 // ID of message in queue.
FromID string // As used in MAIL FROM, can be empty, for incoming messages.
MessageID string // From Message-Id header, as set by submitter or us, with enclosing <>.
Subject string // Of original message.
WebhookQueued time . Time // When webhook was first queued for delivery.
SMTPCode int // Optional, for errors only, e.g. 451, 550. See package smtp for definitions.
SMTPEnhancedCode string // Optional, for errors only, e.g. 5.1.1.
Error string // Error message while delivering, or from DSN from remote, if any.
Extra map [ string ] string // Extra fields set for message during submit, through webapi call or through X-Mox-Extra-* headers during SMTP submission.
}
// Incoming is the data sent to a webhook for incoming deliveries over SMTP.
type Incoming struct {
Version int // Format of hook, currently 0.
// Message "From" header, typically has one address.
From [ ] NameAddress
To [ ] NameAddress
CC [ ] NameAddress
BCC [ ] NameAddress // Often empty, even if you were a BCC recipient.
// Optional Reply-To header, typically absent or with one address.
ReplyTo [ ] NameAddress
Subject string
// Of Message-Id header, typically of the form "<random@hostname>", includes <>.
MessageID string
// Optional, the message-id this message is a reply to. Includes <>.
InReplyTo string
// Optional, zero or more message-ids this message is a reply/forward/related to.
// The last entry is the most recent/immediate message this is a reply to. Earlier
// entries are the parents in a thread. Values include <>.
References [ ] string
// Time in "Date" message header, can be different from time received.
Date * time . Time
// Contents of text/plain and/or text/html part (if any), with "\n" line-endings,
// converted from "\r\n". Values are truncated to 1MB (1024*1024 bytes). Use webapi
// MessagePartGet to retrieve the full part data.
Text string
HTML string
// No files, can be large.
Structure Structure // Parsed form of MIME message.
Meta IncomingMeta // Details about message in storage, and SMTP transaction details.
}
type IncomingMeta struct {
MsgID int64 // ID of message in storage, and to use in webapi calls like MessageGet.
MailFrom string // Address used during SMTP "MAIL FROM" command.
MailFromValidated bool // Whether SMTP MAIL FROM address was SPF-validated.
MsgFromValidated bool // Whether address in message "From"-header was DMARC(-like) validated.
RcptTo string // SMTP RCPT TO address used in SMTP.
DKIMVerifiedDomains [ ] string // Verified domains from DKIM-signature in message. Can be different domain than used in addresses.
RemoteIP string // Where the message was delivered from.
Received time . Time // When message was received, may be different from the Date header.
MailboxName string // Mailbox where message was delivered to, based on configured rules. Defaults to "Inbox".
// Whether this message was automated and should not receive automated replies.
// E.g. out of office or mailing list messages.
Automated bool
}
type NameAddress struct {
Name string // Optional, human-readable "display name" of the addressee.
Address string // Required, email address.
}
type Structure struct {
2024-12-06 16:19:39 +03:00
ContentType string // Lower case, e.g. text/plain.
ContentTypeParams map [ string ] string // Lower case keys, original case values, e.g. {"charset": "UTF-8"}.
ContentID string // Can be empty. Otherwise, should be a value wrapped in <>'s. For use in HTML, referenced as URI `cid:...`.
ContentDisposition string // Lower-case value, e.g. "attachment", "inline" or empty when absent. Without the key/value header parameters.
Filename string // Filename for this part, based on "filename" parameter from Content-Disposition, or "name" from Content-Type after decoding.
DecodedSize int64 // Size of content after decoding content-transfer-encoding. For text and HTML parts, this can be larger than the data returned since this size includes \r\n line endings.
Parts [ ] Structure // Subparts of a multipart message, possibly recursive.
add a webapi and webhooks for a simple http/json-based api
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.
this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.
unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...
matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.
a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.
messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.
suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.
submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".
to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.
admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.
new config options have been introduced. they are editable through the admin
and/or account web interfaces.
the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.
gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.
for issue #31 by cuu508
2024-04-15 22:49:02 +03:00
}