mirror of
https://github.com/mjl-/mox.git
synced 2024-12-26 16:33:47 +03:00
09fcc49223
for applications to compose/send messages, receive delivery feedback, and maintain suppression lists. this is an alternative to applications using a library to compose messages, submitting those messages using smtp, and monitoring a mailbox with imap for DSNs, which can be processed into the equivalent of suppression lists. but you need to know about all these standards/protocols and find libraries. by using the webapi & webhooks, you just need a http & json library. unfortunately, there is no standard for these kinds of api, so mox has made up yet another one... matching incoming DSNs about deliveries to original outgoing messages requires keeping history of "retired" messages (delivered from the queue, either successfully or failed). this can be enabled per account. history is also useful for debugging deliveries. we now also keep history of each delivery attempt, accessible while still in the queue, and kept when a message is retired. the queue webadmin pages now also have pagination, to show potentially large history. a queue of webhook calls is now managed too. failures are retried similar to message deliveries. webhooks can also be saved to the retired list after completing. also configurable per account. messages can be sent with a "unique smtp mail from" address. this can only be used if the domain is configured with a localpart catchall separator such as "+". when enabled, a queued message gets assigned a random "fromid", which is added after the separator when sending. when DSNs are returned, they can be related to previously sent messages based on this fromid. in the future, we can implement matching on the "envid" used in the smtp dsn extension, or on the "message-id" of the message. using a fromid can be triggered by authenticating with a login email address that is configured as enabling fromid. suppression lists are automatically managed per account. if a delivery attempt results in certain smtp errors, the destination address is added to the suppression list. future messages queued for that recipient will immediately fail without a delivery attempt. suppression lists protect your mail server reputation. submitted messages can carry "extra" data through the queue and webhooks for outgoing deliveries. through webapi as a json object, through smtp submission as message headers of the form "x-mox-extra-<key>: value". to make it easy to test webapi/webhooks locally, the "localserve" mode actually puts messages in the queue. when it's time to deliver, it still won't do a full delivery attempt, but just delivers to the sender account. unless the recipient address has a special form, simulating a failure to deliver. admins now have more control over the queue. "hold rules" can be added to mark newly queued messages as "on hold", pausing delivery. rules can be about certain sender or recipient domains/addresses, or apply to all messages pausing the entire queue. also useful for (local) testing. new config options have been introduced. they are editable through the admin and/or account web interfaces. the webapi http endpoints are enabled for newly generated configs with the quickstart, and in localserve. existing configurations must explicitly enable the webapi in mox.conf. gopherwatch.org was created to dogfood this code. it initially used just the compose/smtpclient/imapclient mox packages to send messages and process delivery feedback. it will get a config option to use the mox webapi/webhooks instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and developing that shaped development of the mox webapi/webhooks. for issue #31 by cuu508
297 lines
9.6 KiB
Bash
Executable file
297 lines
9.6 KiB
Bash
Executable file
#!/bin/bash
|
|
set -euo pipefail
|
|
|
|
# this is run with .. as working directory.
|
|
|
|
# note: outgoing hook events are in ../queue/hooks.go, ../mox-/config.go, ../queue.go and ../webapi/gendoc.sh. keep in sync.
|
|
|
|
# todo: find some proper way to generate the curl commands and responses automatically...
|
|
|
|
cat <<EOF
|
|
// NOTE: DO NOT EDIT, this file is generated by gendoc.sh.
|
|
|
|
/*
|
|
Package webapi implements a simple HTTP/JSON-based API for interacting with
|
|
email, and webhooks for notifications about incoming and outgoing deliveries,
|
|
including delivery failures.
|
|
|
|
# Overview
|
|
|
|
The webapi can be used to compose and send outgoing messages. The HTTP/JSON
|
|
API is often easier to use for developers since it doesn't require separate
|
|
libraries and/or having (detailed) knowledge about the format of email messages
|
|
("Internet Message Format"), or the SMTP protocol and its extensions.
|
|
|
|
Webhooks can be configured per account, and help with automated processing of
|
|
incoming email, and with handling delivery failures/success. Webhooks are
|
|
often easier to use for developers than monitoring a mailbox with IMAP and
|
|
processing new incoming email and delivery status notification (DSN) messages.
|
|
|
|
# Webapi
|
|
|
|
The webapi has a base URL at /webapi/v0/ by default, but configurable, which
|
|
serves an introduction that points to this documentation and lists the API
|
|
methods available.
|
|
|
|
An HTTP POST to /webapi/v0/<method> calls a method.The form can be either
|
|
"application/x-www-form-urlencoded" or "multipart/form-data". Form field
|
|
"request" must contain the request parameters, encoded as JSON.
|
|
|
|
HTTP basic authentication is required for calling methods, with an email
|
|
address as user name. Use a login address configured for "unique SMTP MAIL
|
|
FROM" addresses, and configure a period to "keep retired messages delivered
|
|
from the queue" for automatic suppression list management.
|
|
|
|
HTTP response status 200 OK indicates a successful method call, status 400
|
|
indicates an error. The response body of an error is a JSON object with a
|
|
human-readable "Message" field, and a "Code" field for programmatic handling
|
|
(common codes: "user" or user-induced errors, "server" for server-caused
|
|
errors). Most successful calls return a JSON object, but some return data
|
|
(e.g. a raw message or an attachment of a message). See [Methods] for the
|
|
methods and and [Client] for their documentation. The first element of their
|
|
return values indicate their JSON object type or io.ReadCloser for non-JSON
|
|
data. The request and response types are converted from/to JSON. optional and
|
|
missing/empty fields/values are converted into Go zero values: zero for
|
|
numbers, empty strings, empty lists and empty objects. New fields may be added
|
|
in response objects in future versions, parsers should ignore unrecognized
|
|
fields.
|
|
|
|
An HTTP GET to a method URL serves an HTML page showing example
|
|
request/response JSON objects in a form and a button to call the method.
|
|
|
|
# Webhooks
|
|
|
|
Webhooks for outgoing delivery events and incoming deliveries are configured
|
|
per account.
|
|
|
|
A webhook is delivered by an HTTP POST with headers "X-Mox-Webhook-ID" (unique
|
|
ID of webhook) and "X-Mox-Webhook-Attempt" (number of delivery attempts,
|
|
starting at 1), and a JSON body with the webhook data. Webhook delivery
|
|
failures are retried at a schedule similar to message deliveries, until
|
|
permanent failure.
|
|
|
|
See [webhook.Outgoing] for the fields in a webhook for outgoing deliveries, and
|
|
in particular [webhook.OutgoingEvent] for the types of events.
|
|
|
|
Only the latest event for the delivery of a particular outgoing message will be
|
|
delivered, any webhooks for that message still in the queue (after failure to
|
|
deliver) are retired as superseded when a new event occurs.
|
|
|
|
Webhooks for incoming deliveries are configured separately from outgoing
|
|
deliveries. Incoming DSNs for previously sent messages do not cause a webhook
|
|
to the webhook URL for incoming messages, only to the webhook URL for outgoing
|
|
delivery events. The incoming webhook JSON payload contains the message
|
|
envelope (parsed To, Cc, Bcc, Subject and more headers), the MIME structure,
|
|
and the contents of the first text and HTML parts. See [webhook.Incoming] for
|
|
the fields in the JSON object. The full message and individual parts, including
|
|
attachments, can be retrieved using the webapi.
|
|
|
|
# Transactional email
|
|
|
|
When sending transactional emails, potentially to many recipients, it is
|
|
important to process delivery failure notifications. If messages are rejected,
|
|
or email addresses no longer exist, you should stop sending email to those
|
|
addresses. If you try to keep sending, the receiving mail servers may consider
|
|
that spammy behaviour and blocklist your mail server.
|
|
|
|
Automatic suppression list management already prevents most repeated sending
|
|
attempts. The webhooks make it easy to receive failure notifications.
|
|
|
|
To keep spam complaints about your messages a minimum, include links to
|
|
unsubscribe from future messages without requiring further actions from the
|
|
user, such as logins. Include an unsubscribe link in the footer, and include
|
|
List-* message headers, such as List-Id, List-Unsubscribe and
|
|
List-Unsubscribe-Post.
|
|
|
|
# Webapi examples
|
|
|
|
Below are examples for making webapi calls to a locally running "mox
|
|
localserve" with its default credentials.
|
|
|
|
Send a basic message:
|
|
|
|
\$ curl --user mox@localhost:moxmoxmox \\
|
|
--data request='{"To": [{"Address": "mox@localhost"}], "Text": "hi ☺"}' \\
|
|
http://localhost:1080/webapi/v0/Send
|
|
{
|
|
"MessageID": "<kVTha0Q-a5Zh1MuTh5rUjg@localhost>",
|
|
"Submissions": [
|
|
{
|
|
"Address": "mox@localhost",
|
|
"QueueMsgID": 10010,
|
|
"FromID": "ZfV16EATHwKEufrSMo055Q"
|
|
}
|
|
]
|
|
}
|
|
|
|
|
|
Send a message with files both from form upload and base64 included in JSON:
|
|
|
|
\$ curl --user mox@localhost:moxmoxmox \\
|
|
--form request='{"To": [{"Address": "mox@localhost"}], "Subject": "hello", "Text": "hi ☺", "HTML": "<img src=\"cid:hi\" />", "AttachedFiles": [{"Name": "img.png", "ContentType": "image/png", "Data": "bWFkZSB5b3UgbG9vayE="}]}' \\
|
|
--form 'inlinefile=@hi.png;headers="Content-ID: <hi>"' \\
|
|
--form attachedfile=@mox.png \\
|
|
http://localhost:1080/webapi/v0/Send
|
|
{
|
|
"MessageID": "<eZ3OEEA2odXovovIxHE49g@localhost>",
|
|
"Submissions": [
|
|
{
|
|
"Address": "mox@localhost",
|
|
"QueueMsgID": 10011,
|
|
"FromID": "yWiUQ6mvJND8FRPSmc9y5A"
|
|
}
|
|
]
|
|
}
|
|
|
|
Get a message in parsed form:
|
|
|
|
\$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 424}' http://localhost:1080/webapi/v0/MessageGet
|
|
{
|
|
"Message": {
|
|
"From": [
|
|
{
|
|
"Name": "mox",
|
|
"Address": "mox@localhost"
|
|
}
|
|
],
|
|
"To": [
|
|
{
|
|
"Name": "",
|
|
"Address": "mox@localhost"
|
|
}
|
|
],
|
|
"CC": [],
|
|
"BCC": [],
|
|
"ReplyTo": [],
|
|
"MessageID": "<84vCeme_yZXyDzjWDeYBpg@localhost>",
|
|
"References": [],
|
|
"Date": "2024-04-04T14:29:42+02:00",
|
|
"Subject": "hello",
|
|
"Text": "hi \u263a\n",
|
|
"HTML": ""
|
|
},
|
|
"Structure": {
|
|
"ContentType": "multipart/mixed",
|
|
"ContentTypeParams": {
|
|
"boundary": "0ee72dc30dbab2ca6f7a363844a10a9f6111fc6dd31b8ff0b261478c2c48"
|
|
},
|
|
"ContentID": "",
|
|
"DecodedSize": 0,
|
|
"Parts": [
|
|
{
|
|
"ContentType": "multipart/related",
|
|
"ContentTypeParams": {
|
|
"boundary": "b5ed0977ee2b628040f394c3f374012458379a4f3fcda5036371d761c81d"
|
|
},
|
|
"ContentID": "",
|
|
"DecodedSize": 0,
|
|
"Parts": [
|
|
{
|
|
"ContentType": "multipart/alternative",
|
|
"ContentTypeParams": {
|
|
"boundary": "3759771adede7bd191ef37f2aa0e49ff67369f4000c320f198a875e96487"
|
|
},
|
|
"ContentID": "",
|
|
"DecodedSize": 0,
|
|
"Parts": [
|
|
{
|
|
"ContentType": "text/plain",
|
|
"ContentTypeParams": {
|
|
"charset": "utf-8"
|
|
},
|
|
"ContentID": "",
|
|
"DecodedSize": 8,
|
|
"Parts": []
|
|
},
|
|
{
|
|
"ContentType": "text/html",
|
|
"ContentTypeParams": {
|
|
"charset": "us-ascii"
|
|
},
|
|
"ContentID": "",
|
|
"DecodedSize": 22,
|
|
"Parts": []
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"ContentType": "image/png",
|
|
"ContentTypeParams": {},
|
|
"ContentID": "<hi>",
|
|
"DecodedSize": 19375,
|
|
"Parts": []
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"ContentType": "image/png",
|
|
"ContentTypeParams": {},
|
|
"ContentID": "",
|
|
"DecodedSize": 14,
|
|
"Parts": []
|
|
},
|
|
{
|
|
"ContentType": "image/png",
|
|
"ContentTypeParams": {},
|
|
"ContentID": "",
|
|
"DecodedSize": 7766,
|
|
"Parts": []
|
|
}
|
|
]
|
|
},
|
|
"Meta": {
|
|
"Size": 38946,
|
|
"DSN": false,
|
|
"Flags": [
|
|
"\$notjunk",
|
|
"\\seen"
|
|
],
|
|
"MailFrom": "",
|
|
"MailFromValidated": false,
|
|
"MsgFrom": "",
|
|
"MsgFromValidated": false,
|
|
"DKIMVerifiedDomains": [],
|
|
"RemoteIP": "",
|
|
"MailboxName": "Inbox"
|
|
}
|
|
}
|
|
|
|
Errors (with a 400 bad request HTTP status response) include a human-readable
|
|
message and a code for programmatic use:
|
|
|
|
\$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 999}' http://localhost:1080/webapi/v0/MessageGet
|
|
{
|
|
"Code": "notFound",
|
|
"Message": "message not found"
|
|
}
|
|
|
|
Get a raw, unparsed message, as bytes:
|
|
|
|
\$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 123}' http://localhost:1080/webapi/v0/MessageRawGet
|
|
[message as bytes in raw form]
|
|
|
|
Mark a message as read:
|
|
|
|
\$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 424, "Flags": ["\\\\Seen", "custom"]}' http://localhost:1080/webapi/v0/MessageFlagsAdd
|
|
{}
|
|
|
|
# Webhook examples
|
|
|
|
A webhook is delivered by an HTTP POST, wich headers X-Mox-Webhook-ID and
|
|
X-Mox-Webhook-Attempt and a JSON body with the data. To simulate a webhook call
|
|
for incoming messages, use:
|
|
|
|
curl -H 'X-Mox-Webhook-ID: 123' -H 'X-Mox-Webhook-Attempt: 1' --json '{...}' http://localhost/yourapp
|
|
|
|
EOF
|
|
|
|
for ex in $(./mox example | grep webhook); do
|
|
./mox example $ex
|
|
echo
|
|
done
|
|
|
|
cat <<EOF
|
|
*/
|
|
package webapi
|
|
// NOTE: DO NOT EDIT, this file is generated by gendoc.sh.
|
|
EOF
|