Commit graph

692 commits

Author SHA1 Message Date
Mechiel Lukkien
fece75cfe7
automatically install typescript into ./node_modules if missing during build
simplifies process.
2024-04-24 19:48:01 +02:00
Mechiel Lukkien
d9f5625a89
regenerate apidiff, removal due to sherpadoc cleanup 2024-04-24 19:37:47 +02:00
Mechiel Lukkien
960a51242d
add aliases/lists: when sending to an alias, the message gets delivered to all members
the members must currently all be addresses of local accounts.

a message sent to an alias is accepted if at least one of the members accepts
it. if no members accepts it (e.g. due to bad reputation of sender), the
message is rejected.

if a message is submitted to both an alias addresses and to recipients that are
members of the alias in an smtp transaction, the message will be delivered to
such members only once.  the same applies if the address in the message
from-header is the address of a member: that member won't receive the message
(they sent it). this prevents duplicate messages.

aliases have three configuration options:
- PostPublic: whether anyone can send through the alias, or only members.
  members-only lists can be useful inside organizations for internal
  communication. public lists can be useful for support addresses.
- ListMembers: whether members can see the addresses of other members. this can
  be seen in the account web interface. in the future, we could export this in
  other ways, so clients can expand the list.
- AllowMsgFrom: whether messages can be sent through the alias with the alias
  address used in the message from-header. the webmail knows it can use that
  address, and will use it as from-address when replying to a message sent to
  that address.

ideas for the future:
- allow external addresses as members. still with some restrictions, such as
  requiring a valid dkim-signature so delivery has a chance to succeed. will
  also need configuration of an admin that can receive any bounces.
- allow specifying specific members who can sent through the list (instead of
  all members).

for github issue #57 by hmfaysal.
also relevant for #99 by naturalethic.
thanks to damir & marin from sartura for discussing requirements/features.
2024-04-24 19:15:30 +02:00
Mechiel Lukkien
1cf7477642
localserve: change queue to deliver to localserve smtp server
instead of skipping on any smtp and delivering messages to accounts.
we dial the ip of the smtp listener, which is localhost:1025 by default.

the smtp server now uses a mock dns resolver during spf & dkim verification for
hosted domains (localhost by default), so they should pass.

the advantage is that we get regular full smtp server behaviour for delivering
in localserve, including webhooks, and potential first-time sender delays
(though this is disabled by default now).

incoming deliveries now go through normal address resolution, where before we
would always deliver to mox@localhost. we still accept email for unknown
recipients to mox@localhost.

this will be useful upcoming alias/list functionality.

localserve will now generate a dkim key when creating a new config. existing
users may wish to reset (remove) their localserve directory, or add a dkim key.
2024-04-24 11:40:22 +02:00
Mechiel Lukkien
2bb4f78657
remove spurious empty line to fix build, and update roadmap 2024-04-22 14:32:50 +02:00
Mechiel Lukkien
bf5cfca6b9
webmail: add export functionality
per mailbox, or for all mailboxes, in maildir/mbox format, in tar/tgz/zip
archive or without archive format for single mbox, single or recursive. the
webaccount already had an option to export all mailboxes, it now looks similar
to the webmail version.
2024-04-22 13:41:40 +02:00
Mechiel Lukkien
a3f5fd26a6
webmail: less boilerplate code for api functions
open the account at the beginning of the api handler, and close accounts there too
2024-04-21 21:32:24 +02:00
Mechiel Lukkien
ed0c520562
webmail: single db transaction while fetching parsed message 2024-04-21 20:45:06 +02:00
Mechiel Lukkien
8ad32f9ede
improve docs about IPs and ipv4/ipv6 used for outgoing connections
based on feedback from alex on irc, thanks!
2024-04-21 17:22:00 +02:00
Mechiel Lukkien
884f5b5b3f
remove some old todo's from webmail 2024-04-21 17:18:00 +02:00
Mechiel Lukkien
6c0439cf7b
webmail: when moving a single message out of/to the inbox, ask if user wants to create a rule to automatically do that server-side for future deliveries
if the message has a list-id header, we assume this is a (mailing) list
message, and we require a dkim/spf-verified domain (we prefer the shortest that
is a suffix of the list-id value). the rule we would add will mark such
messages as from a mailing list, changing filtering rules on incoming messages
(not enforcing dmarc policies). messages will be matched on list-id header and
will only match if they have the same dkim/spf-verified domain.

if the message doesn't have a list-id header, we'll ask to match based on
"message from" address.

we don't ask the user in several cases:
- if the destination/source mailbox is a special-use mailbox (e.g.
  trash,archive,sent,junk; inbox isn't included)
- if the rule already exist (no point in adding it again).
- if the user said "no, not for this list-id/from-address" in the past.
- if the user said "no, not for messages moved to this mailbox" in the past.

we'll add the rule if the message was moved out of the inbox.
if the message was moved to the inbox, we check if there is a matching rule
that we can remove.

we now remember the "no" answers (for list-id, msg-from-addr and mailbox) in
the account database.

to implement the msgfrom rules, this adds support to rulesets for matching on
message "from" address. before, we could match on smtp from address (and other
fields). rulesets now also have a field for comments. webmail adds a note that
it created the rule, with the date.

manual editing of the rulesets is still in the webaccount page. this webmail
functionality is just a convenient way to add/remove common rules.
2024-04-21 17:14:08 +02:00
Mechiel Lukkien
71c0bd2dd1
for localserve delivery from queue, use the recipient address for finding delivery rules, not sender address 2024-04-21 15:07:50 +02:00
Mechiel Lukkien
0047f09e2b
webmail: new shadowed variables were detected by shadow since previous commit, prevent 2024-04-20 21:33:14 +02:00
Mechiel Lukkien
0f735a1710
webmail: remember per from-address whether we should show the text/html/html-with-external-resources version of a message 2024-04-20 21:25:52 +02:00
Mechiel Lukkien
3a58b2a1f4
webmail: show all images (inline and attachment) below the text part (for the text view, not for html view)
the attachment buttons for images get some opacity for the text view, to
indicate you don't have to open them explicitly.
2024-04-20 21:17:05 +02:00
Mechiel Lukkien
41a62de4d7
webmail: with 6 or more attachments, show the first 4, and a button to show the rest.
for issue #113
2024-04-20 17:53:32 +02:00
Mechiel Lukkien
9529ae0bd4
webmail: store composed message as draft until send, ask about unsaved changes when closing compose window 2024-04-20 17:38:25 +02:00
Mechiel Lukkien
e8bbaa451b
webmail: allow resizing of compose window
in top-left direction. keep textarea filling the height.
remember size in localstorage, only apply either width and/or height when
viewport width/height was the same as when the remembered width/height was set
(independently).

no visual indicator other than a cursor indicating resizability.
2024-04-20 10:26:54 +02:00
Mechiel Lukkien
5229d01601
webmail: for replies/forwards, add button "send and archive thread" next to the "send" button, and give it a control+shift+Enter shortcut
the regular send shortcut is control+Enter. the shift enables "archive thread".
there is no configuration option, you'll always get the button, but only for
reply/forward, not for new compose.

we may do "send and move thread to thrash", but let's wait until people want it.

for github issue #135 by mattfbacon
2024-04-19 21:17:42 +02:00
Mechiel Lukkien
b54e903f01
webmail: ctrl Backspace now removes an address input field if it is empty
instead of "ctrl -". i found ctrl backspace more intuitive.
2024-04-19 18:03:56 +02:00
Mechiel Lukkien
8a1d81c29a
webmail: show link to webaccount interface in top right
only if account web interface is enabled on the same listener and same http/https scheme.
2024-04-19 18:02:30 +02:00
Mechiel Lukkien
70adf353ee
webmail: add server-side stored settings, for signature, top/bottom reply and showing the security indications below address input fields
should solve #102
2024-04-19 18:02:24 +02:00
Mechiel Lukkien
3bbd7c7d9b
website: mention "mox localserve" as a good way to get a feeling for mox 2024-04-19 11:12:17 +02:00
Mechiel Lukkien
ec967ef321
use new sherpadoc rename mechanism to remove some typename stuttering
the stuttering was introduced to make the same type name declared in multiple
packages, and used in the admin sherpa api, unique. with sherpadoc's new
rename, we can make them unique when generating the api definition/docs, and
the Go code can use nicer names.
2024-04-19 10:51:24 +02:00
Mechiel Lukkien
962575f21b
mention webhook retry intervals in webhook docs
for github issue #31, feedback from cuu508
2024-04-19 10:33:28 +02:00
Mechiel Lukkien
e702f45d32
webadmin: make remaining domain settings configurable via admin web interface
for dmarc reporting address, tls reporting address, mtasts policy, dkim keys/selectors.

should make it easier for webadmin-using admins to discover these settings.

the webadmin interface is now on par with functionality you would set through
the configuration file, let's keep it that way.
2024-04-19 10:23:53 +02:00
Mechiel Lukkien
a69887bfab
webadmin: make routes configurable: globally, per domain, per account
this simplifies some of the code that makes modifications to the config file. a
few protected functions can make changes to the dynamic config, which webadmin
can use. instead of having separate functions in mox-/admin.go for each type of
change.

this also exports the parsed full dynamic config to webadmin, so we need fewer
functions for specific config fields too.
2024-04-18 11:14:24 +02:00
Mechiel Lukkien
baf4df55a6
make more account config fields configurable through web interface
so users can change it themselves, instead of requiring an admin to change the
settings.
2024-04-17 21:31:26 +02:00
Mechiel Lukkien
8bcce40c55
webmail: recognize multiple urls in List-Post addresses
there may be a http(s)-address, which we'll ignore. the mailto may come after
that. like in google groups.
2024-04-16 20:26:37 +02:00
Mechiel Lukkien
8654a1f901
with localserve, in queue, when "delivering" to the sender account, mark domain "localhost" as dkimverified
may be useful for testing, e.g. for rulesets to deliver messages to mailboxes other than Inbox.
2024-04-16 19:26:26 +02:00
Mechiel Lukkien
0a10283de0
show separate localpart and dropdown of domains instead of full email field when adding a new account (with initial email address) 2024-04-16 19:23:00 +02:00
Mechiel Lukkien
c9451d4d06
in webmail & webapisrv, store bcc header in sent messages
when sending a message with bcc's, prepend the bcc header to the message we
store in the sent folder. still not in the message we send to the recipients.
2024-04-16 17:57:46 +02:00
Mechiel Lukkien
abd098e8c0
in more tests, after closing accounts, check the last reference is indeed gone 2024-04-16 17:33:54 +02:00
Mechiel Lukkien
afc47c8108
if webauth login cookie is missing, and forwarding was configured, hint that reverse proxy may be stripping path
the cookies are set with a specific path, because the webadmin, webaccount and
webmail cookies can be on the same domain (this is the default). if the reverse
proxy strips the path while forwarding, the browser won't set the cookie and
the login attempt will fail.

based on github issue #151 from naturalethic
2024-04-16 16:06:31 +02:00
Mechiel Lukkien
daa88480cb
fix potential endless loop during queue msg/hook pagination when environment has TZ UTC, triggered by tests introduced in previous test
time.Now() returns a timestamp with timezone Local. if you marshal & unmarshal
it again, it'll get the Local timezone again. unless the local timezone is UTC.
then it will get the UTC timezone. the same time.Time but with explicit UTC
timezone vs explicit UTC-as-Local timezone are not the same when comparing with
==. so comparison should be done with time.Time.Equal, or comparison should be
done after having called .Local() on parsed timestamps (so the explicit UTC
timezone gets converted to the UTC-as-Local timezone). somewhat surprising that
time.Local isn't the same as time.UTC if TZ=/TZ=UTC. there are warnings
throughout the time package about handling of UTC.
2024-04-16 14:18:11 +02:00
Mechiel Lukkien
09fcc49223
add a webapi and webhooks for a simple http/json-based api
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.

this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.

unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...

matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.

a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.

messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.

suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.

submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".

to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.

admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.

new config options have been introduced. they are editable through the admin
and/or account web interfaces.

the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.

gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.

for issue #31 by cuu508
2024-04-15 21:49:02 +02:00
Mechiel Lukkien
8bec5ef7d4
also trigger use of smtputf8 for utf8 localpart in Reply-To header 2024-04-15 20:47:53 +02:00
Mechiel Lukkien
d014303617
use wlock when delivering message about new mox version 2024-04-15 20:40:16 +02:00
Mechiel Lukkien
b7ed035730
add godoc to metrics/ 2024-04-15 20:33:44 +02:00
Mechiel Lukkien
e1dbc07dba
fix harmless race where the same value is written to a tls config concurrently 2024-04-15 20:07:39 +02:00
Mechiel Lukkien
11eaa8cd1a
make imapserver faster like before again
in the precis password change before the previous release, the password used in
fuzzing wasn't correct, triggering sleeps due to botched protocols often, which
made the tests run much longer.
2024-04-14 17:41:36 +02:00
Mechiel Lukkien
12e6975aa7
return smtp response/error correctly in more cases 2024-04-14 17:28:00 +02:00
Mechiel Lukkien
4012b72d96
use type config.Account in sherpa api for better typing, and update to latest sherpa lib
typescript now knows the full types, not just "any" for account config.
inline structs previously in config.Account are given their own type definition
so sherpa can generate types.

also update to latest sherpa lib that knows about time.Duration, to be used soon.
2024-04-14 17:18:20 +02:00
Mechiel Lukkien
b7d6540d51
style nit: only take address of structs when passed on 2024-04-14 12:46:24 +02:00
Mechiel Lukkien
2a949f9f79
fix typo in smtp error code 2024-04-14 12:42:47 +02:00
Mechiel Lukkien
e585a4d180
don't fail to generate apidiff when packages are introduced 2024-04-14 12:38:58 +02:00
Mechiel Lukkien
4b459af4a8
add install as target, calling "go install"
convenient for local testing, i'm often running "mox localserve", often helpful
if it's the latest.
2024-04-14 12:37:52 +02:00
Mechiel Lukkien
1ea851bb53
Merge commit 'feb8e6c37947b21baaa7dcf724ade0f2435a8280'
github PR #152, also for issue #149
2024-04-13 13:36:11 +02:00
Mechiel Lukkien
34572d14d0
regenerate apidiff/next.txt after change to smtpclient
by calling "make genapidiff"
2024-04-13 13:31:32 +02:00
Mechiel Lukkien
73381d26ed
Merge commit 'be570d1c7d3de0ddacb011b6411a302d7f7e9f9e'
from github PR #153
2024-04-13 13:31:02 +02:00