Commit graph

149 commits

Author SHA1 Message Date
Mechiel Lukkien
92e0d2a682
webadmin: be more helpful when adding domains/accounts/addresses
by explaining (in the titles/hovers) what the concepts and requirements are, by
using selects/dropdowns or datalist suggestions where we have a known list, by
automatically suggesting a good account name, and putting the input fields in a
more sensible order.

based on issue #132 by ally9335
2024-03-09 11:11:52 +01:00
Mechiel Lukkien
63cef8e3a5
webmail: fix for ignoring error about sending to invalid address
before, an error about an invalid address was not used, causing a delivery
attempt to an empty address (empty localpart/domain). delivery to that address
would fail, but we should've prevented that message from being queued at all.

additionally, an error in adding the message to the queue was ignored too.
2024-03-09 09:51:24 +01:00
Mechiel Lukkien
c57aeac7f0
prevent unicode-confusion in password by applying PRECIS, and username/email address by applying unicode NFC normalization
an é (e with accent) can also be written as e+\u0301. the first form is NFC,
the second NFD. when logging in, we transform usernames (email addresses) to
NFC. so both forms will be accepted. if a client is using NFD, they can log
in too.

for passwords, we apply the PRECIS "opaquestring", which (despite the name)
transforms the value too: unicode spaces are replaced with ascii spaces. the
string is also normalized to NFC. PRECIS may reject confusing passwords when
you set a password.
2024-03-09 09:20:29 +01:00
Mechiel Lukkien
8e6fe7459b
normalize localparts with unicode nfc when parsing
both when parsing our configs, and for incoming on smtp or in messages.
so we properly compare things like é and e+accent as equal, and accept the
different encodings of that same address.
2024-03-08 21:08:40 +01:00
Mechiel Lukkien
9e7d6b85b7
queue: deliver to multiple recipients in a single smtp transaction
transferring the data only once. we only do this when the recipient domains
are the same. when queuing, we now take care to set the same NextAttempt
timestamp, so queued messages are actually eligable for combined delivery.

this adds a DeliverMultiple to the smtp client. for pipelined requests, it will
send all RCPT TO (and MAIL and DATA) in one go, and handles the various
responses and error conditions, returning either an overal error, or per
recipient smtp responses. the results of the smtp LIMITS extension are also
available in the smtp client now.

this also takes the "LIMITS RCPTMAX" smtp extension into account: if the server
only accepts a single recipient, we won't send multiple.
if a server doesn't announce a RCPTMAX limit, but still has one (like mox does
for non-spf-verified transactions), we'll recognize code 452 and 552 (for
historic reasons) as temporary error, and try again in a separate transaction
immediately after. we don't yet implement "LIMITS MAILMAX", doesn't seem likely
in practice.
2024-03-07 10:07:53 +01:00
Mechiel Lukkien
47ebfa8152
queue: implement adding a message to the queue that gets sent to multiple recipients
and in a way that allows us to send that message to multiple recipients in a
single smtp transaction.
2024-03-05 20:10:28 +01:00
Mechiel Lukkien
af968f7614
webmail: for junk/rejects messages, show sender address instead of name in list 2024-03-05 09:04:59 +01:00
Mechiel Lukkien
79f91ebd87
webmail: don't switch back focus after autocompleting address
actually, this fix can reduce focus changes for more operations. withStatus is
often used to show an operation in progress in the status bar, only when the
operation isn't done within 1 second. we would restore focus to the element
before the operation started. that was done because we disable elements
sometimes (preventing duplicate form submission). for things like the
autocomplete, with the tab key, which also moves focus to the next element, we
don't want that focus switched back again.
2024-03-05 08:46:56 +01:00
Mechiel Lukkien
63c3c1fd6a
webmail: leave out own address in reply all when we have addresses remaining 2024-03-04 20:21:41 +01:00
Mechiel Lukkien
13923e4b7b
better thread matching for dsns
keep track of whether a message is a dsn, and match dsn's against their sent
message by ignoring the message subject.
2024-03-04 16:40:27 +01:00
Mechiel Lukkien
93c52b01a0
implement "future release"
the smtp extension, rfc 4865.
also implement in the webmail.
the queueing/delivery part hardly required changes: we just set the first
delivery time in the future instead of immediately.

still have to find the first client that implements it.
2024-02-10 17:55:56 +01:00
Mechiel Lukkien
ee1db2dde7
webmail: implement registering and handling "mailto:" links
to start composing a message.

the help popup now has a button to register the "mailto:" links with the mox
webmail (typically only works over https, not all browsers support it).

the mailto links are specified in 6068. we support the to/cc/bcc/subject/body
parameters. other parameters should be seen as custom headers, but we don't
support messages with custom headers at all at the moment, so we ignore them.

we now also turn text of the form "mailto:user@host" into a clickable link
(will not be too common). we could be recognizing any "x@x.x" as email address
and make them clickable in the future.

thanks to Hans-Jörg for explaining this functionality.
2024-02-09 11:21:33 +01:00
Mechiel Lukkien
f3bf348214
webmail: show unicode for internationalized email addresses by default
before, we showed the xn-- ascii names, along with the unicode name. but users
of internationalized email don't want to see any xn-- names. we now put those
in an html title attribute for some cases, so you can still see them if you
really want to, by hovering.

after talking to arnt at fosdem.
2024-02-08 18:03:48 +01:00
Mechiel Lukkien
d1b87cdb0d
replace packages slog and slices from golang.org/x/exp with stdlib
since we are now at go1.21 as minimum.
2024-02-08 14:49:01 +01:00
Mechiel Lukkien
9cf8ee2162
webmail: don't who an age of "-<1min", drop the -
if a browser is ahead just a few seconds, we would show "-<1min", not great.
just show "<1min" in that case. we'll still show negative age if drift is more
than 1 minute, which seems like a good hint to get time fixed on either client
or server.
2024-01-23 17:01:34 +01:00
Mechiel Lukkien
46aacdb79b
webmail: when q/b-word-decoding attachment filenames, recognize more charset encodings
based on #113 by jsfan3
2024-01-12 15:25:23 +01:00
Mechiel Lukkien
2392f79aa9
for username/email input field in login form, automatically resize so also longer addresses are fully visible
feedback from jsfan3 in issue #58, thanks!
2024-01-08 22:00:42 +01:00
Mechiel Lukkien
c348834ce9
prevent firefox from autocompleting the current password in the form/fields for changing password 2024-01-05 12:15:55 +01:00
Mechiel Lukkien
0f8bf2f220
replace http basic auth for web interfaces with session cookie & csrf-based auth
the http basic auth we had was very simple to reason about, and to implement.
but it has a major downside:

there is no way to logout, browsers keep sending credentials. ideally, browsers
themselves would show a button to stop sending credentials.

a related downside: the http auth mechanism doesn't indicate for which server
paths the credentials are.

another downside: the original password is sent to the server with each
request. though sending original passwords to web servers seems to be
considered normal.

our new approach uses session cookies, along with csrf values when we can. the
sessions are server-side managed, automatically extended on each use. this
makes it easy to invalidate sessions and keeps the frontend simpler (than with
long- vs short-term sessions and refreshing). the cookies are httponly,
samesite=strict, scoped to the path of the web interface. cookies are set
"secure" when set over https. the cookie is set by a successful call to Login.
a call to Logout invalidates a session. changing a password invalidates all
sessions for a user, but keeps the session with which the password was changed
alive. the csrf value is also random, and associated with the session cookie.
the csrf must be sent as header for api calls, or as parameter for direct form
posts (where we cannot set a custom header). rest-like calls made directly by
the browser, e.g. for images, don't have a csrf protection. the csrf value is
returned by the Login api call and stored in localstorage.

api calls without credentials return code "user:noAuth", and with bad
credentials return "user:badAuth". the api client recognizes this and triggers
a login. after a login, all auth-failed api calls are automatically retried.
only for "user:badAuth" is an error message displayed in the login form (e.g.
session expired).

in an ideal world, browsers would take care of most session management. a
server would indicate authentication is needed (like http basic auth), and the
browsers uses trusted ui to request credentials for the server & path. the
browser could use safer mechanism than sending original passwords to the
server, such as scram, along with a standard way to create sessions.  for now,
web developers have to do authentication themselves: from showing the login
prompt, ensuring the right session/csrf cookies/localstorage/headers/etc are
sent with each request.

webauthn is a newer way to do authentication, perhaps we'll implement it in the
future. though hardware tokens aren't an attractive option for many users, and
it may be overkill as long as we still do old-fashioned authentication in smtp
& imap where passwords can be sent to the server.

for issue #58
2024-01-05 10:48:42 +01:00
Mechiel Lukkien
1f9b640d9a
add faq for smtp smuggling, fix bug around handling "\nX\n" for any X, reject bare carriage returns and possibly smtp-smuggling attempts
mox was already strict in its "\r\n.\r\n" handling for end-of-message in an
smtp transaction.

due to a mostly unrelated bug, sequences of "\nX\n", including "\n.\n" were
rejected with a "local processing error".

the sequence "\r\n.\n" dropped the dot, not necessarily a big problem, this is
unlikely to happen in a legimate transaction and the behaviour not
unreasonable.

we take this opportunity to reject all bare \r.  we detect all slightly
incorrect combinations of "\r\n.\r\n" with an error mentioning smtp smuggling,
in part to appease the tools checking for it.

smtp errors are 500 "bad syntax", and mention smtp smuggling.
2024-01-01 20:11:16 +01:00
Mechiel Lukkien
fce3a5bf73
webmail: moxVersion was too similar to moxversion, choose better name 2024-01-01 14:51:17 +01:00
Mechiel Lukkien
b887539ee4
webmail/*.ts needed rebuild after changing tcs.sh to target es2022
hopefully last of embarrassing string of commits...
2024-01-01 14:13:05 +01:00
Mechiel Lukkien
a9940f9855
change javascript into typescript for webaccount and webadmin interface
all ui frontend code is now in typescript. we no longer need jshint, and we
build the frontend code during "make build".

this also changes tlsrpt types for a Report, not encoding field names with
dashes, but to keep them valid identifiers in javascript. this makes it more
conveniently to work with in the frontend, and works around a sherpats
limitation.
2023-12-31 12:05:31 +01:00
Mechiel Lukkien
802dcef192
webmail: for messages in designated Sent mailbox, show To/Cc/Bcc in italics, and show all correspondents in collapsed thread
showing addressees for Sent messages for issue #104 by mattfbacon, thanks for the report!
2023-12-21 09:23:06 +01:00
Mechiel Lukkien
d73bda7511
add per-account quota for total message size disk usage
so a single user cannot fill up the disk.
by default, there is (still) no limit. a default can be set in the config file
for all accounts, and a per-account max size can be set that would override any
global setting.

this does not take into account disk usage of the index database. and also not
of any file system overhead.
2023-12-20 20:54:12 +01:00
Mechiel Lukkien
e048d0962b
small fixes
a typo, using ongoing tx instead of making a new one, don't pass literal string
to formatting function.

found while working on quota support.
2023-12-16 11:53:14 +01:00
Mechiel Lukkien
dfddf0e874
for webapi requests, make canceled contexts a user instead of server error
no need to trigger alerts for user-initiated errors
2023-12-15 15:47:54 +01:00
Mechiel Lukkien
406fdc312d
when autocompleting, abort previous still pending request
should prevent a long list of "Autocompleting address" mentions in the status
bar at the top in case of non-responsive network
2023-12-14 20:20:17 +01:00
Mechiel Lukkien
d1b66035a9
add more documentation, examples with tests to illustrate reusable components 2023-12-14 20:20:17 +01:00
Mechiel Lukkien
72ac1fde29
expose fewer internals in packages, for easier software reuse
- prometheus is now behind an interface, they aren't dependencies for the
  reusable components anymore.
- some dependencies have been inverted: instead of packages importing a main
  package to get configuration, the main package now sets configuration in
  these packages. that means fewer internals are pulled in.
- some functions now have new parameters for values that were retrieved from
  package "mox-".
2023-12-14 15:39:36 +01:00
Mechiel Lukkien
5b20cba50a
switch to slog.Logger for logging, for easier reuse of packages by external software
we don't want external software to include internal details like mlog.
slog.Logger is/will be the standard.

we still have mlog for its helper functions, and its handler that logs in
concise logfmt used by mox.

packages that are not meant for reuse still pass around mlog.Log for
convenience.

we use golang.org/x/exp/slog because we also support the previous Go toolchain
version. with the next Go release, we'll switch to the builtin slog.
2023-12-14 13:45:52 +01:00
Mechiel Lukkien
7c1879da82
webmail: when replying to message we sent, don't compose the reply to ourselve, but copy the original to/cc/bcc headers 2023-11-27 12:26:31 +01:00
Mechiel Lukkien
fb81effe45
webmail: for domain in From address, show if domain is dmarc(-like) validated
i'm not sure this is good enough.
this is based on field MsgFromValidation, but it doesn't hold the full DMARC information.
we also don't know mailing list-status for all historic messages.
so the red underline can occur too often.
2023-11-27 12:11:05 +01:00
Mechiel Lukkien
8e37fadc13
webmail: in initial start (sse) event, send the version, and ask user to reload if it changes
will prevent showing errors to users about new unknown fields that may be added
in the new version.
2023-11-27 08:06:27 +01:00
Mechiel Lukkien
416113af72
webmail: do not automatically mark read messages in Rejects mailbox as nonjunk 2023-11-27 07:34:18 +01:00
Mechiel Lukkien
3d80c05423
webmail: for long to/cc/bcc address list (>5) show the first 4 and a button to show the rest
for issue #98 by mattfbacon, thanks
2023-11-20 21:36:40 +01:00
Mechiel Lukkien
8f55d0ada6
fix build, missing api build 2023-11-11 20:06:42 +01:00
Mechiel Lukkien
dcee0345ef
nits, removing a old todo and a stray newline 2023-11-11 19:14:19 +01:00
Mechiel Lukkien
42f6f9cbb3
change the message composing code from webmail over to message.Composer too 2023-11-09 21:15:27 +01:00
Mechiel Lukkien
96faf4b5ec
webmail: don't select requiretls when mta-sts and dane are both not implemented (even though requiretls extension is announced) 2023-11-09 19:57:53 +01:00
Mechiel Lukkien
893a6f8911
implement outgoing tls reports
we were already accepting, processing and displaying incoming tls reports. now
we start tracking TLS connection and security-policy-related errors for
outgoing message deliveries as well. we send reports once a day, to the
reporting addresses specified in TLSRPT records (rua) of a policy domain. these
reports are about MTA-STS policies and/or DANE policies, and about
STARTTLS-related failures.

sending reports is enabled by default, but can be disabled through setting
NoOutgoingTLSReports in mox.conf.

only at the end of the implementation process came the realization that the
TLSRPT policy domain for DANE (MX) hosts are separate from the TLSRPT policy
for the recipient domain, and that MTA-STS and DANE TLS/policy results are
typically delivered in separate reports. so MX hosts need their own TLSRPT
policies.

config for the per-host TLSRPT policy should be added to mox.conf for existing
installs, in field HostTLSRPT. it is automatically configured by quickstart for
new installs. with a HostTLSRPT config, the "dns records" and "dns check" admin
pages now suggest the per-host TLSRPT record. by creating that record, you're
requesting TLS reports about your MX host.

gathering all the TLS/policy results is somewhat tricky. the tentacles go
throughout the code. the positive result is that the TLS/policy-related code
had to be cleaned up a bit. for example, the smtpclient TLS modes now reflect
reality better, with independent settings about whether PKIX and/or DANE
verification has to be done, and/or whether verification errors have to be
ignored (e.g. for tls-required: no header). also, cached mtasts policies of
mode "none" are now cleaned up once the MTA-STS DNS record goes away.
2023-11-09 19:47:26 +01:00
Mechiel Lukkien
d02ac0cb86
webmail: fix received date shown on message
we were trying to offset the timezone, but that makes no sense: we already
created a date in the local timezone based on (milli)seconds passed. so we can
just use that date instead of calculating a wrong date.
2023-11-04 23:35:44 +01:00
Mechiel Lukkien
4510e0ce78
webmail: add Delivered-To to example settingsPut call 2023-11-02 21:56:59 +01:00
Mechiel Lukkien
0200e539a9
when message is delivered, save whether it is from a mailing list; in webmail, show if message was a forward or mailing list, and don't enable requiretls when sending to a list. 2023-11-02 20:03:47 +01:00
Mechiel Lukkien
38694d3928
Merge remote-tracking branch 'github.com/mattfbacon/mox/message-is-text' 2023-11-02 14:41:43 +01:00
Mechiel Lukkien
9896639ff9
for incoming smtp deliveries, track whether tls and requiretls was used, and display this in the webmail
we store the tls version used, and cipher suite. we don't currently show that
in the webmail.
2023-11-02 09:12:47 +01:00
Mechiel Lukkien
f7686b7db8
webmail: show email address instead of display name of "from" header in message listing if display name contains chars from "<@>"
it could be an attempt to confuse the reader with an email address. a classic.
2023-11-02 09:12:47 +01:00
Mechiel Lukkien
725f030d3c
webmail: add clear marker between message header and body, so if html message tries to fake ui elements, it'll be noticed (hopefully) 2023-11-02 09:12:47 +01:00
Mechiel Lukkien
ef50f4abf0
refactor common pattern of close & remove temporary file into calling the new store.CloseRemoveTempFile 2023-11-02 09:12:46 +01:00
Mechiel Lukkien
e7699708ef
implement outgoing dmarc aggregate reporting
in smtpserver, we store dmarc evaluations (under the right conditions).
in dmarcdb, we periodically (hourly) send dmarc reports if there are
evaluations. for failed deliveries, we deliver the dsn quietly to a submailbox
of the postmaster mailbox.

this is on by default, but can be disabled in mox.conf.
2023-11-02 09:12:30 +01:00
Matt Fellenz
3b6e1851cb
Treat messages as text 2023-11-01 14:17:02 -07:00
Mechiel Lukkien
2f5d6069bf
implement "requiretls", rfc 8689
with requiretls, the tls verification mode/rules for email deliveries can be
changed by the sender/submitter. in two ways:

1. "requiretls" smtp extension to always enforce verified tls (with mta-sts or
dnssec+dane), along the entire delivery path until delivery into the final
destination mailbox (so entire transport is verified-tls-protected).

2. "tls-required: no" message header, to ignore any tls and tls verification
errors even if the recipient domain has a policy that requires tls verification
(mta-sts and/or dnssec+dane), allowing delivery of non-sensitive messages in
case of misconfiguration/interoperability issues (at least useful for sending
tls reports).

we enable requiretls by default (only when tls is active), for smtp and
submission. it can be disabled through the config.

for each delivery attempt, we now store (per recipient domain, in the account
of the sender) whether the smtp server supports starttls and requiretls. this
support is shown (after having sent a first message) in the webmail when
sending a message (the previous 3 bars under the address input field are now 5
bars, the first for starttls support, the last for requiretls support). when
all recipient domains for a message are known to implement requiretls,
requiretls is automatically selected for sending (instead of "default" tls
behaviour). users can also select the "fallback to insecure" to add the
"tls-required: no" header.

new metrics are added for insight into requiretls errors and (some, not yet
all) cases where tls-required-no ignored a tls/verification error.

the admin can change the requiretls status for messages in the queue. so with
default delivery attempts, when verified tls is required by failing, an admin
could potentially change the field to "tls-required: no"-behaviour.

messages received (over smtp) with the requiretls option, get a comment added
to their Received header line, just before "id", after "with".
2023-10-24 10:10:46 +02:00
Mechiel Lukkien
08995c7806
webmail: when composing a message, show security status in a bar below addressee input field
the bar is currently showing 3 properties:
1. mta-sts enforced;
2. mx lookup returned dnssec-signed response;
3. first delivery destination host has dane records

the colors are: red for not-implemented, green for implemented, gray for error,
nothing for unknown/irrelevant.

the plan is to implement "requiretls" soon and start caching per domain whether
delivery can be done with starttls and whether the domain supports requiretls.
and show that in two new parts of the bar.

thanks to damian poddebniak for pointing out that security indicators should
always be visible, not only for positive/negative result. otherwise users won't
notice their absence.
2023-10-15 15:40:13 +02:00
Mechiel Lukkien
4ab3e6bc9b
webmail: autoresize address input field in compose window
so full name/email address is visible.

using a hidden grid element that gets the same content as the input element.
from https://css-tricks.com/auto-growing-inputs-textareas/

a recent commit probably also make the compose window full-screen-width on
chrome, this restores to the intended behaviour of a less wide default size.

if you add multiple address fields, the compose window will still grow. not
great, in the future, we should make the compose window resizable by dragging.
2023-10-15 10:53:57 +02:00
Mechiel Lukkien
42d817ef3d
quick fix for making compose window resizable by expanding/shrinking when textarea is resized
the textarea is resizable (though it's not convenient to do in firefox which
only shows a dragcorner in the bottomright, usually located in the bottom
corner of the screen, so there is little space left to drag the corner; the
workaround is to move the window temporarily).
2023-10-14 21:02:54 +02:00
Mechiel Lukkien
56956c224b
webmail: when quoting text that switches unicode blocks (as highlighted), don't lose the switched text
by using a String object as the textarea child.  instead of a regular js string
that would be unicode-block-switch-highlighted, which would cause it to be
split into parts, with odd or even parts added as span elements, which the
textarea would then ignore.
2023-10-14 14:47:24 +02:00
Mechiel Lukkien
a40f5a5eb3
webmail: recognize q/b-word-encoded filenames in attachments in messages
according to the rfc's (2231, and 2047), non-ascii filenames in content-type
and content-disposition headers should be encoded like this:

	Content-Type: text/plain; name*=utf-8''hi%E2%98%BA.txt
	Content-Disposition: attachment; filename*=utf-8''hi%E2%98%BA.txt

and that is what the Go standard library mime.ParseMediaType and
mime.FormatMediaType parse and generate.

this is what thunderbird sends:

	Content-Type: text/plain; charset=UTF-8; name="=?UTF-8?B?aGnimLoudHh0?="
	Content-Disposition: attachment; filename*=UTF-8''%68%69%E2%98%BA%2E%74%78%74

(thunderbird will also correctly split long filenames over multiple parameters,
named "filename*0*", "filename*1*", etc.)

this is what gmail sends:

	Content-Type: text/plain; charset="US-ASCII"; name="=?UTF-8?B?aGnimLoudHh0?="
	Content-Disposition: attachment; filename="=?UTF-8?B?aGnimLoudHh0?="

i cannot find where the q/b-word encoded values in "name" and "filename" are
allowed. until that time, we try parsing them unless in pedantic mode.

we didn't generate correctly encoded filenames yet, this commit also fixes that.

for issue #82 by mattfbacon, thanks for reporting!
2023-10-14 14:14:13 +02:00
Mechiel Lukkien
6e391c3be0
ensure there is a space between active requests mentioned in the status bar at the top 2023-10-14 11:13:26 +02:00
Mechiel Lukkien
28fae96a9b
make mox compile on windows, without "mox serve" but with working "mox localserve"
getting mox to compile required changing code in only a few places where
package "syscall" was used: for accessing file access times and for umask
handling. an open problem is how to start a process as an unprivileged user on
windows.  that's why "mox serve" isn't implemented yet. and just finding a way
to implement it now may not be good enough in the near future: we may want to
starting using a more complete privilege separation approach, with a process
handling sensitive tasks (handling private keys, authentication), where we may
want to pass file descriptors between processes. how would that work on
windows?

anyway, getting mox to compile for windows doesn't mean it works properly on
windows. the largest issue: mox would normally open a file, rename or remove
it, and finally close it. this happens during message delivery. that doesn't
work on windows, the rename/remove would fail because the file is still open.
so this commit swaps many "remove" and "close" calls. renames are a longer
story: message delivery had two ways to deliver: with "consuming" the
(temporary) message file (which would rename it to its final destination), and
without consuming (by hardlinking the file, falling back to copying). the last
delivery to a recipient of a message (and the only one in the common case of a
single recipient) would consume the message, and the earlier recipients would
not.  during delivery, the already open message file was used, to parse the
message.  we still want to use that open message file, and the caller now stays
responsible for closing it, but we no longer try to rename (consume) the file.
we always hardlink (or copy) during delivery (this works on windows), and the
caller is responsible for closing and removing (in that order) the original
temporary file. this does cost one syscall more. but it makes the delivery code
(responsibilities) a bit simpler.

there is one more obvious issue: the file system path separator. mox already
used the "filepath" package to join paths in many places, but not everywhere.
and it still used strings with slashes for local file access. with this commit,
the code now uses filepath.FromSlash for path strings with slashes, uses
"filepath" in a few more places where it previously didn't. also switches from
"filepath" to regular "path" package when handling mailbox names in a few
places, because those always use forward slashes, regardless of local file
system conventions.  windows can handle forward slashes when opening files, so
test code that passes path strings with forward slashes straight to go stdlib
file i/o functions are left unchanged to reduce code churn. the regular
non-test code, or test code that uses path strings in places other than
standard i/o functions, does have the paths converted for consistent paths
(otherwise we would end up with paths with mixed forward/backward slashes in
log messages).

windows cannot dup a listening socket. for "mox localserve", it isn't
important, and we can work around the issue. the current approach for "mox
serve" (forking a process and passing file descriptors of listening sockets on
"privileged" ports) won't work on windows. perhaps it isn't needed on windows,
and any user can listen on "privileged" ports? that would be welcome.

on windows, os.Open cannot open a directory, so we cannot call Sync on it after
message delivery. a cursory internet search indicates that directories cannot
be synced on windows. the story is probably much more nuanced than that, with
long deep technical details/discussions/disagreement/confusion, like on unix.
for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 10:54:07 +02:00
Mechiel Lukkien
8640fd8cff
webmail: top-post with no text selected and add "on ... wrote"-line, keep bottom-quoting with text selected
top-posting causes "On $datetime, $sender wrote:" above the quoted text to be
added (unless there was no Date header or valid address in a From header).

in the near future we should create settings, and add a setting for adding the
"on ... wrote"-line, ideally including a template.

for issue #83 by mattfbacon, thanks!
2023-10-13 19:28:04 +02:00
Mechiel Lukkien
4e26fd13e2
when api docs cannot be loaded, say which 2023-10-13 08:52:06 +02:00
Mechiel Lukkien
a93dd348fe
webmail: ensure wrap of long header lines, instead of horizontal scrollbar in message header section 2023-10-12 22:08:13 +02:00
Mechiel Lukkien
8dacc31445
webmail: for high images (aspect ratio), don't let image extend beyond window height
apparently the flex parent and flex child with grow 1 is unbounded even with a parent height of 100%
2023-10-12 21:53:05 +02:00
Mechiel Lukkien
f19f16bd8b
webmail: when scrolling down, don't send another parsed message that will cause one of the new messages to be selected (unexpected jump in the ui) 2023-09-23 18:36:24 +02:00
Mechiel Lukkien
a0f3856e40
when moving a message out of a Rejects mailbox, mark the message as "not seen" so stands out in the destination mailbox (e.g. inbox)
we set the flag both for move in imap and in webmail.

this also ensures the "MailboxDestinedID", used for per-mailbox reputation
analysis, is set in more reject-situations. before this change, some rejects
(such as based on DMARC reject) wouldn't result in reputation being used after
having been moved the message out of the rejects mailbox.

in the future, we need more tests for scenario's like this...

for issue #63 reported by x8x
may also help with issue #64
2023-09-22 15:53:05 +02:00
Mechiel Lukkien
3353062dbe
webmail: when moving out all messages in a thread (none remaining in view), don't cause js error but select next message
removing an item from the selected list should be done regardless of focus,
i.e. the code snippet shouldn't have been behind the "if (focus...)" condition.
2023-09-22 14:25:25 +02:00
Mechiel Lukkien
be5f804d5b
webmail: use the "threads: on" mode by default
with "threads: unread", there is a bit too much change between different times
of opening the mailbox. perhaps the mode wasn't a good idea...
2023-09-22 14:12:46 +02:00
Mechiel Lukkien
e6d8049548
webmail: in attachment viewer, for text/* content-type, show the text immediately too
instead of claiming it may be a binary file and showing a button to display the contents.
2023-09-21 15:29:38 +02:00
Mechiel Lukkien
2e16d8025d
when moving message to mailbox with special-use flag "Junk", mark the message as junk too, for retraining
i had been using the AutomaticJunkFlags option, so hadn't noticed the special use flag wasn't used.
2023-09-21 15:20:24 +02:00
Mechiel Lukkien
79774c15ec
add todo's about mime header parameter decoding
not sure what the correct approach is, would need to analyze email archive for practices.
2023-09-21 15:18:25 +02:00
Mechiel Lukkien
f87f286b80
webmail: dragging works on selected items, so tell user they cannot drag if they try to drag a non-selected message 2023-09-21 14:39:40 +02:00
Mechiel Lukkien
20f11409b6
webmail: when open the first unread message of a thread by default when opening a mailbox with threading enabled and the most recent message is in a thread 2023-09-21 12:56:51 +02:00
Mechiel Lukkien
fc6e61e9a5
webmail: add arrow left/right to collapse/expanse threads 2023-09-21 11:51:38 +02:00
Mechiel Lukkien
9bc860e207
webmail: make double click on mailbox expand/collapse, and make mailbox text unselectable (so the double click doesn't also select text) 2023-09-21 11:40:22 +02:00
Mechiel Lukkien
941a2311f0
webmail: try a bit harder not to get mailbox names or search queries in the potential stacktrace
we want to user to submit the stack trace. user can still edit before
submitting, but it won't look attractive to submit stacktraces with info that
shouldn't be there. not great that firefox is including too much info and the
effort we need to make to get it out again, but well.
2023-09-21 11:31:07 +02:00
Mechiel Lukkien
d07c871f5c
webmail: better recognize URLs in text wrapped in () or <> if it follows interpunction
e.g. "text... (https://localhost)." would keep ) as part of the url before, but not anymore.
2023-09-21 11:09:27 +02:00
Mechiel Lukkien
cde54442d2
webmail: in status line about (re|dis)connecting, make error message more readable
with space after line, so a next line doesn't get concatenated. and with capital.
2023-09-21 09:07:49 +02:00
Mechiel Lukkien
3620d6f05e
initialize metric mox_panic_total with 0, so the alerting rule also catches the first panic for a label
increase() and rate() don't seem to assume a previous value of 0 when a vector
gets a first value for a label. you would think that an increase() on a
first-value mox_panic_total{"..."}=1 would return 1, and similar for rate(), but
that doesn't appear to be the behaviour. so we just explicitly initialize the
count to 0 for each possible label value. mox has more vector metrics, but
panics feels like the most important, and it's too much code to initialize them
all, for all combinations of label values. there is probably a better way that
fixes this for all cases...
2023-09-15 16:47:17 +02:00
Mechiel Lukkien
bff0131164
webmail: new shortcut "T" for showing html version of email, and t for text version
shortcut X used to be "show html version", but with threading support became
"toggle collapse", so there was a clash.
2023-09-15 15:51:59 +02:00
Mechiel Lukkien
3fb41ff073
implement message threading in backend and webmail
we match messages to their parents based on the "references" and "in-reply-to"
headers (requiring the same base subject), and in absense of those headers we
also by only base subject (against messages received max 4 weeks ago).

we store a threadid with messages. all messages in a thread have the same
threadid.  messages also have a "thread parent ids", which holds all id's of
parent messages up to the thread root.  then there is "thread missing link",
which is set when a referenced immediate parent wasn't found (but possibly
earlier ancestors can still be found and will be in thread parent ids".

threads can be muted: newly delivered messages are automatically marked as
read/seen.  threads can be marked as collapsed: if set, the webmail collapses
the thread to a single item in the basic threading view (default is to expand
threads).  the muted and collapsed fields are copied from their parent on
message delivery.

the threading is implemented in the webmail. the non-threading mode still works
as before. the new default threading mode "unread" automatically expands only
the threads with at least one unread (not seen) meessage. the basic threading
mode "on" expands all threads except when explicitly collapsed (as saved in the
thread collapsed field). new shortcuts for navigation/interaction threads have
been added, e.g. go to previous/next thread root, toggle collapse/expand of
thread (or double click), toggle mute of thread. some previous shortcuts have
changed, see the help for details.

the message threading are added with an explicit account upgrade step,
automatically started when an account is opened. the upgrade is done in the
background because it will take too long for large mailboxes to block account
operations. the upgrade takes two steps: 1. updating all message records in the
database to add a normalized message-id and thread base subject (with "re:",
"fwd:" and several other schemes stripped). 2. going through all messages in
the database again, reading the "references" and "in-reply-to" headers from
disk, and matching against their parents. this second step is also done at the
end of each import of mbox/maildir mailboxes. new deliveries are matched
immediately against other existing messages, currently no attempt is made to
rematch previously delivered messages (which could be useful for related
messages being delivered out of order).

the threading is not yet exposed over imap.
2023-09-13 15:44:57 +02:00
Mechiel Lukkien
4a4ccb83a3
when making a message preview, also recognize []-enclosed "horizontal ellipsis" unicode character as a snip 2023-09-11 14:41:50 +02:00
Mechiel Lukkien
fc7b0cc71e
fix parsing List-Post header in webmail 2023-09-11 11:55:28 +02:00
Mechiel Lukkien
a6ae87d7ac
webmail: fix showing attachments that are text/plain and have content-disposition: attachment
they were not added to the list of attachments when sending the message to the
webmail frontend. they were shown on the "open message in new tab" page.
2023-09-03 15:20:56 +02:00
Mechiel Lukkien
4283ceecfc
fix serving static webmail files in development mode
due to a missing return, the content was served again.
this path doesn't happen on release binaries, only during local development,
where there is a local file that can be served.
2023-09-03 15:17:09 +02:00
Mechiel Lukkien
affb057a0c
webmail: fix case where tree of mailboxes wasn't displayed properly
for example, when these mailboxes existed: "a", "a.b", "a/b", then "a.b" (.
before / in ascii) prevented "a/b" from being displayed in the tree below "a".
2023-08-23 14:57:05 +02:00
Mechiel Lukkien
9e248860ee
implement transparent gzip compression in the webserver
we only compress if applicable (content-type indicates likely compressible),
client supports it, response doesn't already have a content-encoding).

for internal handlers, we always enable compression.  for reverse proxied and
static files, compression must be enabled per handler.

for internal & reverse proxy handlers, we do streaming compression at
"bestspeed" quality (probably level 1).

for static files, we have a cache based on mtime with fixed max size, where we
evict based on least recently used. we compress with the default level (more
cpu, better ratio).
2023-08-21 21:52:35 +02:00
Mechiel Lukkien
80547df6ee
webmail: don't have two spaces between header and address(es) (e.g. for From/To)
because outlook.com will reformat the message and then fail to verify the message.
proton.me also reformats and invalidates the dkim signature, but seemingly
after it verifies the dkim signature.
2023-08-16 15:22:38 +02:00
Mechiel Lukkien
9f46879377
webmail: correct label for Subject in search form 2023-08-15 13:03:02 +02:00
Mechiel Lukkien
d7df70acd8
webmail: don't lose display of additional headers when a flag/keyword changes (e.g. marked as read) 2023-08-11 08:38:57 +02:00
Mechiel Lukkien
383eb483df
webmail: for html-only messages, also show the "show html with external resources" button 2023-08-10 14:55:30 +02:00
Mechiel Lukkien
0434e49c3a
webmail: while attachment viewer is open, don't handle global keyboard shortcuts (like search, going to inbox)
feedback from jonathan, thanks!
2023-08-10 11:02:13 +02:00
Mechiel Lukkien
c24bb063e5
webmail tweaks
- padding on small attachment download button.
- don't remember "show html" but always display text first.
- propagate modseq to message when flags/keywords change, so "show internals" shows the update.
2023-08-10 10:56:04 +02:00
Mechiel Lukkien
f48a53726e
when clearing search, open inbox
feedback from jonathan, thanks!
2023-08-10 10:42:54 +02:00
Mechiel Lukkien
01bcd98a42
add flag to ruleset that indicates a message is forwarded, slightly modifying how junk analysis is done
part of PR #50 by bobobo1618
2023-08-09 22:31:37 +02:00
Mechiel Lukkien
383fe4f53a
explicitly store in a Message whether it was delivered to the rejects mailbox
soon, we can have multiple rejects mailboxes.  and checking against the
configured rejects mailbox name wasn't foolproof to begin with, because it may
have changed between delivery to the rejects mailbox and the message being
moved.

after upgrading, messages currently in rejects mailboxes don't have IsReject
set, so they don't get the special rejecs treatment when being moved. they are
removed from the rejects mailbox after some time though, and newly added
rejects will be treated correctly. so this means some existing messages wrongly
delivered to the rejects mailbox, and moved out, aren't used (for a positive
signal) for future deliveries.  saves a bit of complexity in the
implementation.  i think the tradeoff is worth it.

related to discussion in issue #50
2023-08-09 16:52:24 +02:00
Mechiel Lukkien
20ebdae8ea
in webmail, automatically mark message as nonjunk when open for 5 seconds, and prevent extraneous newlines when composing a reply to selected text 2023-08-09 09:45:54 +02:00
Mechiel Lukkien
34ede1075d
remove last remnants of treating a mailbox named "Sent" specially, in favor of special-use mailbox flags
a few places still looked at the name "Sent". but since we have special-use
flags, we should always look at those. this also changes the config so admins
can specify different names for the special-use mailboxes to create for new
accounts, e.g. in a different language. the old config option is still
understood, just deprecated.
2023-08-09 09:31:23 +02:00
Mechiel Lukkien
49cf16d3f2
fix race in test setup/teardown
not easily triggered, but it happened just now on a build server.
2023-08-07 23:14:31 +02:00
Mechiel Lukkien
849b4ec9e9
add webmail
it was far down on the roadmap, but implemented earlier, because it's
interesting, and to help prepare for a jmap implementation. for jmap we need to
implement more client-like functionality than with just imap. internal data
structures need to change. jmap has lots of other requirements, so it's already
a big project. by implementing a webmail now, some of the required data
structure changes become clear and can be made now, so the later jmap
implementation can do things similarly to the webmail code. the webmail
frontend and webmail are written together, making their interface/api much
smaller and simpler than jmap.

one of the internal changes is that we now keep track of per-mailbox
total/unread/unseen/deleted message counts and mailbox sizes.  keeping this
data consistent after any change to the stored messages (through the code base)
is tricky, so mox now has a consistency check that verifies the counts are
correct, which runs only during tests, each time an internal account reference
is closed. we have a few more internal "changes" that are propagated for the
webmail frontend (that imap doesn't have a way to propagate on a connection),
like changes to the special-use flags on mailboxes, and used keywords in a
mailbox. more changes that will be required have revealed themselves while
implementing the webmail, and will be implemented next.

the webmail user interface is modeled after the mail clients i use or have
used: thunderbird, macos mail, mutt; and webmails i normally only use for
testing: gmail, proton, yahoo, outlook. a somewhat technical user is assumed,
but still the goal is to make this webmail client easy to use for everyone. the
user interface looks like most other mail clients: a list of mailboxes, a
search bar, a message list view, and message details. there is a top/bottom and
a left/right layout for the list/message view, default is automatic based on
screen size. the panes can be resized by the user. buttons for actions are just
text, not icons. clicking a button briefly shows the shortcut for the action in
the bottom right, helping with learning to operate quickly. any text that is
underdotted has a title attribute that causes more information to be displayed,
e.g. what a button does or a field is about. to highlight potential phishing
attempts, any text (anywhere in the webclient) that switches unicode "blocks"
(a rough approximation to (language) scripts) within a word is underlined
orange. multiple messages can be selected with familiar ui interaction:
clicking while holding control and/or shift keys.  keyboard navigation works
with arrows/page up/down and home/end keys, and also with a few basic vi-like
keys for list/message navigation. we prefer showing the text instead of
html (with inlined images only) version of a message. html messages are shown
in an iframe served from an endpoint with CSP headers to prevent dangerous
resources (scripts, external images) from being loaded. the html is also
sanitized, with javascript removed. a user can choose to load external
resources (e.g. images for tracking purposes).

the frontend is just (strict) typescript, no external frameworks. all
incoming/outgoing data is typechecked, both the api request parameters and
response types, and the data coming in over SSE. the types and checking code
are generated with sherpats, which uses the api definitions generated by
sherpadoc based on the Go code. so types from the backend are automatically
propagated to the frontend.  since there is no framework to automatically
propagate properties and rerender components, changes coming in over the SSE
connection are propagated explicitly with regular function calls.  the ui is
separated into "views", each with a "root" dom element that is added to the
visible document. these views have additional functions for getting changes
propagated, often resulting in the view updating its (internal) ui state (dom).
we keep the frontend compilation simple, it's just a few typescript files that
get compiled (combined and types stripped) into a single js file, no additional
runtime code needed or complicated build processes used.  the webmail is served
is served from a compressed, cachable html file that includes style and the
javascript, currently just over 225kb uncompressed, under 60kb compressed (not
minified, including comments). we include the generated js files in the
repository, to keep Go's easily buildable self-contained binaries.

authentication is basic http, as with the account and admin pages. most data
comes in over one long-term SSE connection to the backend. api requests signal
which mailbox/search/messages are requested over the SSE connection. fetching
individual messages, and making changes, are done through api calls. the
operations are similar to imap, so some code has been moved from package
imapserver to package store. the future jmap implementation will benefit from
these changes too. more functionality will probably be moved to the store
package in the future.

the quickstart enables webmail on the internal listener by default (for new
installs). users can enable it on the public listener if they want to. mox
localserve enables it too. to enable webmail on existing installs, add settings
like the following to the listeners in mox.conf, similar to AccountHTTP(S):

	WebmailHTTP:
		Enabled: true
	WebmailHTTPS:
		Enabled: true

special thanks to liesbeth, gerben, andrii for early user feedback.

there is plenty still to do, see the list at the top of webmail/webmail.ts.
feedback welcome as always.
2023-08-07 21:57:03 +02:00