Commit graph

572 commits

Author SHA1 Message Date
Mechiel Lukkien
2ec8c79e10
update roadmap with http auth other than http basic, and add per-domain configs
these work on a list, not as high up as before, but moved up with requests in
issue #58, #67, #62.
2023-09-22 14:30:34 +02:00
Mechiel Lukkien
3353062dbe
webmail: when moving out all messages in a thread (none remaining in view), don't cause js error but select next message
removing an item from the selected list should be done regardless of focus,
i.e. the code snippet shouldn't have been behind the "if (focus...)" condition.
2023-09-22 14:25:25 +02:00
Mechiel Lukkien
be5f804d5b
webmail: use the "threads: on" mode by default
with "threads: unread", there is a bit too much change between different times
of opening the mailbox. perhaps the mode wasn't a good idea...
2023-09-22 14:12:46 +02:00
Mechiel Lukkien
89c543f662
if there is special-use junk flag on mailbox, don't also look at AutomaticJunkFlags option
the special-use flag should take precedence.
2023-09-22 10:51:42 +02:00
Mechiel Lukkien
6315d57166
ignore new pprof files from test-upgrade.sh after previous commit 2023-09-21 18:42:01 +02:00
Mechiel Lukkien
4de0af4fa5
add another automated upgrade test
for the path from v0.0.5 with lots of messages straight to the latest
development version. this can do multiple database changes in one go, so it's a
bit different than for installs where an admin has upgraded each version when
it was released.
2023-09-21 16:09:40 +02:00
Mechiel Lukkien
d618cbf918
mention funding through nlnet/eu ngi0 entrust
small line in the readme, but this means a lot for the project: continued
development for a year. expect lots of improvements and features.
2023-09-21 16:08:43 +02:00
Mechiel Lukkien
e6d8049548
webmail: in attachment viewer, for text/* content-type, show the text immediately too
instead of claiming it may be a binary file and showing a button to display the contents.
2023-09-21 15:29:38 +02:00
Mechiel Lukkien
2e16d8025d
when moving message to mailbox with special-use flag "Junk", mark the message as junk too, for retraining
i had been using the AutomaticJunkFlags option, so hadn't noticed the special use flag wasn't used.
2023-09-21 15:20:24 +02:00
Mechiel Lukkien
79774c15ec
add todo's about mime header parameter decoding
not sure what the correct approach is, would need to analyze email archive for practices.
2023-09-21 15:18:25 +02:00
Mechiel Lukkien
f87f286b80
webmail: dragging works on selected items, so tell user they cannot drag if they try to drag a non-selected message 2023-09-21 14:39:40 +02:00
Mechiel Lukkien
20f11409b6
webmail: when open the first unread message of a thread by default when opening a mailbox with threading enabled and the most recent message is in a thread 2023-09-21 12:56:51 +02:00
Mechiel Lukkien
fc6e61e9a5
webmail: add arrow left/right to collapse/expanse threads 2023-09-21 11:51:38 +02:00
Mechiel Lukkien
9bc860e207
webmail: make double click on mailbox expand/collapse, and make mailbox text unselectable (so the double click doesn't also select text) 2023-09-21 11:40:22 +02:00
Mechiel Lukkien
941a2311f0
webmail: try a bit harder not to get mailbox names or search queries in the potential stacktrace
we want to user to submit the stack trace. user can still edit before
submitting, but it won't look attractive to submit stacktraces with info that
shouldn't be there. not great that firefox is including too much info and the
effort we need to make to get it out again, but well.
2023-09-21 11:31:07 +02:00
Mechiel Lukkien
d07c871f5c
webmail: better recognize URLs in text wrapped in () or <> if it follows interpunction
e.g. "text... (https://localhost)." would keep ) as part of the url before, but not anymore.
2023-09-21 11:09:27 +02:00
Mechiel Lukkien
d649cf7dc2
quickstart: recognize likely NAT setup and set up host IPs in "NATIPs" field in the public listener
for issue #59 by pmarini, thanks!
2023-09-21 10:55:15 +02:00
Mechiel Lukkien
cde54442d2
webmail: in status line about (re|dis)connecting, make error message more readable
with space after line, so a next line doesn't get concatenated. and with capital.
2023-09-21 09:07:49 +02:00
Mechiel Lukkien
9534e464f9
add comment about the sconf config file format at the top of the config files
hopefully this helps admins editing the file and prevent mistakes about config files.

for issue #56 by kikoreis, thanks!
2023-09-21 08:59:10 +02:00
Mechiel Lukkien
0d8603f9e1
update latest deps 2023-09-20 16:52:18 +02:00
Mechiel Lukkien
ca5ef645f3
rename Account.Deliver to Account.DeliverDestination
the name was too generic compared with the other Deliver functions
2023-09-15 17:51:28 +02:00
Mechiel Lukkien
3620d6f05e
initialize metric mox_panic_total with 0, so the alerting rule also catches the first panic for a label
increase() and rate() don't seem to assume a previous value of 0 when a vector
gets a first value for a label. you would think that an increase() on a
first-value mox_panic_total{"..."}=1 would return 1, and similar for rate(), but
that doesn't appear to be the behaviour. so we just explicitly initialize the
count to 0 for each possible label value. mox has more vector metrics, but
panics feels like the most important, and it's too much code to initialize them
all, for all combinations of label values. there is probably a better way that
fixes this for all cases...
2023-09-15 16:47:17 +02:00
Mechiel Lukkien
af71e9855b
add package-level comments for webadmin and webaccount 2023-09-15 16:01:23 +02:00
Mechiel Lukkien
bff0131164
webmail: new shortcut "T" for showing html version of email, and t for text version
shortcut X used to be "show html version", but with threading support became
"toggle collapse", so there was a clash.
2023-09-15 15:51:59 +02:00
Mechiel Lukkien
3fb41ff073
implement message threading in backend and webmail
we match messages to their parents based on the "references" and "in-reply-to"
headers (requiring the same base subject), and in absense of those headers we
also by only base subject (against messages received max 4 weeks ago).

we store a threadid with messages. all messages in a thread have the same
threadid.  messages also have a "thread parent ids", which holds all id's of
parent messages up to the thread root.  then there is "thread missing link",
which is set when a referenced immediate parent wasn't found (but possibly
earlier ancestors can still be found and will be in thread parent ids".

threads can be muted: newly delivered messages are automatically marked as
read/seen.  threads can be marked as collapsed: if set, the webmail collapses
the thread to a single item in the basic threading view (default is to expand
threads).  the muted and collapsed fields are copied from their parent on
message delivery.

the threading is implemented in the webmail. the non-threading mode still works
as before. the new default threading mode "unread" automatically expands only
the threads with at least one unread (not seen) meessage. the basic threading
mode "on" expands all threads except when explicitly collapsed (as saved in the
thread collapsed field). new shortcuts for navigation/interaction threads have
been added, e.g. go to previous/next thread root, toggle collapse/expand of
thread (or double click), toggle mute of thread. some previous shortcuts have
changed, see the help for details.

the message threading are added with an explicit account upgrade step,
automatically started when an account is opened. the upgrade is done in the
background because it will take too long for large mailboxes to block account
operations. the upgrade takes two steps: 1. updating all message records in the
database to add a normalized message-id and thread base subject (with "re:",
"fwd:" and several other schemes stripped). 2. going through all messages in
the database again, reading the "references" and "in-reply-to" headers from
disk, and matching against their parents. this second step is also done at the
end of each import of mbox/maildir mailboxes. new deliveries are matched
immediately against other existing messages, currently no attempt is made to
rematch previously delivered messages (which could be useful for related
messages being delivered out of order).

the threading is not yet exposed over imap.
2023-09-13 15:44:57 +02:00
Mechiel Lukkien
b754b5f9ac
fix flushing of transparently compressed gzip output
this is a problem for connections like SSE, that only send data on events.
those events would stay in the gzip buffer until lots more data was written.

bug because of automatically typing "if err != nil"...

found while testing the maildir/mbox web-based import while working on message
threading support. the import gets progress SSE events that were now hanging.
2023-09-12 21:22:08 +02:00
Mechiel Lukkien
6f1e38f2ce
add flag to mox to store execution trace, similar to cpu/memory profiling
useful for performance testing
2023-09-12 14:43:52 +02:00
Mechiel Lukkien
4a4ccb83a3
when making a message preview, also recognize []-enclosed "horizontal ellipsis" unicode character as a snip 2023-09-11 14:41:50 +02:00
Mechiel Lukkien
fc7b0cc71e
fix parsing List-Post header in webmail 2023-09-11 11:55:28 +02:00
Mechiel Lukkien
f6d03a0eab
track more unexpected panics in metrics 2023-09-11 11:43:49 +02:00
Mechiel Lukkien
a5006a9090
fix not adding duplicate domains to the list of "verified dkim domains" for incoming messages 2023-09-11 11:37:45 +02:00
Mechiel Lukkien
cb1b133e28
add more rfc's, for jmap, caldav, carddav, lemonade profile
being on the list does not mean it is implemented.
2023-09-11 11:26:40 +02:00
Mechiel Lukkien
a6ae87d7ac
webmail: fix showing attachments that are text/plain and have content-disposition: attachment
they were not added to the list of attachments when sending the message to the
webmail frontend. they were shown on the "open message in new tab" page.
2023-09-03 15:20:56 +02:00
Mechiel Lukkien
4283ceecfc
fix serving static webmail files in development mode
due to a missing return, the content was served again.
this path doesn't happen on release binaries, only during local development,
where there is a local file that can be served.
2023-09-03 15:17:09 +02:00
Mechiel Lukkien
165639cb38
smtpserver: in helo/ehlo for submission don't fail on bad domain/ip address
for submission, we don't care about the value. users typically won't be able to
easily fix the errors (of their mail client software). so we just ignore the
domain/ip address, unless in pedantic mode.

this also parses "additional information after literal addresses" more
strictly/correctly.

for issue #55 by gimpf, thanks for the report!
2023-08-25 15:29:02 +02:00
Mechiel Lukkien
f4c20673ff
don't generate duplicate spf record if hostname is equal to domain name, e.g. postmaster@mail.domain.
the assumption has been that the hostname is something like mail.<domain>, when
setting up mox with the quickstart for user@<domain>. but users can use the
quickstart for postmaster@mail.<domain> as well.

for issue #46 by x8x, thanks for reporting!
2023-08-25 14:32:28 +02:00
Mechiel Lukkien
61a5eb61a4
remove needless fmt.Sprintf
by staticcheck
2023-08-23 16:27:02 +02:00
Mechiel Lukkien
f029db3f47
imapserver bugfix: fix expunging for messages marked junk/nonjunk
such messages would be marked expunged in the database, then the junkfilter
would be retrained for the removal of the message. but during retraining, the
expunged flag would be cleared again. the on-disk message file would still be
removed. so when opening the mailbox, the message would appear to still exist,
but cannot be retrieved from disk.

if you run "mox fixmsgsize", and you get warnings about missing message files,
you could create empty files (with "touch"), run "mox fixsmsgsize" again,
followed by "mox recalculatemailboxcounts <affectedaccount>" and run "mox
bumpuidvalidity <affectaccount>".

"mox backup" would probably also complain, as would "mox verifydata".

this may have caused the "wrong mailbox counts" error i got from "mox
verifydata" on a backup.
2023-08-23 16:20:06 +02:00
Mechiel Lukkien
da9f1d9d0d
in admin pages, make the literal instruction text on the dnscheck page visible, and set a max-width for easier readability 2023-08-23 15:10:02 +02:00
Mechiel Lukkien
b3dd4a55c3
fix a spello, and reword so misspell doesn't complain about it 2023-08-23 14:59:43 +02:00
Mechiel Lukkien
affb057a0c
webmail: fix case where tree of mailboxes wasn't displayed properly
for example, when these mailboxes existed: "a", "a.b", "a/b", then "a.b" (.
before / in ascii) prevented "a/b" from being displayed in the tree below "a".
2023-08-23 14:57:05 +02:00
Mechiel Lukkien
aebfd78a9f
implement accepting dmarc & tls reports for other domains
to accept reports for another domain, first add that domain to the config,
leaving all options empty except DMARC/TLSRPT in which you configure a Domain.

the suggested DNS DMARC/TLSRPT records will show the email address with
configured domain. for DMARC, the dnscheck functionality will verify that the
destination domain has opted in to receiving reports.

there is a new command-line subcommand "mox dmarc checkreportaddrs" that
verifies if dmarc reporting destination addresses have opted in to received
reports.

this also changes the suggested dns records (in quickstart, and through admin
pages and cli subcommand) to take into account whether DMARC and TLSRPT is
configured, and with which localpart/domain (previously it always printed
records as if reporting was enabled for the domain). and when generating the
suggested DNS records, the dmarc.Record and tlsrpt.Record code is used, with
proper uri-escaping.
2023-08-23 14:27:21 +02:00
Mechiel Lukkien
9e248860ee
implement transparent gzip compression in the webserver
we only compress if applicable (content-type indicates likely compressible),
client supports it, response doesn't already have a content-encoding).

for internal handlers, we always enable compression.  for reverse proxied and
static files, compression must be enabled per handler.

for internal & reverse proxy handlers, we do streaming compression at
"bestspeed" quality (probably level 1).

for static files, we have a cache based on mtime with fixed max size, where we
evict based on least recently used. we compress with the default level (more
cpu, better ratio).
2023-08-21 21:52:35 +02:00
Mechiel Lukkien
4c72184b44
update link to docker image
user was being redirected to the new url
2023-08-20 18:45:19 +02:00
Mechiel Lukkien
b43529a2e9
sendmail: bugfix: set remote hostname to verify for tls connections
due to logic bug we weren't setting it, and tls connections would fail with a
warning that either the remote hostname must be set or insecurityskipverify
must be set.
2023-08-20 18:26:20 +02:00
Mechiel Lukkien
0b9475271c
add possible future todo for working around ios messages with wrong q-encoded headers 2023-08-16 16:22:00 +02:00
Mechiel Lukkien
80547df6ee
webmail: don't have two spaces between header and address(es) (e.g. for From/To)
because outlook.com will reformat the message and then fail to verify the message.
proton.me also reformats and invalidates the dkim signature, but seemingly
after it verifies the dkim signature.
2023-08-16 15:22:38 +02:00
Mechiel Lukkien
1ccc5d0177
fix message size in a message in gentestdata
and work around the message in test-upgrade.sh.
and add subcommand to open an account, triggering data upgrades.
2023-08-16 14:36:17 +02:00
Mechiel Lukkien
ddf3cb3653
mention there are now webmail screenshots, and small release process tweaks 2023-08-16 10:16:48 +02:00
Mechiel Lukkien
9f46879377
webmail: correct label for Subject in search form 2023-08-15 13:03:02 +02:00