Commit graph

786 commits

Author SHA1 Message Date
Mechiel Lukkien
c4324fdaa1
fix bug in fixmsgize that makes it stop after the first batch of 10k messages
for issue #71 reported by naturalethic, thanks!

users upgrading from v0.0.6 to v0.0.7 could run into this. the release notes
have been updated with a link to the issue. the issue will stay open until at
least the next release.
2023-10-05 22:59:53 +02:00
tkivisik
3aa5026e11
fix typo in README.md (#72) 2023-10-04 07:39:44 +02:00
Mechiel Lukkien
91140da3a7
when logging about threading operations, include info about account
in verifydata, when warning about missing threading, print the db file.
otherwise it isn't clear which account this is about

when upgrading account thread storage, pass the logger that has the account
name.
2023-09-24 13:36:55 +02:00
Mechiel Lukkien
f2de89e365
shuffle sections in readme 2023-09-24 12:42:19 +02:00
Mechiel Lukkien
024c13c551
tweak readme 2023-09-24 12:34:46 +02:00
Mechiel Lukkien
55febe304e
imapserver: always send special-use attributes for mailboxes
even if not asked for with the "return (special-use)" extended list parameter.
macos x mail does not request the special-use flags, but will use them when present.

for issue #66, thanks x8x for providing the imap protocol transcript that
showed how it is done!
2023-09-23 21:00:26 +02:00
Mechiel Lukkien
f19f16bd8b
webmail: when scrolling down, don't send another parsed message that will cause one of the new messages to be selected (unexpected jump in the ui) 2023-09-23 18:36:24 +02:00
Mechiel Lukkien
d19c75559b
include all email addresses of an account in the mobileconfig profile for apple devices
after feedback from x8x, pointing out the support, thanks!

for issue #65
2023-09-23 17:50:32 +02:00
Mechiel Lukkien
f1f3135135
change "mox setaccountpassword" to use an account name as parameter, not email address
because with the name you would expect an account name.
and the email-resolving behaviour is surprising: with wildcard addresses you
can use any address, including a typo. you would change the password of the
address with the wildcard, without any warning. accounts are more precise and
less error-prone.

for issue #68 by x8x
2023-09-23 17:18:49 +02:00
Mechiel Lukkien
8c2814df89
imapserver: fix returning special-use mailbox "\Drafts" instead of "\Draft"
related to issue #66 by x8x, though this doesn't fix that (macos mail doesn't
yet request the special-use flags).
2023-09-23 14:50:02 +02:00
Mechiel Lukkien
0707f53361
in "mox uidbumpvalidity", bump to the next uidvalidity, otherwise we likely leave the uidvalidities in inconsistent state
the inconsistent state isn't really harmful, but we don't want inconsistencies.

pointed out in issue #61 by x8x
2023-09-23 12:15:13 +02:00
Mechiel Lukkien
85cef2a06c
when warning about not being able to hardlink during a backup, make it clear we continue with regular copying and that there won't be another warning
for issue #61 by x8x
2023-09-23 12:09:20 +02:00
Mechiel Lukkien
2b97c21f99
make setting up apple mail clients easier by providing .mobileconfig device management profiles
including showing a qr code to easily get the file on iphones.
the profile is currently in the "account" page.

idea by x8x in issue #65
2023-09-23 12:08:35 +02:00
Mechiel Lukkien
a0f3856e40
when moving a message out of a Rejects mailbox, mark the message as "not seen" so stands out in the destination mailbox (e.g. inbox)
we set the flag both for move in imap and in webmail.

this also ensures the "MailboxDestinedID", used for per-mailbox reputation
analysis, is set in more reject-situations. before this change, some rejects
(such as based on DMARC reject) wouldn't result in reputation being used after
having been moved the message out of the rejects mailbox.

in the future, we need more tests for scenario's like this...

for issue #63 reported by x8x
may also help with issue #64
2023-09-22 15:53:05 +02:00
Mechiel Lukkien
2ec8c79e10
update roadmap with http auth other than http basic, and add per-domain configs
these work on a list, not as high up as before, but moved up with requests in
issue #58, #67, #62.
2023-09-22 14:30:34 +02:00
Mechiel Lukkien
3353062dbe
webmail: when moving out all messages in a thread (none remaining in view), don't cause js error but select next message
removing an item from the selected list should be done regardless of focus,
i.e. the code snippet shouldn't have been behind the "if (focus...)" condition.
2023-09-22 14:25:25 +02:00
Mechiel Lukkien
be5f804d5b
webmail: use the "threads: on" mode by default
with "threads: unread", there is a bit too much change between different times
of opening the mailbox. perhaps the mode wasn't a good idea...
2023-09-22 14:12:46 +02:00
Mechiel Lukkien
89c543f662
if there is special-use junk flag on mailbox, don't also look at AutomaticJunkFlags option
the special-use flag should take precedence.
2023-09-22 10:51:42 +02:00
Mechiel Lukkien
6315d57166
ignore new pprof files from test-upgrade.sh after previous commit 2023-09-21 18:42:01 +02:00
Mechiel Lukkien
4de0af4fa5
add another automated upgrade test
for the path from v0.0.5 with lots of messages straight to the latest
development version. this can do multiple database changes in one go, so it's a
bit different than for installs where an admin has upgraded each version when
it was released.
2023-09-21 16:09:40 +02:00
Mechiel Lukkien
d618cbf918
mention funding through nlnet/eu ngi0 entrust
small line in the readme, but this means a lot for the project: continued
development for a year. expect lots of improvements and features.
2023-09-21 16:08:43 +02:00
Mechiel Lukkien
e6d8049548
webmail: in attachment viewer, for text/* content-type, show the text immediately too
instead of claiming it may be a binary file and showing a button to display the contents.
2023-09-21 15:29:38 +02:00
Mechiel Lukkien
2e16d8025d
when moving message to mailbox with special-use flag "Junk", mark the message as junk too, for retraining
i had been using the AutomaticJunkFlags option, so hadn't noticed the special use flag wasn't used.
2023-09-21 15:20:24 +02:00
Mechiel Lukkien
79774c15ec
add todo's about mime header parameter decoding
not sure what the correct approach is, would need to analyze email archive for practices.
2023-09-21 15:18:25 +02:00
Mechiel Lukkien
f87f286b80
webmail: dragging works on selected items, so tell user they cannot drag if they try to drag a non-selected message 2023-09-21 14:39:40 +02:00
Mechiel Lukkien
20f11409b6
webmail: when open the first unread message of a thread by default when opening a mailbox with threading enabled and the most recent message is in a thread 2023-09-21 12:56:51 +02:00
Mechiel Lukkien
fc6e61e9a5
webmail: add arrow left/right to collapse/expanse threads 2023-09-21 11:51:38 +02:00
Mechiel Lukkien
9bc860e207
webmail: make double click on mailbox expand/collapse, and make mailbox text unselectable (so the double click doesn't also select text) 2023-09-21 11:40:22 +02:00
Mechiel Lukkien
941a2311f0
webmail: try a bit harder not to get mailbox names or search queries in the potential stacktrace
we want to user to submit the stack trace. user can still edit before
submitting, but it won't look attractive to submit stacktraces with info that
shouldn't be there. not great that firefox is including too much info and the
effort we need to make to get it out again, but well.
2023-09-21 11:31:07 +02:00
Mechiel Lukkien
d07c871f5c
webmail: better recognize URLs in text wrapped in () or <> if it follows interpunction
e.g. "text... (https://localhost)." would keep ) as part of the url before, but not anymore.
2023-09-21 11:09:27 +02:00
Mechiel Lukkien
d649cf7dc2
quickstart: recognize likely NAT setup and set up host IPs in "NATIPs" field in the public listener
for issue #59 by pmarini, thanks!
2023-09-21 10:55:15 +02:00
Mechiel Lukkien
cde54442d2
webmail: in status line about (re|dis)connecting, make error message more readable
with space after line, so a next line doesn't get concatenated. and with capital.
2023-09-21 09:07:49 +02:00
Mechiel Lukkien
9534e464f9
add comment about the sconf config file format at the top of the config files
hopefully this helps admins editing the file and prevent mistakes about config files.

for issue #56 by kikoreis, thanks!
2023-09-21 08:59:10 +02:00
Mechiel Lukkien
0d8603f9e1
update latest deps 2023-09-20 16:52:18 +02:00
Mechiel Lukkien
ca5ef645f3
rename Account.Deliver to Account.DeliverDestination
the name was too generic compared with the other Deliver functions
2023-09-15 17:51:28 +02:00
Mechiel Lukkien
3620d6f05e
initialize metric mox_panic_total with 0, so the alerting rule also catches the first panic for a label
increase() and rate() don't seem to assume a previous value of 0 when a vector
gets a first value for a label. you would think that an increase() on a
first-value mox_panic_total{"..."}=1 would return 1, and similar for rate(), but
that doesn't appear to be the behaviour. so we just explicitly initialize the
count to 0 for each possible label value. mox has more vector metrics, but
panics feels like the most important, and it's too much code to initialize them
all, for all combinations of label values. there is probably a better way that
fixes this for all cases...
2023-09-15 16:47:17 +02:00
Mechiel Lukkien
af71e9855b
add package-level comments for webadmin and webaccount 2023-09-15 16:01:23 +02:00
Mechiel Lukkien
bff0131164
webmail: new shortcut "T" for showing html version of email, and t for text version
shortcut X used to be "show html version", but with threading support became
"toggle collapse", so there was a clash.
2023-09-15 15:51:59 +02:00
Mechiel Lukkien
3fb41ff073
implement message threading in backend and webmail
we match messages to their parents based on the "references" and "in-reply-to"
headers (requiring the same base subject), and in absense of those headers we
also by only base subject (against messages received max 4 weeks ago).

we store a threadid with messages. all messages in a thread have the same
threadid.  messages also have a "thread parent ids", which holds all id's of
parent messages up to the thread root.  then there is "thread missing link",
which is set when a referenced immediate parent wasn't found (but possibly
earlier ancestors can still be found and will be in thread parent ids".

threads can be muted: newly delivered messages are automatically marked as
read/seen.  threads can be marked as collapsed: if set, the webmail collapses
the thread to a single item in the basic threading view (default is to expand
threads).  the muted and collapsed fields are copied from their parent on
message delivery.

the threading is implemented in the webmail. the non-threading mode still works
as before. the new default threading mode "unread" automatically expands only
the threads with at least one unread (not seen) meessage. the basic threading
mode "on" expands all threads except when explicitly collapsed (as saved in the
thread collapsed field). new shortcuts for navigation/interaction threads have
been added, e.g. go to previous/next thread root, toggle collapse/expand of
thread (or double click), toggle mute of thread. some previous shortcuts have
changed, see the help for details.

the message threading are added with an explicit account upgrade step,
automatically started when an account is opened. the upgrade is done in the
background because it will take too long for large mailboxes to block account
operations. the upgrade takes two steps: 1. updating all message records in the
database to add a normalized message-id and thread base subject (with "re:",
"fwd:" and several other schemes stripped). 2. going through all messages in
the database again, reading the "references" and "in-reply-to" headers from
disk, and matching against their parents. this second step is also done at the
end of each import of mbox/maildir mailboxes. new deliveries are matched
immediately against other existing messages, currently no attempt is made to
rematch previously delivered messages (which could be useful for related
messages being delivered out of order).

the threading is not yet exposed over imap.
2023-09-13 15:44:57 +02:00
Mechiel Lukkien
b754b5f9ac
fix flushing of transparently compressed gzip output
this is a problem for connections like SSE, that only send data on events.
those events would stay in the gzip buffer until lots more data was written.

bug because of automatically typing "if err != nil"...

found while testing the maildir/mbox web-based import while working on message
threading support. the import gets progress SSE events that were now hanging.
2023-09-12 21:22:08 +02:00
Mechiel Lukkien
6f1e38f2ce
add flag to mox to store execution trace, similar to cpu/memory profiling
useful for performance testing
2023-09-12 14:43:52 +02:00
Mechiel Lukkien
4a4ccb83a3
when making a message preview, also recognize []-enclosed "horizontal ellipsis" unicode character as a snip 2023-09-11 14:41:50 +02:00
Mechiel Lukkien
fc7b0cc71e
fix parsing List-Post header in webmail 2023-09-11 11:55:28 +02:00
Mechiel Lukkien
f6d03a0eab
track more unexpected panics in metrics 2023-09-11 11:43:49 +02:00
Mechiel Lukkien
a5006a9090
fix not adding duplicate domains to the list of "verified dkim domains" for incoming messages 2023-09-11 11:37:45 +02:00
Mechiel Lukkien
cb1b133e28
add more rfc's, for jmap, caldav, carddav, lemonade profile
being on the list does not mean it is implemented.
2023-09-11 11:26:40 +02:00
Mechiel Lukkien
a6ae87d7ac
webmail: fix showing attachments that are text/plain and have content-disposition: attachment
they were not added to the list of attachments when sending the message to the
webmail frontend. they were shown on the "open message in new tab" page.
2023-09-03 15:20:56 +02:00
Mechiel Lukkien
4283ceecfc
fix serving static webmail files in development mode
due to a missing return, the content was served again.
this path doesn't happen on release binaries, only during local development,
where there is a local file that can be served.
2023-09-03 15:17:09 +02:00
Mechiel Lukkien
165639cb38
smtpserver: in helo/ehlo for submission don't fail on bad domain/ip address
for submission, we don't care about the value. users typically won't be able to
easily fix the errors (of their mail client software). so we just ignore the
domain/ip address, unless in pedantic mode.

this also parses "additional information after literal addresses" more
strictly/correctly.

for issue #55 by gimpf, thanks for the report!
2023-08-25 15:29:02 +02:00
Mechiel Lukkien
f4c20673ff
don't generate duplicate spf record if hostname is equal to domain name, e.g. postmaster@mail.domain.
the assumption has been that the hostname is something like mail.<domain>, when
setting up mox with the quickstart for user@<domain>. but users can use the
quickstart for postmaster@mail.<domain> as well.

for issue #46 by x8x, thanks for reporting!
2023-08-25 14:32:28 +02:00