Commit graph

296 commits

Author SHA1 Message Date
Mechiel Lukkien
6f1e38f2ce
add flag to mox to store execution trace, similar to cpu/memory profiling
useful for performance testing
2023-09-12 14:43:52 +02:00
Mechiel Lukkien
4a4ccb83a3
when making a message preview, also recognize []-enclosed "horizontal ellipsis" unicode character as a snip 2023-09-11 14:41:50 +02:00
Mechiel Lukkien
fc7b0cc71e
fix parsing List-Post header in webmail 2023-09-11 11:55:28 +02:00
Mechiel Lukkien
f6d03a0eab
track more unexpected panics in metrics 2023-09-11 11:43:49 +02:00
Mechiel Lukkien
a5006a9090
fix not adding duplicate domains to the list of "verified dkim domains" for incoming messages 2023-09-11 11:37:45 +02:00
Mechiel Lukkien
cb1b133e28
add more rfc's, for jmap, caldav, carddav, lemonade profile
being on the list does not mean it is implemented.
2023-09-11 11:26:40 +02:00
Mechiel Lukkien
a6ae87d7ac
webmail: fix showing attachments that are text/plain and have content-disposition: attachment
they were not added to the list of attachments when sending the message to the
webmail frontend. they were shown on the "open message in new tab" page.
2023-09-03 15:20:56 +02:00
Mechiel Lukkien
4283ceecfc
fix serving static webmail files in development mode
due to a missing return, the content was served again.
this path doesn't happen on release binaries, only during local development,
where there is a local file that can be served.
2023-09-03 15:17:09 +02:00
Mechiel Lukkien
165639cb38
smtpserver: in helo/ehlo for submission don't fail on bad domain/ip address
for submission, we don't care about the value. users typically won't be able to
easily fix the errors (of their mail client software). so we just ignore the
domain/ip address, unless in pedantic mode.

this also parses "additional information after literal addresses" more
strictly/correctly.

for issue #55 by gimpf, thanks for the report!
2023-08-25 15:29:02 +02:00
Mechiel Lukkien
f4c20673ff
don't generate duplicate spf record if hostname is equal to domain name, e.g. postmaster@mail.domain.
the assumption has been that the hostname is something like mail.<domain>, when
setting up mox with the quickstart for user@<domain>. but users can use the
quickstart for postmaster@mail.<domain> as well.

for issue #46 by x8x, thanks for reporting!
2023-08-25 14:32:28 +02:00
Mechiel Lukkien
61a5eb61a4
remove needless fmt.Sprintf
by staticcheck
2023-08-23 16:27:02 +02:00
Mechiel Lukkien
f029db3f47
imapserver bugfix: fix expunging for messages marked junk/nonjunk
such messages would be marked expunged in the database, then the junkfilter
would be retrained for the removal of the message. but during retraining, the
expunged flag would be cleared again. the on-disk message file would still be
removed. so when opening the mailbox, the message would appear to still exist,
but cannot be retrieved from disk.

if you run "mox fixmsgsize", and you get warnings about missing message files,
you could create empty files (with "touch"), run "mox fixsmsgsize" again,
followed by "mox recalculatemailboxcounts <affectedaccount>" and run "mox
bumpuidvalidity <affectaccount>".

"mox backup" would probably also complain, as would "mox verifydata".

this may have caused the "wrong mailbox counts" error i got from "mox
verifydata" on a backup.
2023-08-23 16:20:06 +02:00
Mechiel Lukkien
da9f1d9d0d
in admin pages, make the literal instruction text on the dnscheck page visible, and set a max-width for easier readability 2023-08-23 15:10:02 +02:00
Mechiel Lukkien
b3dd4a55c3
fix a spello, and reword so misspell doesn't complain about it 2023-08-23 14:59:43 +02:00
Mechiel Lukkien
affb057a0c
webmail: fix case where tree of mailboxes wasn't displayed properly
for example, when these mailboxes existed: "a", "a.b", "a/b", then "a.b" (.
before / in ascii) prevented "a/b" from being displayed in the tree below "a".
2023-08-23 14:57:05 +02:00
Mechiel Lukkien
aebfd78a9f
implement accepting dmarc & tls reports for other domains
to accept reports for another domain, first add that domain to the config,
leaving all options empty except DMARC/TLSRPT in which you configure a Domain.

the suggested DNS DMARC/TLSRPT records will show the email address with
configured domain. for DMARC, the dnscheck functionality will verify that the
destination domain has opted in to receiving reports.

there is a new command-line subcommand "mox dmarc checkreportaddrs" that
verifies if dmarc reporting destination addresses have opted in to received
reports.

this also changes the suggested dns records (in quickstart, and through admin
pages and cli subcommand) to take into account whether DMARC and TLSRPT is
configured, and with which localpart/domain (previously it always printed
records as if reporting was enabled for the domain). and when generating the
suggested DNS records, the dmarc.Record and tlsrpt.Record code is used, with
proper uri-escaping.
2023-08-23 14:27:21 +02:00
Mechiel Lukkien
9e248860ee
implement transparent gzip compression in the webserver
we only compress if applicable (content-type indicates likely compressible),
client supports it, response doesn't already have a content-encoding).

for internal handlers, we always enable compression.  for reverse proxied and
static files, compression must be enabled per handler.

for internal & reverse proxy handlers, we do streaming compression at
"bestspeed" quality (probably level 1).

for static files, we have a cache based on mtime with fixed max size, where we
evict based on least recently used. we compress with the default level (more
cpu, better ratio).
2023-08-21 21:52:35 +02:00
Mechiel Lukkien
4c72184b44
update link to docker image
user was being redirected to the new url
2023-08-20 18:45:19 +02:00
Mechiel Lukkien
b43529a2e9
sendmail: bugfix: set remote hostname to verify for tls connections
due to logic bug we weren't setting it, and tls connections would fail with a
warning that either the remote hostname must be set or insecurityskipverify
must be set.
2023-08-20 18:26:20 +02:00
Mechiel Lukkien
0b9475271c
add possible future todo for working around ios messages with wrong q-encoded headers 2023-08-16 16:22:00 +02:00
Mechiel Lukkien
80547df6ee
webmail: don't have two spaces between header and address(es) (e.g. for From/To)
because outlook.com will reformat the message and then fail to verify the message.
proton.me also reformats and invalidates the dkim signature, but seemingly
after it verifies the dkim signature.
2023-08-16 15:22:38 +02:00
Mechiel Lukkien
1ccc5d0177
fix message size in a message in gentestdata
and work around the message in test-upgrade.sh.
and add subcommand to open an account, triggering data upgrades.
2023-08-16 14:36:17 +02:00
Mechiel Lukkien
ddf3cb3653
mention there are now webmail screenshots, and small release process tweaks 2023-08-16 10:16:48 +02:00
Mechiel Lukkien
9f46879377
webmail: correct label for Subject in search form 2023-08-15 13:03:02 +02:00
Mechiel Lukkien
aed23d900a
update dependencies 2023-08-15 10:58:01 +02:00
Mechiel Lukkien
02a03710dc
don't try to (non-recursively) remove directories from the data tmp dir
mox only creates files there. directories could be a backup that is being
transferred to elsewhere.
2023-08-15 09:51:52 +02:00
Mechiel Lukkien
fdbbfb765b
point users to spamhaus and spamcop pages and terms of use 2023-08-15 09:48:53 +02:00
Mechiel Lukkien
983002b074
with strict message parsing, don't allow lines longer than 1000 bytes 2023-08-15 09:21:36 +02:00
Mechiel Lukkien
34c2dcd49d
add strict mode when parsing messages, typically enabled for incoming special-use messages like tls/dmarc reports, subjectpass emails
and pass a logger to the message parser, so problems with message parsing get
the cid logged.
2023-08-15 08:25:56 +02:00
Mechiel Lukkien
f5f953b3ab
handle parsing message header without header/body separator
the commit before the previous added tests with a message with only 1 header
line. it's a valid message, but Go's mail.ReadMessage doesn't handle it with
go1.20 and earlier. the automated "test with previous go release" caught it.
work around it by adding the expected but absent \r\n to the parse function.
2023-08-14 15:40:27 +02:00
Mechiel Lukkien
f96310fdd5
fix checking for tls certificates, and the quickstart with the -existing-webserver flag
some time ago, the flag to ParseConfig() to do or skip checking the tls
keys/certs was inverted, but it looks like i didn't change the call sites... so
during "mox config test", and after a regular "mox quickstart" there was no
check for the tls keys/certs, and during "mox quickstart -existing-webserver"
there was a check where there shouldn't be. this made using -existing-webserver
impossible.

this became clear with the question by morki in issue #5.
2023-08-14 15:01:17 +02:00
Mechiel Lukkien
48eb530b1f
improve message parsing: allow bare carriage return (unless in pedantic mode), allow empty header, and no longer treat a message with only headers as a message with only a body 2023-08-11 14:07:49 +02:00
Mechiel Lukkien
79d06184ab
fix flaky test, event doesn't have to be set 2023-08-11 10:46:22 +02:00
Mechiel Lukkien
55d05c6bea
replace listener config option IPsNATed with NATIPs, and let autotls check NATIPs
NATIPs lists the public IPs, so we can still do the DNS checks on them. with
IPsNATed, we disabled the checks.

based on feedback by kikoreis in issue #52
2023-08-11 10:13:17 +02:00
Mechiel Lukkien
d7df70acd8
webmail: don't lose display of additional headers when a flag/keyword changes (e.g. marked as read) 2023-08-11 08:38:57 +02:00
Mechiel Lukkien
383eb483df
webmail: for html-only messages, also show the "show html with external resources" button 2023-08-10 14:55:30 +02:00
Mechiel Lukkien
a4c6fe815f
make some maintenance commands that were previously unlisted listed
we refer to these commands in output of "mox verifydata", so they should be
findable other than through the code...
2023-08-10 12:29:46 +02:00
Mechiel Lukkien
7cceb3d834
add comment about not verifying Sender for submissions 2023-08-10 12:18:05 +02:00
Mechiel Lukkien
6b68920a3a
Go's LookupAddr will return non-absolute names, seemingly for single-label names from /etc/hosts, turn them into absolute names so our verifying forward lookups can succeed 2023-08-10 11:52:35 +02:00
Mechiel Lukkien
a30d8c1378
for localserve, don't special-case smtp submit
the recent webmail addition added localserve local delivery in queue.Add, so we
just that for smtpserver too.

and don't drop incoming smtp deliver messages, but deliver as normal.
2023-08-10 11:28:57 +02:00
Mechiel Lukkien
ce91b7d23e
update roadmap 2023-08-10 11:05:38 +02:00
Mechiel Lukkien
0434e49c3a
webmail: while attachment viewer is open, don't handle global keyboard shortcuts (like search, going to inbox)
feedback from jonathan, thanks!
2023-08-10 11:02:13 +02:00
Mechiel Lukkien
c24bb063e5
webmail tweaks
- padding on small attachment download button.
- don't remember "show html" but always display text first.
- propagate modseq to message when flags/keywords change, so "show internals" shows the update.
2023-08-10 10:56:04 +02:00
Mechiel Lukkien
f48a53726e
when clearing search, open inbox
feedback from jonathan, thanks!
2023-08-10 10:42:54 +02:00
Mechiel Lukkien
038b478d16
listen/bind in deterministic order for consistent error messages, and warn if quickstart cannot find public ip's
without public ip's, the generated mox config will try to listen on 0.0.0.0 and
::, but because there is already a listener for 127.0.0.1:80 (and possibly
others), a bind for 0.0.0.0:80 will fail. explicit public ip's are needed.

the public http listener is useful for ACME validation over http.

for issue #52
2023-08-10 10:29:06 +02:00
Mechiel Lukkien
01bcd98a42
add flag to ruleset that indicates a message is forwarded, slightly modifying how junk analysis is done
part of PR #50 by bobobo1618
2023-08-09 22:31:37 +02:00
Mechiel Lukkien
9c31789c56
add option to ruleset to accept incoming spammy messages to a configured mailbox
this is based on @bobobo1618's PR #50. bobobo1618 had the right idea, i tried
including an "is forwarded email" configuration option but that indeed became
too tightly coupled. the "is forwarded" option is still planned, but it is
separate from the "accept rejects to mailbox" config option, because one could
still want to push back on forwarded spam messages.

we do an actual accept, delivering to a configured mailbox, instead of storing
to the rejects mailbox where messages can automatically be removed from.  one
of the goals of mox is not pretend to accept email while actually junking it.
users can still configure delivery to a junk folder (as was already possible),
but aren't deleted automatically. there is still an X-Mox-Reason header in the
message, and a log line about accepting the reject, but otherwise it is
registered and treated as an (smtp) accept.

the ruleset mailbox is still required to keep that explicit. users can specify
Inbox again.

hope this is good enough for PR #50, otherwise we'll change it.
2023-08-09 22:25:10 +02:00
Mechiel Lukkien
383fe4f53a
explicitly store in a Message whether it was delivered to the rejects mailbox
soon, we can have multiple rejects mailboxes.  and checking against the
configured rejects mailbox name wasn't foolproof to begin with, because it may
have changed between delivery to the rejects mailbox and the message being
moved.

after upgrading, messages currently in rejects mailboxes don't have IsReject
set, so they don't get the special rejecs treatment when being moved. they are
removed from the rejects mailbox after some time though, and newly added
rejects will be treated correctly. so this means some existing messages wrongly
delivered to the rejects mailbox, and moved out, aren't used (for a positive
signal) for future deliveries.  saves a bit of complexity in the
implementation.  i think the tradeoff is worth it.

related to discussion in issue #50
2023-08-09 16:52:24 +02:00
Mechiel Lukkien
0fc59af9a8
add Deliver-To header for delivered messages
for (experimental) rfc 9228
2023-08-09 10:20:45 +02:00
Mechiel Lukkien
20ebdae8ea
in webmail, automatically mark message as nonjunk when open for 5 seconds, and prevent extraneous newlines when composing a reply to selected text 2023-08-09 09:45:54 +02:00