Commit graph

686 commits

Author SHA1 Message Date
Mechiel Lukkien
beee03574a
mention that imported messages are not deduplicated
so importing twice can result in duplicates.

related to issue #180
2024-06-24 11:46:50 +02:00
Mechiel Lukkien
fdcd2eb0eb
webadmin: remove stray text "pre" in on the "required dns records" page 2024-06-24 10:22:42 +02:00
Mechiel Lukkien
9bab3124f6
show correct host tlsrpt record in dns selfcheck, and make all suggested dns records absolute
the host tlsrpt record implied it was for the domain, but should have been for
the mail host.

some dns records were absolute, others weren't. now they all are for
consistency.

for issue #182 by mdavids, thanks for reporting!
2024-06-22 11:46:12 +02:00
Mechiel Lukkien
ac3596a7d7
try fixing race in tests of ctl socket
there were a few test failures on the github runners. i can't reproduce it
locally. but i can see how they are happening: a gorouting running servectlcmd
could still be doing cleanup (removing files) while a next ctl command was
being run. with this change, we wait for servectlcmd to be done before starting
on a next test.
2024-06-10 23:07:01 +02:00
Mechiel Lukkien
8254e9ce66
webmail: only show "edit" button on drafts, and similar for "e" shortcut
always showing the "edit" button was a bug.
2024-06-10 20:19:17 +02:00
Mechiel Lukkien
a4f7e71457
webmail: ensure white background when viewing attachments, for the black text of plain text attachments
otherwise, in dark mode, the plain text iframe content would be black text on
the white background of the iframe as set by webmail. i can't find a way to set
the content text on the iframe that contains it.
2024-06-10 20:11:26 +02:00
Mechiel Lukkien
f56b04805b
make tests pass with "go test -count n" with n > 1
by closing initialized resources during tests.
2024-06-10 18:18:20 +02:00
Mechiel Lukkien
dde2258f69
update to latest sconf, for improved error messages for mixed tab/space indenting in config files
based on chat with niklas/broitzer
2024-06-10 18:02:47 +02:00
Mechiel Lukkien
aef99a72d8
imapserver: prevent unbounded memory allocations when handling a command
some commands, like search, can specify any number of literals, of arbitrary
size.  we already limited individual literals to 100kb. but you could specify
many of them, causing unbounded memory consumption. this change adds a limit of
1000 literals in a command, and a limit of 1mb of total combined memory for
literals. once the limits are exceeded, a TOOBIG error code is returned.

unbounded memory use could only be triggered on authenticated connections.

this addresses the same issue as CVE-2024-34055 for cyrus-imap, by damian
poddebniak.
2024-06-10 15:00:18 +02:00
Mechiel Lukkien
614576e409
improve http request handling for internal services and multiple domains
per listener, you could enable the admin/account/webmail/webapi handlers. but
that would serve those services on their configured paths (/admin/, /,
/webmail/, /webapi/) on all domains mox would be webserving, including any
non-mail domains. so your www.example/admin/ would be serving the admin web
interface, with no way to disabled that.

with this change, the admin interface is only served on requests to (based on
Host header):
- ip addresses
- the listener host name (explicitly configured in the listener, with fallback
  to global hostname)
- "localhost" (for ssh tunnel/forwarding scenario's)

the account/webmail/webapi interfaces are served on the same domains as the
admin interface, and additionally:
- the client settings domains, as optionally configured in each Domain in
  domains.conf. typically "mail.<yourdomain>".

this means the internal services are no longer served on other domains
configured in the webserver, e.g. www.example.org/admin/ will not be handled
specially.

the order of evaluation of routes/services is also changed:
before this change, the internal handlers would always be evaluated first.
with this change, only the system handlers for
MTA-STS/autoconfig/ACME-validation will be evaluated first. then the webserver
handlers. and finally the internal services (admin/account/webmail/webapi).
this allows an admin to configure overrides for some of the domains (per
hostname-matching rules explained above) that would normally serve these
services.

webserver handlers can now be configured that pass the request to an internal
service: in addition to the existing static/redirect/forward config options,
there is now an "internal" config option, naming the service
(admin/account/webmail/webapi) for handling the request. this allows enabling
the internal services on custom domains.

for issue #160 by TragicLifeHu, thanks for reporting!
2024-05-11 11:13:14 +02:00
Mechiel Lukkien
9152384fd3
use debug logging in tests
by setting the loglevel to debug in package mlog.
we restore the "info" logging in main.
except for "mox localserve", which still sets debug by default.
2024-05-10 15:51:48 +02:00
Mechiel Lukkien
bf8cfd9724
add debug logging about bstore db schema upgrades
bstore was updated to v0.0.6 to add this logging.
this simplifies some of the db-handling code in mtastsdb,tlsrptdb,dmarcdb. we
now call the package-level Init() and Close() in all tests properly.
2024-05-10 14:44:37 +02:00
Mechiel Lukkien
3e4cce826e
webaccount: change xcheckf to handle mox.ErrConfig as user error
like in webadmin
2024-05-09 22:45:44 +02:00
Mechiel Lukkien
3f000fd4e0
make most fields of junk filter configurable by account itself
finally remove the message saying that not all config options can be configured
through the web interface.
2024-05-09 22:45:16 +02:00
Mechiel Lukkien
ebb8ad06b5
use shorter smtp.NewAddress() instead of smtp.Address{...} 2024-05-09 21:26:22 +02:00
Mechiel Lukkien
1179d9d80a
webmail: when opening message in new tab, set document title to subject, message from address(es) and id of message 2024-05-09 21:19:58 +02:00
Mechiel Lukkien
a06a4de5ec
for ctl commands, read all lines before processing, to prevent out of sync protocol when handling errors.
the protocol is often: read one or more lines. only then return error. if we
would return an error after reading 1 line, parsing it and failing, the writer
(client connecting) may be busy writing more lines, not reading an error
response yet.
2024-05-09 21:11:20 +02:00
Mechiel Lukkien
1a0a396713
webmail: in list of From address to use in compose window, don't add the catchall address
it was even selected by default.
2024-05-09 20:55:03 +02:00
Mechiel Lukkien
1fc8f165f7
clarify behaviour of backup command
from RobSlgm, issue #172
2024-05-09 17:48:22 +02:00
Mechiel Lukkien
83004bb18e
give more helpful pointers for dns-related settings
in quickstart, add troubleshooting hints.
in suggested dns records, explain the multiline long dkim record should
probably be converted into a single string.

the quickstart output is quite long already. i'm hoping for a "mox setup" in
the future where we help a user step-by-step to a fully working system. we'll
have more space to prevent hints and check the settings after a user made
changes. it's on the roadmap.

based on issues #158 and #164, thanks vipas84 and RobSlgm for reporting and
discussion.
2024-05-09 17:28:29 +02:00
Mechiel Lukkien
30ac690c8f
when removing account, remove its data directory instead of leaving it around
recreating the account would resurface the old messages, certainly not what you'ld expect.
it's about time to just remove the files. we do ask admins to confirm that when
removing through admin interface. it's also in the "mox config account rm" help
output now.

for issue #162 by RobSlgm with feedback from x8x, thanks!
2024-05-09 16:30:11 +02:00
Mechiel Lukkien
a2c9cfc55b
webadmin: don't show runtime typecheck error for invalid values in dmarc and tls reports
several fields in dmarc and tls reports have known string values. we have a Go
string type for them. sherpats (through sherpadoc) turns those strings into
typescript enums, and sherpats generates runtime-typechecking code (to enforce
correct types for incoming json, to prevent failing deeper in the code when we
get invalid data (much harder to debug)). the Go not-really-enum types allow
other values, and real-world reports have unknown/unspecified/invalid values.
this uses the sherpadoc -rename flag to turn those enums into regular untyped
strings, so sherpats doesn't generate enum-enforcing runtime type checking
code.

this required an update to sherpadoc, to properly handling renaming a type to a
basic type instead of another named type.

for issue #161 by RobSlgm, thanks for reporting!
2024-05-09 15:58:14 +02:00
Mechiel Lukkien
44a6927379
add hint about systemd ReadWritePaths if hardlinking fails on linux due to cross-device link
may help admin figure out more easily how to work around this.

for issue #170 by rdelaage
2024-05-09 14:25:24 +02:00
Mechiel Lukkien
4d28a02621
webmail: better save/close/cancel buttons in compose window
- keep them on the right side of the window (more important now that we can resize)
- merge the close & cancel buttons into a close button, with a popup asking what to do for changes not saved as draft.
2024-05-09 11:46:00 +02:00
Mechiel Lukkien
76aa96ab6f
webadmin: consistent pattern for client api calls wrapped in async/await
adding await in the closure. makes no functional different. but let's stick to one form.
2024-05-09 11:31:04 +02:00
Mechiel Lukkien
98ce133203
update to latest adns with fix for endless loop for incoming corrupt packets 2024-05-09 11:30:42 +02:00
Mechiel Lukkien
09ee89d5c8
update roadmap 2024-05-09 10:51:11 +02:00
Mechiel Lukkien
72be3e8423
webadmin: propagate error when quota size cannot be parsed, improve parsing and hint in error message
- the error wasn't caught because the parseInt() call wasn't evaluated inside the promise.
- we no longer require that the input (e.g. 2G) is the same as how we would format it (2g).
- tooltips and error message should now steer people to using these units.

feedback from pmarini-nc in #115, thanks!
2024-05-09 10:46:18 +02:00
Mechiel Lukkien
db3e44913c
update to latest bbolt
with two changes, both not resulting in different behaviour for us.
2024-05-09 10:32:27 +02:00
Sebastian Haas
587beb75b1 fix typo in SRV validation message
_.tcp => ._tcp
2024-05-07 07:47:26 +02:00
Mechiel Lukkien
a16c08681b
webmail: change many inline styles to using css classes, and add dark mode
this started with looking into the dark mode of PR #163 by mattfbacon. it's a
very good solution, especially for the amount of code. while looking into dark
mode, some common problems with inverting colors are:
- box-shadow start "glowing" which isn't great. likewise, semitransparent
  layers would become brighter, not darker.
- while popups/overlays in light mode just stay the same white, in dark mode
  they should become lighter than the regular content because box shadows don't
  give enough contrast in dark mode.

while looking at adding explicit styles for dark mode, it turns out that's
easier when we work more with css rules/classes instead of inline styles (so we
can use the @media rule).

so we now also create css rules instead of working with inline styles a lot.
benefits:
- creating css rules is useful for items that repeat. they'll have a single css
  class. changing a style on a css class is now reflected in all elements of that
  kind (with that class)
- css class names are helpful when inspecting the DOM while developing: they
  typically describe the function of the element.

most css classes are defined near where they are used, often while making the
element using the class (the css rule is created on first use).

this changes moves colors used for styling to a single place in webmail/lib.ts.
each property can get two values: one for regular/light mode, one for dark mode.
that should prevent forgetting one of them and makes it easy to configure both.
this change sets colors for the dark mode. i think the popups look better than
in PR #163, but in other ways it may be worse. this is a start, we can tweak
the styling.

if we can reduce the number of needed colors some more, we could make them
configurable in the webmail settings in the future. so this is also a step
towards making the ui looks configurable as discussed in issue #107.
2024-05-06 09:13:50 +02:00
Mechiel Lukkien
195c57f06e
update website with latest release v0.0.11 2024-04-30 20:54:32 +02:00
Mechiel Lukkien
7ba18609cd
rotate apidiff/next.txt before release 2024-04-30 20:52:50 +02:00
Mechiel Lukkien
78a59b3476
webadmin: after looking up cid, show it
seems like the useful line of that functionality got lost...
2024-04-29 21:14:05 +02:00
Mechiel Lukkien
5f00f7662e
update readme and docs 2024-04-29 21:10:25 +02:00
Mechiel Lukkien
e34b2c3730
remove log.Print added for debugging 2024-04-29 21:09:41 +02:00
Mechiel Lukkien
b7ec84b80a
queue: when shutting down, wait for pending deliveries before signaling that shutdown is complete
also fixes flaky test, which is how i found it
2024-04-28 22:48:51 +02:00
Mechiel Lukkien
ff6cca1bf9
fix flaky test: close account before marking thread-upgrade as finished
store/threads_test.go opens an account, starts the threading upgrade, waits for
it to finish, runs some tests, and closes the account at the end, verifying all
references are gone. the "thread upgrade" goroutine has its own account
reference. it closes its account after having signaled completion of the
upgrade. in between that time, all checks from the tests could run, its account
closed and its no-more-account-references check would fail. the fix is
hopefully to mark the thread upgrade process finished after closing the
account. hard to verify, but this only happens very rarely.
2024-04-28 14:09:40 +02:00
Mechiel Lukkien
b3a693ee31
update to latest golang.org/x dependencies 2024-04-28 13:53:37 +02:00
Mechiel Lukkien
8cc795b2ec
in smtp submission, if a fromid is present in the mailfrom command, use it when queueing
it's the responsibility of the sender to use unique fromid's.
we do check if that's the case, and return an error if not.

also make it more clear that "unique smtp mail from addresses" map to the
"FromIDLoginAddresses" account config field.

based on feedback from cuu508 for #31, thanks!
2024-04-28 13:18:25 +02:00
Mechiel Lukkien
32cf6500bd
when removing an address, remove it as member from aliases
unless the address is the last member, then the admin must either remove the
alias first, or add new members. we don't want to accidentally remove an alias
address.

in the admin page for removing addresses, we warn the admin that the address
will be removed from any aliases.
2024-04-28 11:44:51 +02:00
Mechiel Lukkien
e2924af8d2
ensure senderaccount is always set for messages in queue
before, the smtpserver that queued a dsn would set an empty senderaccount,
which was interpreted in a few places as the globally configured postmaster
cacount. the empty senderaccount would be used by the smtpserver that queued a
dsn with null return path. we now set the postmaster account when we add a
message to the queue. more code in the queue pretty much needs a non-empty
senderaccount, such as the filters when listing, and the suppression list.
2024-04-28 11:03:47 +02:00
Mechiel Lukkien
6e7f15e0e4
smtpserver tests: use shared function to check expected smtp error codes 2024-04-24 21:00:20 +02:00
Mechiel Lukkien
f749eb2a05
use css white-space: pre-wrap for email addresses displayed
since email addresses can contain multiple consecutive spaces.
this is a valid address: "   "@localhost
and this is a different valid address: " "@localhost

webmail still todo
2024-04-24 20:37:56 +02:00
Mechiel Lukkien
fece75cfe7
automatically install typescript into ./node_modules if missing during build
simplifies process.
2024-04-24 19:48:01 +02:00
Mechiel Lukkien
d9f5625a89
regenerate apidiff, removal due to sherpadoc cleanup 2024-04-24 19:37:47 +02:00
Mechiel Lukkien
960a51242d
add aliases/lists: when sending to an alias, the message gets delivered to all members
the members must currently all be addresses of local accounts.

a message sent to an alias is accepted if at least one of the members accepts
it. if no members accepts it (e.g. due to bad reputation of sender), the
message is rejected.

if a message is submitted to both an alias addresses and to recipients that are
members of the alias in an smtp transaction, the message will be delivered to
such members only once.  the same applies if the address in the message
from-header is the address of a member: that member won't receive the message
(they sent it). this prevents duplicate messages.

aliases have three configuration options:
- PostPublic: whether anyone can send through the alias, or only members.
  members-only lists can be useful inside organizations for internal
  communication. public lists can be useful for support addresses.
- ListMembers: whether members can see the addresses of other members. this can
  be seen in the account web interface. in the future, we could export this in
  other ways, so clients can expand the list.
- AllowMsgFrom: whether messages can be sent through the alias with the alias
  address used in the message from-header. the webmail knows it can use that
  address, and will use it as from-address when replying to a message sent to
  that address.

ideas for the future:
- allow external addresses as members. still with some restrictions, such as
  requiring a valid dkim-signature so delivery has a chance to succeed. will
  also need configuration of an admin that can receive any bounces.
- allow specifying specific members who can sent through the list (instead of
  all members).

for github issue #57 by hmfaysal.
also relevant for #99 by naturalethic.
thanks to damir & marin from sartura for discussing requirements/features.
2024-04-24 19:15:30 +02:00
Mechiel Lukkien
1cf7477642
localserve: change queue to deliver to localserve smtp server
instead of skipping on any smtp and delivering messages to accounts.
we dial the ip of the smtp listener, which is localhost:1025 by default.

the smtp server now uses a mock dns resolver during spf & dkim verification for
hosted domains (localhost by default), so they should pass.

the advantage is that we get regular full smtp server behaviour for delivering
in localserve, including webhooks, and potential first-time sender delays
(though this is disabled by default now).

incoming deliveries now go through normal address resolution, where before we
would always deliver to mox@localhost. we still accept email for unknown
recipients to mox@localhost.

this will be useful upcoming alias/list functionality.

localserve will now generate a dkim key when creating a new config. existing
users may wish to reset (remove) their localserve directory, or add a dkim key.
2024-04-24 11:40:22 +02:00
Mechiel Lukkien
2bb4f78657
remove spurious empty line to fix build, and update roadmap 2024-04-22 14:32:50 +02:00
Mechiel Lukkien
bf5cfca6b9
webmail: add export functionality
per mailbox, or for all mailboxes, in maildir/mbox format, in tar/tgz/zip
archive or without archive format for single mbox, single or recursive. the
webaccount already had an option to export all mailboxes, it now looks similar
to the webmail version.
2024-04-22 13:41:40 +02:00