2023-03-10 18:25:18 +03:00
This file has notes useful for mox developers.
2024-03-26 11:16:33 +03:00
# Building & testing
For a full build, you'll need a recent Go compiler/toolchain and nodejs/npm for
the frontend. First install frontend dependencies (typescript) with "make
install-js". Then run "make build" to do a full build. Run "make test" to run
the test suite. With docker installed, you can run "make test-integration" to
start up a few mox instances, a dns server, a postfix instance, and send email
between them.
The mox localserve command is a convenient way to test locally. Most of the
code paths are reachable/testable with mox localserve, but some use cases will
require a full setup.
Before committing, run at least "make fmt" and "make check" (which requires
staticcheck, run "make install-staticcheck" once). Also run "make check-shadow"
and fix any shadowed variables other than "err" (which are filtered out, but
causes the command to always exit with an error code; run "make install-shadow"
once to install the shadow command). If you've updated RFC references, run
"make" in rfc/, it verifies the referenced files exist.
When making changes to the public API of a package listed in
apidiff/packages.txt, run "make genapidiff" to update the list of changes in
the upcoming release (run "make install-apidiff" once to install the apidiff
command).
New features may be worth mentioning on the website, see website/ and
instructions below.
2023-11-01 20:52:51 +03:00
# Code style, guidelines, notes
make mox compile on windows, without "mox serve" but with working "mox localserve"
getting mox to compile required changing code in only a few places where
package "syscall" was used: for accessing file access times and for umask
handling. an open problem is how to start a process as an unprivileged user on
windows. that's why "mox serve" isn't implemented yet. and just finding a way
to implement it now may not be good enough in the near future: we may want to
starting using a more complete privilege separation approach, with a process
handling sensitive tasks (handling private keys, authentication), where we may
want to pass file descriptors between processes. how would that work on
windows?
anyway, getting mox to compile for windows doesn't mean it works properly on
windows. the largest issue: mox would normally open a file, rename or remove
it, and finally close it. this happens during message delivery. that doesn't
work on windows, the rename/remove would fail because the file is still open.
so this commit swaps many "remove" and "close" calls. renames are a longer
story: message delivery had two ways to deliver: with "consuming" the
(temporary) message file (which would rename it to its final destination), and
without consuming (by hardlinking the file, falling back to copying). the last
delivery to a recipient of a message (and the only one in the common case of a
single recipient) would consume the message, and the earlier recipients would
not. during delivery, the already open message file was used, to parse the
message. we still want to use that open message file, and the caller now stays
responsible for closing it, but we no longer try to rename (consume) the file.
we always hardlink (or copy) during delivery (this works on windows), and the
caller is responsible for closing and removing (in that order) the original
temporary file. this does cost one syscall more. but it makes the delivery code
(responsibilities) a bit simpler.
there is one more obvious issue: the file system path separator. mox already
used the "filepath" package to join paths in many places, but not everywhere.
and it still used strings with slashes for local file access. with this commit,
the code now uses filepath.FromSlash for path strings with slashes, uses
"filepath" in a few more places where it previously didn't. also switches from
"filepath" to regular "path" package when handling mailbox names in a few
places, because those always use forward slashes, regardless of local file
system conventions. windows can handle forward slashes when opening files, so
test code that passes path strings with forward slashes straight to go stdlib
file i/o functions are left unchanged to reduce code churn. the regular
non-test code, or test code that uses path strings in places other than
standard i/o functions, does have the paths converted for consistent paths
(otherwise we would end up with paths with mixed forward/backward slashes in
log messages).
windows cannot dup a listening socket. for "mox localserve", it isn't
important, and we can work around the issue. the current approach for "mox
serve" (forking a process and passing file descriptors of listening sockets on
"privileged" ports) won't work on windows. perhaps it isn't needed on windows,
and any user can listen on "privileged" ports? that would be welcome.
on windows, os.Open cannot open a directory, so we cannot call Sync on it after
message delivery. a cursory internet search indicates that directories cannot
be synced on windows. the story is probably much more nuanced than that, with
long deep technical details/discussions/disagreement/confusion, like on unix.
for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
- Keep the same style as existing code.
- For Windows: use package "path/filepath" when dealing with files/directories.
Test code can pass forward-slashed paths directly to standard library functions,
but use proper filepath functions when parameters are passed and in non-test
code. Mailbox names always use forward slash, so use package "path" for mailbox
name/path manipulation. Do not remove/rename files that are still open.
2023-11-01 20:52:51 +03:00
- Not all code uses adns, the DNSSEC-aware resolver. Such as code that makes
http requests, like mtasts and autotls/autocert.
2023-12-05 15:35:58 +03:00
- We don't have an internal/ directory, really just to prevent long paths in
the repo, and to keep all Go code matching *.go */*.go (without matching
vendor/). Part of the packages are reusable by other software. Those reusable
packages must not cause mox implementation details (such as bstore) to get out,
which would cause unexpected dependencies. Those packages also only expose the
standard slog package for logging, not our mlog package. Packages not intended
for reuse do use mlog as it is more convenient. Internally, we always use
mlog.Log to do the logging, wrapping an slog.Logger.
make mox compile on windows, without "mox serve" but with working "mox localserve"
getting mox to compile required changing code in only a few places where
package "syscall" was used: for accessing file access times and for umask
handling. an open problem is how to start a process as an unprivileged user on
windows. that's why "mox serve" isn't implemented yet. and just finding a way
to implement it now may not be good enough in the near future: we may want to
starting using a more complete privilege separation approach, with a process
handling sensitive tasks (handling private keys, authentication), where we may
want to pass file descriptors between processes. how would that work on
windows?
anyway, getting mox to compile for windows doesn't mean it works properly on
windows. the largest issue: mox would normally open a file, rename or remove
it, and finally close it. this happens during message delivery. that doesn't
work on windows, the rename/remove would fail because the file is still open.
so this commit swaps many "remove" and "close" calls. renames are a longer
story: message delivery had two ways to deliver: with "consuming" the
(temporary) message file (which would rename it to its final destination), and
without consuming (by hardlinking the file, falling back to copying). the last
delivery to a recipient of a message (and the only one in the common case of a
single recipient) would consume the message, and the earlier recipients would
not. during delivery, the already open message file was used, to parse the
message. we still want to use that open message file, and the caller now stays
responsible for closing it, but we no longer try to rename (consume) the file.
we always hardlink (or copy) during delivery (this works on windows), and the
caller is responsible for closing and removing (in that order) the original
temporary file. this does cost one syscall more. but it makes the delivery code
(responsibilities) a bit simpler.
there is one more obvious issue: the file system path separator. mox already
used the "filepath" package to join paths in many places, but not everywhere.
and it still used strings with slashes for local file access. with this commit,
the code now uses filepath.FromSlash for path strings with slashes, uses
"filepath" in a few more places where it previously didn't. also switches from
"filepath" to regular "path" package when handling mailbox names in a few
places, because those always use forward slashes, regardless of local file
system conventions. windows can handle forward slashes when opening files, so
test code that passes path strings with forward slashes straight to go stdlib
file i/o functions are left unchanged to reduce code churn. the regular
non-test code, or test code that uses path strings in places other than
standard i/o functions, does have the paths converted for consistent paths
(otherwise we would end up with paths with mixed forward/backward slashes in
log messages).
windows cannot dup a listening socket. for "mox localserve", it isn't
important, and we can work around the issue. the current approach for "mox
serve" (forking a process and passing file descriptors of listening sockets on
"privileged" ports) won't work on windows. perhaps it isn't needed on windows,
and any user can listen on "privileged" ports? that would be welcome.
on windows, os.Open cannot open a directory, so we cannot call Sync on it after
message delivery. a cursory internet search indicates that directories cannot
be synced on windows. the story is probably much more nuanced than that, with
long deep technical details/discussions/disagreement/confusion, like on unix.
for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
2024-01-10 18:48:53 +03:00
2023-12-11 17:01:56 +03:00
# Reusable packages
Most non-server Go packages are meant to be reusable. This means internal
details are not exposed in the API, and we don't make unneeded changes. We can
still make breaking changes when it improves mox: We don't want to be stuck
with bad API. Third party users aren't affected too seriously due to Go's
minimal version selection. The reusable packages are in apidiff/packages.txt.
We generate the incompatible changes with each release.
make mox compile on windows, without "mox serve" but with working "mox localserve"
getting mox to compile required changing code in only a few places where
package "syscall" was used: for accessing file access times and for umask
handling. an open problem is how to start a process as an unprivileged user on
windows. that's why "mox serve" isn't implemented yet. and just finding a way
to implement it now may not be good enough in the near future: we may want to
starting using a more complete privilege separation approach, with a process
handling sensitive tasks (handling private keys, authentication), where we may
want to pass file descriptors between processes. how would that work on
windows?
anyway, getting mox to compile for windows doesn't mean it works properly on
windows. the largest issue: mox would normally open a file, rename or remove
it, and finally close it. this happens during message delivery. that doesn't
work on windows, the rename/remove would fail because the file is still open.
so this commit swaps many "remove" and "close" calls. renames are a longer
story: message delivery had two ways to deliver: with "consuming" the
(temporary) message file (which would rename it to its final destination), and
without consuming (by hardlinking the file, falling back to copying). the last
delivery to a recipient of a message (and the only one in the common case of a
single recipient) would consume the message, and the earlier recipients would
not. during delivery, the already open message file was used, to parse the
message. we still want to use that open message file, and the caller now stays
responsible for closing it, but we no longer try to rename (consume) the file.
we always hardlink (or copy) during delivery (this works on windows), and the
caller is responsible for closing and removing (in that order) the original
temporary file. this does cost one syscall more. but it makes the delivery code
(responsibilities) a bit simpler.
there is one more obvious issue: the file system path separator. mox already
used the "filepath" package to join paths in many places, but not everywhere.
and it still used strings with slashes for local file access. with this commit,
the code now uses filepath.FromSlash for path strings with slashes, uses
"filepath" in a few more places where it previously didn't. also switches from
"filepath" to regular "path" package when handling mailbox names in a few
places, because those always use forward slashes, regardless of local file
system conventions. windows can handle forward slashes when opening files, so
test code that passes path strings with forward slashes straight to go stdlib
file i/o functions are left unchanged to reduce code churn. the regular
non-test code, or test code that uses path strings in places other than
standard i/o functions, does have the paths converted for consistent paths
(otherwise we would end up with paths with mixed forward/backward slashes in
log messages).
windows cannot dup a listening socket. for "mox localserve", it isn't
important, and we can work around the issue. the current approach for "mox
serve" (forking a process and passing file descriptors of listening sockets on
"privileged" ports) won't work on windows. perhaps it isn't needed on windows,
and any user can listen on "privileged" ports? that would be welcome.
on windows, os.Open cannot open a directory, so we cannot call Sync on it after
message delivery. a cursory internet search indicates that directories cannot
be synced on windows. the story is probably much more nuanced than that, with
long deep technical details/discussions/disagreement/confusion, like on unix.
for "mox localserve" we can get away with making syncdir a no-op.
2023-10-14 11:54:07 +03:00
2024-01-10 18:48:53 +03:00
2023-12-31 13:55:22 +03:00
# Web interfaces/frontend
The web interface frontends (for webmail/, webadmin/ and webaccount/) are
written in strict TypeScript. The web API is a simple self-documenting
HTTP/JSON RPC API mechanism called sherpa,
https://www.ueber.net/who/mjl/sherpa/. The web API exposes types and functions
as implemented in Go, using https://github.com/mjl-/sherpa. API definitions in
JSON form are generated with https://github.com/mjl-/sherpadoc. Those API
definitions are used to generate TypeScript clients with by
https://github.com/mjl-/sherpats/.
The JavaScript that is generated from the TypeScript is included in the
repository. This makes it available for inclusion in the binary, which is
practical for users, and desirable given Go's reproducible builds. When
2024-03-26 11:16:33 +03:00
developing, run "make" to also build the frontend code. Run "make
install-frontend" once to install the TypeScript compiler into ./node_modules/.
2023-12-31 13:55:22 +03:00
There are no other external (runtime or devtime) frontend dependencies. A
light-weight abstraction over the DOM is provided by ./lib.ts. A bit more
manual UI state management must be done compared to "frameworks", but it is
little code, and this allows JavaScript/TypeScript developer to quickly get
started. UI state is often encapsulated in a JavaScript object with a
TypeScript interface exposing a "root" HTMLElement that is added to the DOM,
and functions for accessing/changing the internal state, keeping the UI
managable.
2024-01-10 18:48:53 +03:00
# Website
The content of the public website at https://www.xmox.nl is in website/, as
2024-03-26 11:16:33 +03:00
markdown files. The website HTML is generated with "make genwebsite", which
writes to website/html/ (files not committed). The FAQ is taken from
README.md, the protocol support table is generated from rfc/index.txt. The
website is kept in this repository so a commit can change both the
implementation and the documentation on the website. Some of the info in
README.md is duplicated on the website, often more elaborate and possibly with
a slightly less technical audience. The website should also mostly be readable
through the markdown in the git repo.
2024-01-10 18:48:53 +03:00
Large files (images/videos) are in https://github.com/mjl-/mox-website-files to
keep the repository reasonably sized.
2024-03-26 11:16:33 +03:00
The public website may serve the content from the "website" branch. After a
2024-01-10 18:48:53 +03:00
release release, the main branch (with latest development code and
corresponding changes to the website about new features) is merged into the
website branch. Commits to the website branch (e.g. for a news item, or any
other change unrelated to a new release) is merged back into the main branch.
2023-03-10 18:25:18 +03:00
# TLS certificates
https://github.com/cloudflare/cfssl is useful for testing with TLS
certificates. Create a CA and configure it in mox.conf TLS.CA.CertFiles, and
sign host certificates and configure them in the listeners TLS.KeyCerts.
Setup a local CA with cfssl, run once:
```sh
go install github.com/cloudflare/cfssl/cmd/cfssl@latest
go install github.com/cloudflare/cfssl/cmd/cfssljson@latest
mkdir -p local/cfssl
cd local/cfssl
cfssl print-defaults config > ca-config.json # defaults are fine
# Based on: cfssl print-defaults csr > ca-csr.json
cat <<EOF >ca-csr.json
{
"CN": "mox ca",
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "NL"
}
]
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca - # Generate ca key and cert.
# Generate wildcard certificates for one or more domains, add localhost for use with pebble, see below.
domains="moxtest.example localhost"
for domain in $domains; do
cat <<EOF >wildcard.$domain.csr.json
{
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"O": "mox"
}
],
"hosts": [
"$domain",
"*.$domain"
]
}
EOF
cfssl gencert -ca ca.pem -ca-key ca-key.pem -profile=www wildcard.$domain.csr.json | cfssljson -bare wildcard.$domain
done
```
Now configure mox.conf to add the cfssl CA root certificate:
```
TLS:
CA:
AdditionalToSystem: true
CertFiles:
# Assuming local/<env>/config/mox.conf and local/cfssl/.
- ../../cfssl/ca.pem
[...]
Listeners:
public:
TLS:
KeyCerts:
# Assuming local/<env>/config/mox.conf and local/cfssl/.
CertFile: ../../cfssl/wildcard.$domain.pem
KeyFile: ../../cfssl/wildcard.$domain-key.pem
```
2024-01-10 18:48:53 +03:00
2023-03-10 18:25:18 +03:00
# ACME
https://github.com/letsencrypt/pebble is useful for testing with ACME. Start a
pebble instance that uses the localhost TLS cert/key created by cfssl for its
TLS serving. Pebble generates a new CA certificate for its own use each time it
2023-06-04 21:38:10 +03:00
is started. Fetch it from https://localhost:15000/roots/0, write it to a file, and
2023-03-10 18:25:18 +03:00
add it to mox.conf TLS.CA.CertFiles. See below.
Setup pebble, run once:
```sh
go install github.com/letsencrypt/pebble/cmd/pebble@latest
mkdir -p local/pebble
cat <<EOF >local/pebble/config.json
{
"pebble": {
"listenAddress": "localhost:14000",
"managementListenAddress": "localhost:15000",
"certificate": "local/cfssl/localhost.pem",
"privateKey": "local/cfssl/localhost-key.pem",
"httpPort": 80,
"tlsPort": 443,
"ocspResponderURL": "",
"externalAccountBindingRequired": false
}
}
EOF
```
Start pebble, this generates a new temporary pebble CA certificate:
```sh
pebble -config local/pebble/config.json
```
Write new CA bundle that includes pebble's temporary CA cert:
```sh
export CURL_CA_BUNDLE=local/ca-bundle.pem # for curl
export SSL_CERT_FILE=local/ca-bundle.pem # for go apps
cat /etc/ssl/certs/ca-certificates.crt local/cfssl/ca.pem >local/ca-bundle.pem
2023-06-04 21:38:10 +03:00
curl https://localhost:15000/roots/0 >local/pebble/ca.pem # fetch temp pebble ca, DO THIS EVERY TIME PEBBLE IS RESTARTED!
2023-03-10 18:25:18 +03:00
cat /etc/ssl/certs/ca-certificates.crt local/cfssl/ca.pem local/pebble/ca.pem >local/ca-bundle.pem # create new list that includes cfssl ca and temp pebble ca.
rm -r local/*/data/acme/keycerts/pebble # remove existing pebble-signed certs in acme cert/key cache, they are invalid due to newly generated temp pebble ca.
```
Edit mox.conf, adding pebble ACME and its ca.pem:
```
ACME:
pebble:
DirectoryURL: https://localhost:14000/dir
ContactEmail: root@mox.example
TLS:
CA:
AdditionalToSystem: true
CertFiles:
# Assuming local/<env>/config/mox.conf and local/pebble/ca.pem and local/cfssl/ca.pem.
- ../../pebble/ca.pem
- ../../cfssl/ca.pem
[...]
Listeners:
public:
TLS:
ACME: pebble
```
For mail clients and browsers to accept pebble-signed certificates, you must add
the temporary pebble CA cert to their trusted root CA store each time pebble is
started (e.g. to your thunderbird/firefox testing profile). Pebble has no option
to not regenerate its CA certificate, presumably for fear of people using it for
non-testing purposes. Unfortunately, this also makes it inconvenient to use for
testing purposes.
2023-03-12 23:52:07 +03:00
2023-07-24 12:00:11 +03:00
# Messages for testing
For compatibility and preformance testing, it helps to have many messages,
created a long time ago and recently, by different mail user agents. A helpful
source is the Linux kernel mailing list. Archives are available as multiple git
repositories (split due to size) at
https://lore.kernel.org/lkml/_/text/mirror/. The git repo's can be converted
to compressed mbox files (about 800MB each) with:
```
# 0 is the first epoch (with over half a million messages), 12 is last
# already-complete epoch at the time of writing (with a quarter million
# messages). The archives are large, converting will take some time.
for i in 0 12; do
git clone --mirror http://lore.kernel.org/lkml/$i lkml-$i.git
(cd lkml-$i.git && time ./tombox.sh | gzip >../lkml-$i.mbox.gz)
done
```
With the following "tobmox.sh" script:
```
#!/bin/sh
pre=''
for rev in $(git rev-list master | reverse); do
printf "$pre"
echo "From sender@host $(date '+%a %b %e %H:%M:%S %Y' -d @$(git show -s --format=%ct $rev))"
git show ${rev}:m | sed 's/^>*From />&/'
pre='\n'
done
```
2023-03-12 23:52:07 +03:00
# Release proces
- Gather feedback on recent changes.
- Check if dependencies need updates.
- Check code if there are deprecated features that can be removed.
2024-03-07 12:50:18 +03:00
- Generate apidiff and check if breaking changes can be prevented. Update moxtools.
2023-03-12 23:52:07 +03:00
- Update features & roadmap in README.md
2023-08-16 11:16:48 +03:00
- Write release notes.
2023-03-12 23:52:07 +03:00
- Build and run tests with previous major Go release.
2023-07-24 00:32:02 +03:00
- Run tests, including with race detector.
- Run integration and upgrade tests.
2023-03-12 23:52:07 +03:00
- Run fuzzing tests for a while.
- Deploy to test environment. Test the update instructions.
2024-03-08 17:31:34 +03:00
- Test mox localserve on various OSes (linux, bsd, macos, windows).
2023-03-12 23:52:07 +03:00
- Send and receive email through the major webmail providers, check headers.
- Send and receive email with imap4/smtp clients.
- Check DNS check admin page.
2024-03-09 13:54:38 +03:00
- Check with https://internet.nl.
2024-03-11 17:27:25 +03:00
- Move apidiff/next.txt to apidiff/<version>.txt, and create empty next.txt.
2024-03-09 22:32:54 +03:00
- Add release to the Latest release & News sections of website/index.md.
2023-03-12 23:52:07 +03:00
- Create git tag, push code.
- Publish new docker image.
- Publish signed release notes for updates.xmox.nl and update DNS record.
2024-03-09 22:32:54 +03:00
- Deploy update to website.
2023-03-12 23:52:07 +03:00
- Create new release on the github page, so watchers get a notification.
2024-03-09 22:32:54 +03:00
Copy/paste it manually from the tag text, and add link to download/compile
instructions to prevent confusion about "assets" github links to.
- Publish new cross-referenced code/rfc to www.xmox.nl/xr/.
- Update moxtools with latest version.
2024-03-11 17:27:25 +03:00
- Update implementations support matrix.
