make mox compile on windows, without "mox serve" but with working "mox localserve"

getting mox to compile required changing code in only a few places where
package "syscall" was used: for accessing file access times and for umask
handling. an open problem is how to start a process as an unprivileged user on
windows.  that's why "mox serve" isn't implemented yet. and just finding a way
to implement it now may not be good enough in the near future: we may want to
starting using a more complete privilege separation approach, with a process
handling sensitive tasks (handling private keys, authentication), where we may
want to pass file descriptors between processes. how would that work on
windows?

anyway, getting mox to compile for windows doesn't mean it works properly on
windows. the largest issue: mox would normally open a file, rename or remove
it, and finally close it. this happens during message delivery. that doesn't
work on windows, the rename/remove would fail because the file is still open.
so this commit swaps many "remove" and "close" calls. renames are a longer
story: message delivery had two ways to deliver: with "consuming" the
(temporary) message file (which would rename it to its final destination), and
without consuming (by hardlinking the file, falling back to copying). the last
delivery to a recipient of a message (and the only one in the common case of a
single recipient) would consume the message, and the earlier recipients would
not.  during delivery, the already open message file was used, to parse the
message.  we still want to use that open message file, and the caller now stays
responsible for closing it, but we no longer try to rename (consume) the file.
we always hardlink (or copy) during delivery (this works on windows), and the
caller is responsible for closing and removing (in that order) the original
temporary file. this does cost one syscall more. but it makes the delivery code
(responsibilities) a bit simpler.

there is one more obvious issue: the file system path separator. mox already
used the "filepath" package to join paths in many places, but not everywhere.
and it still used strings with slashes for local file access. with this commit,
the code now uses filepath.FromSlash for path strings with slashes, uses
"filepath" in a few more places where it previously didn't. also switches from
"filepath" to regular "path" package when handling mailbox names in a few
places, because those always use forward slashes, regardless of local file
system conventions.  windows can handle forward slashes when opening files, so
test code that passes path strings with forward slashes straight to go stdlib
file i/o functions are left unchanged to reduce code churn. the regular
non-test code, or test code that uses path strings in places other than
standard i/o functions, does have the paths converted for consistent paths
(otherwise we would end up with paths with mixed forward/backward slashes in
log messages).

windows cannot dup a listening socket. for "mox localserve", it isn't
important, and we can work around the issue. the current approach for "mox
serve" (forking a process and passing file descriptors of listening sockets on
"privileged" ports) won't work on windows. perhaps it isn't needed on windows,
and any user can listen on "privileged" ports? that would be welcome.

on windows, os.Open cannot open a directory, so we cannot call Sync on it after
message delivery. a cursory internet search indicates that directories cannot
be synced on windows. the story is probably much more nuanced than that, with
long deep technical details/discussions/disagreement/confusion, like on unix.
for "mox localserve" we can get away with making syncdir a no-op.

Makefile

@@ -107,3 +107,19 @@ docker:
 
 docker-release:
 	./docker-release.sh
+
+buildall:
+	GOOS=linux GOARCH=arm go build
+	GOOS=linux GOARCH=arm64 go build
+	GOOS=linux GOARCH=amd64 go build
+	GOOS=linux GOARCH=386 go build
+	GOOS=openbsd GOARCH=amd64 go build
+	GOOS=freebsd GOARCH=amd64 go build
+	GOOS=netbsd GOARCH=amd64 go build
+	GOOS=darwin GOARCH=amd64 go build
+	GOOS=dragonfly GOARCH=amd64 go build
+	GOOS=illumos GOARCH=amd64 go build
+	GOOS=solaris GOARCH=amd64 go build
+	GOOS=aix GOARCH=ppc64 go build
+	GOOS=windows GOARCH=amd64 go build
+	# no plan9 for now

README.md

@@ -91,7 +91,10 @@ Verify you have a working mox binary:
 
 	./mox version
 
-Mox only compiles for/works on unix systems, not on Plan 9 or Windows.
+Mox only compiles for and fully works on unix systems. Mox also compiles for
+Windows, but "mox serve" does not yet work, though "mox localserve" (for a
+local test instance) and most other subcommands do. Mox does not compile for
+Plan 9.
 
 You can also run mox with docker image `r.xmox.nl/mox`, with tags like `v0.0.1`
 and `v0.0.1-go1.20.1-alpine3.17.2`, see https://r.xmox.nl/r/mox/. Though new

autotls/autotls.go

@@ -83,7 +83,7 @@ func Load(name, acmeDir, contactEmail, directoryURL string, getPrivateKey func(h
 	}
 
 	// Load identity key if it exists. Otherwise, create a new key.
-	p := filepath.Join(acmeDir + "/" + name + ".key")
+	p := filepath.Join(acmeDir, name+".key")
 	var key crypto.Signer
 	f, err := os.Open(p)
 	if f != nil {
@@ -135,7 +135,7 @@ func Load(name, acmeDir, contactEmail, directoryURL string, getPrivateKey func(h
 	}
 
 	m := &autocert.Manager{
-		Cache:  dirCache(acmeDir + "/keycerts/" + name),
+		Cache:  dirCache(filepath.Join(acmeDir, "keycerts", name)),
 		Prompt: autocert.AcceptTOS,
 		Email:  contactEmail,
 		Client: &acme.Client{

backup.go

@@ -372,7 +372,7 @@ func backupctl(ctx context.Context, ctl *ctl) {
 
 		xvlog("queue backed finished", mlog.Field("duration", time.Since(tmQueue)))
 	}
-	backupQueue("queue/index.db")
+	backupQueue(filepath.FromSlash("queue/index.db"))
 
 	backupAccount := func(acc *store.Account) {
 		defer acc.Close()
@@ -469,7 +469,7 @@ func backupctl(ctx context.Context, ctl *ctl) {
 				return nil
 			}
 			ap := filepath.Join("accounts", acc.Name, p)
-			if strings.HasPrefix(p, "msg/") {
+			if strings.HasPrefix(p, "msg"+string(filepath.Separator)) {
 				xwarnx("backing up unrecognized file in account message directory (should be moved away)", nil, mlog.Field("path", ap))
 			} else {
 				xwarnx("backing up unrecognized file in account directory", nil, mlog.Field("path", ap))

ctl.go

@@ -320,12 +320,11 @@ func servectlcmd(ctx context.Context, ctl *ctl, shutdown func()) {
 		msgFile, err := store.CreateMessageTemp("ctl-deliver")
 		ctl.xcheck(err, "creating temporary message file")
 		defer func() {
-			if msgFile != nil {
+			name := msgFile.Name()
-				err := os.Remove(msgFile.Name())
+			err := msgFile.Close()
-				log.Check(err, "removing temporary message file", mlog.Field("path", msgFile.Name()))
-				err = msgFile.Close()
 			log.Check(err, "closing temporary message file")
-			}
+			err = os.Remove(name)
+			log.Check(err, "removing temporary message file", mlog.Field("path", name))
 		}()
 		mw := message.NewWriter(msgFile)
 		ctl.xwriteok()
@@ -340,14 +339,11 @@ func servectlcmd(ctx context.Context, ctl *ctl, shutdown func()) {
 		}
 
 		a.WithWLock(func() {
-			err := a.DeliverDestination(log, addr, m, msgFile, true)
+			err := a.DeliverDestination(log, addr, m, msgFile)
 			ctl.xcheck(err, "delivering message")
 			log.Info("message delivered through ctl", mlog.Field("to", to))
 		})
 
-		err = msgFile.Close()
-		log.Check(err, "closing delivered message file")
-		msgFile = nil
 		err = a.Close()
 		ctl.xcheck(err, "closing account")
 		ctl.xwriteok()

ctl_test.go

@@ -7,6 +7,7 @@ import (
 	"flag"
 	"net"
 	"os"
+	"path/filepath"
 	"testing"
 
 	"github.com/mjl-/mox/dmarcdb"
@@ -33,8 +34,8 @@ func tcheck(t *testing.T, err error, errmsg string) {
 // unhandled errors would cause a panic.
 func TestCtl(t *testing.T) {
 	os.RemoveAll("testdata/ctl/data")
-	mox.ConfigStaticPath = "testdata/ctl/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("testdata/ctl/mox.conf")
-	mox.ConfigDynamicPath = "testdata/ctl/domains.conf"
+	mox.ConfigDynamicPath = filepath.FromSlash("testdata/ctl/domains.conf")
 	if errs := mox.LoadConfig(ctxbg, true, false); len(errs) > 0 {
 		t.Fatalf("loading mox config: %v", errs)
 	}
@@ -147,13 +148,13 @@ func TestCtl(t *testing.T) {
 	})
 
 	// Export data, import it again
-	xcmdExport(true, []string{"testdata/ctl/data/tmp/export/mbox/", "testdata/ctl/data/accounts/mjl"}, nil)
+	xcmdExport(true, []string{filepath.FromSlash("testdata/ctl/data/tmp/export/mbox/"), filepath.FromSlash("testdata/ctl/data/accounts/mjl")}, nil)
-	xcmdExport(false, []string{"testdata/ctl/data/tmp/export/maildir/", "testdata/ctl/data/accounts/mjl"}, nil)
+	xcmdExport(false, []string{filepath.FromSlash("testdata/ctl/data/tmp/export/maildir/"), filepath.FromSlash("testdata/ctl/data/accounts/mjl")}, nil)
 	testctl(func(ctl *ctl) {
-		ctlcmdImport(ctl, true, "mjl", "inbox", "testdata/ctl/data/tmp/export/mbox/Inbox.mbox")
+		ctlcmdImport(ctl, true, "mjl", "inbox", filepath.FromSlash("testdata/ctl/data/tmp/export/mbox/Inbox.mbox"))
 	})
 	testctl(func(ctl *ctl) {
-		ctlcmdImport(ctl, false, "mjl", "inbox", "testdata/ctl/data/tmp/export/maildir/Inbox")
+		ctlcmdImport(ctl, false, "mjl", "inbox", filepath.FromSlash("testdata/ctl/data/tmp/export/maildir/Inbox"))
 	})
 
 	// "recalculatemailboxcounts"
@@ -177,12 +178,12 @@ func TestCtl(t *testing.T) {
 			m.Size = int64(len(content))
 			msgf, err := store.CreateMessageTemp("ctltest")
 			tcheck(t, err, "create temp file")
+			defer os.Remove(msgf.Name())
+			defer msgf.Close()
 			_, err = msgf.Write(content)
 			tcheck(t, err, "write message file")
-			err = acc.DeliverMailbox(xlog, "Inbox", m, msgf, true)
+			err = acc.DeliverMailbox(xlog, "Inbox", m, msgf)
 			tcheck(t, err, "deliver message")
-			err = msgf.Close()
-			tcheck(t, err, "close message file")
 		}
 
 		var msgBadSize store.Message
@@ -236,13 +237,13 @@ func TestCtl(t *testing.T) {
 		os.RemoveAll("testdata/ctl/data/tmp/backup-data")
 		err := os.WriteFile("testdata/ctl/data/receivedid.key", make([]byte, 16), 0600)
 		tcheck(t, err, "writing receivedid.key")
-		ctlcmdBackup(ctl, "testdata/ctl/data/tmp/backup-data", false)
+		ctlcmdBackup(ctl, filepath.FromSlash("testdata/ctl/data/tmp/backup-data"), false)
 	})
 
 	// Verify the backup.
 	xcmd := cmd{
 		flag:     flag.NewFlagSet("", flag.ExitOnError),
-		flagArgs: []string{"testdata/ctl/data/tmp/backup-data"},
+		flagArgs: []string{filepath.FromSlash("testdata/ctl/data/tmp/backup-data")},
 	}
 	cmdVerifydata(&xcmd)
 }

develop.txt

@@ -1,5 +1,15 @@
 This file has notes useful for mox developers.
 
+# Code style & guidelines
+
+- Keep the same style as existing code.
+- For Windows: use package "path/filepath" when dealing with files/directories.
+  Test code can pass forward-slashed paths directly to standard library functions,
+  but use proper filepath functions when parameters are passed and in non-test
+  code.  Mailbox names always use forward slash, so use package "path" for mailbox
+  name/path manipulation. Do not remove/rename files that are still open.
+
+
 # TLS certificates
 
 https://github.com/cloudflare/cfssl is useful for testing with TLS

dmarcdb/db_test.go

@@ -17,16 +17,17 @@ var ctxbg = context.Background()
 
 func TestDMARCDB(t *testing.T) {
 	mox.Shutdown = ctxbg
-	mox.ConfigStaticPath = "../testdata/dmarcdb/fake.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/dmarcdb/fake.conf")
 	mox.Conf.Static.DataDir = "."
 
 	dbpath := mox.DataDirPath("dmarcrpt.db")
 	os.MkdirAll(filepath.Dir(dbpath), 0770)
-	defer os.Remove(dbpath)
 
 	if err := Init(); err != nil {
 		t.Fatalf("init database: %s", err)
 	}
+	defer os.Remove(dbpath)
+	defer DB.Close()
 
 	feedback := &dmarcrpt.Feedback{
 		ReportMetadata: dmarcrpt.ReportMetadata{

dmarcrpt/parse_test.go

@@ -2,6 +2,7 @@ package dmarcrpt
 
 import (
 	"os"
+	"path/filepath"
 	"reflect"
 	"strings"
 	"testing"
@@ -122,14 +123,14 @@ func TestParseReport(t *testing.T) {
 }
 
 func TestParseMessageReport(t *testing.T) {
-	const dir = "../testdata/dmarc-reports"
+	dir := filepath.FromSlash("../testdata/dmarc-reports")
 	files, err := os.ReadDir(dir)
 	if err != nil {
 		t.Fatalf("listing dmarc report emails: %s", err)
 	}
 
 	for _, file := range files {
-		p := dir + "/" + file.Name()
+		p := filepath.Join(dir, file.Name())
 		f, err := os.Open(p)
 		if err != nil {
 			t.Fatalf("open %q: %s", p, err)

doc.go

@@ -89,6 +89,8 @@ IMAP. HTTP listeners are started for the admin/account web interfaces, and for
 automated TLS configuration. Missing essential TLS certificates are immediately
 requested, other TLS certificates are requested on demand.
 
+Only implemented on unix systems, not Windows.
+
 	usage: mox serve
 
 # mox quickstart

dsn/dsn_test.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"io"
 	"net"
+	"path/filepath"
 	"reflect"
 	"strings"
 	"testing"
@@ -131,7 +132,7 @@ func TestDSN(t *testing.T) {
 
 	// Test for valid DKIM signature.
 	mox.Context = context.Background()
-	mox.ConfigStaticPath = "../testdata/dsn/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/dsn/mox.conf")
 	mox.MustLoadConfig(true, false)
 	msgbuf, err = m.Compose(log, false)
 	if err != nil {

gentestdata.go

@@ -85,8 +85,8 @@ Accounts:
 				IgnoreWords: 0.1
 `
 
-	mox.ConfigStaticPath = "/tmp/mox-bogus/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("/tmp/mox-bogus/mox.conf")
-	mox.ConfigDynamicPath = "/tmp/mox-bogus/domains.conf"
+	mox.ConfigDynamicPath = filepath.FromSlash("/tmp/mox-bogus/domains.conf")
 	mox.Conf.DynamicLastCheck = time.Now() // Should prevent warning.
 	mox.Conf.Static = config.Static{
 		DataDir: destDataDir,
@@ -228,11 +228,12 @@ Accounts:
 	prefix := []byte{}
 	mf := tempfile()
 	xcheckf(err, "temp file for queue message")
+	defer os.Remove(mf.Name())
 	defer mf.Close()
 	const qmsg = "From: <test0@mox.example>\r\nTo: <other@remote.example>\r\nSubject: test\r\n\r\nthe message...\r\n"
 	_, err = fmt.Fprint(mf, qmsg)
 	xcheckf(err, "writing message")
-	_, err = queue.Add(ctxbg, log, "test0", mailfrom, rcptto, false, false, int64(len(qmsg)), "<test@localhost>", prefix, mf, nil, true)
+	_, err = queue.Add(ctxbg, log, "test0", mailfrom, rcptto, false, false, int64(len(qmsg)), "<test@localhost>", prefix, mf, nil)
 	xcheckf(err, "enqueue message")
 
 	// Create three accounts.
@@ -283,10 +284,14 @@ Accounts:
 		xcheckf(err, "creating temp file for delivery")
 		_, err = fmt.Fprint(mf, msg)
 		xcheckf(err, "writing deliver message to file")
-		err = accTest1.DeliverMessage(log, tx, &m, mf, true, false, true, false)
+		err = accTest1.DeliverMessage(log, tx, &m, mf, false, true, false)
+
+		mfname := mf.Name()
 		xcheckf(err, "add message to account test1")
 		err = mf.Close()
 		xcheckf(err, "closing file")
+		err = os.Remove(mfname)
+		xcheckf(err, "removing temp message file")
 
 		err = tx.Get(&inbox)
 		xcheckf(err, "get inbox")
@@ -339,10 +344,14 @@ Accounts:
 		xcheckf(err, "creating temp file for delivery")
 		_, err = fmt.Fprint(mf0, msg0)
 		xcheckf(err, "writing deliver message to file")
-		err = accTest2.DeliverMessage(log, tx, &m0, mf0, true, false, false, false)
+		err = accTest2.DeliverMessage(log, tx, &m0, mf0, false, false, false)
 		xcheckf(err, "add message to account test2")
+
+		mf0name := mf0.Name()
 		err = mf0.Close()
 		xcheckf(err, "closing file")
+		err = os.Remove(mf0name)
+		xcheckf(err, "removing temp message file")
 
 		err = tx.Get(&inbox)
 		xcheckf(err, "get inbox")
@@ -366,10 +375,14 @@ Accounts:
 		xcheckf(err, "creating temp file for delivery")
 		_, err = fmt.Fprint(mf1, msg1)
 		xcheckf(err, "writing deliver message to file")
-		err = accTest2.DeliverMessage(log, tx, &m1, mf1, true, false, false, false)
+		err = accTest2.DeliverMessage(log, tx, &m1, mf1, false, false, false)
 		xcheckf(err, "add message to account test2")
+
+		mf1name := mf1.Name()
 		err = mf1.Close()
 		xcheckf(err, "closing file")
+		err = os.Remove(mf1name)
+		xcheckf(err, "removing temp message file")
 
 		err = tx.Get(&sent)
 		xcheckf(err, "get sent")

go.mod

@@ -3,7 +3,7 @@ module github.com/mjl-/mox
 go 1.20
 
 require (
-	github.com/mjl-/adns v0.0.0-20231009145311-e3834995f16c
+	github.com/mjl-/adns v0.0.0-20231013194548-ea0378d616ab
 	github.com/mjl-/autocert v0.0.0-20231013072455-c361ae2e20a6
 	github.com/mjl-/bstore v0.0.2
 	github.com/mjl-/sconf v0.0.5

go.sum

@@ -145,10 +145,8 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
-github.com/mjl-/adns v0.0.0-20231009145311-e3834995f16c h1:ZOr9KnCxfAwJWSeZn8Qs6cSF7TrmBa8hVIpLcEvx/Ec=
+github.com/mjl-/adns v0.0.0-20231013194548-ea0378d616ab h1:fL+dZP+IxX08+ugLq42bkvOfV42muXET+T+Ei1K16bI=
-github.com/mjl-/adns v0.0.0-20231009145311-e3834995f16c/go.mod h1:JWhGACVviyVUEra9Zv1M8JMkDVXArVt+AIXjTXtuwb4=
+github.com/mjl-/adns v0.0.0-20231013194548-ea0378d616ab/go.mod h1:v47qUMJnipnmDTRGaHwpCwzE6oypa5K33mUvBfzZBn8=
-github.com/mjl-/autocert v0.0.0-20231009155929-d0d48f2f0290 h1:0hCRSu8+XCZ2cSRW+ZtP/7L5wMYjOKFSQthoyj+4cN8=
-github.com/mjl-/autocert v0.0.0-20231009155929-d0d48f2f0290/go.mod h1:taMFU86abMxKLPV4Bynhv8enbYmS67b8LG80qZv2Qus=
 github.com/mjl-/autocert v0.0.0-20231013072455-c361ae2e20a6 h1:TEXyTghAN9pmV2ffzdnhmzkML08e1Z/oGywJ9eunbRI=
 github.com/mjl-/autocert v0.0.0-20231013072455-c361ae2e20a6/go.mod h1:taMFU86abMxKLPV4Bynhv8enbYmS67b8LG80qZv2Qus=
 github.com/mjl-/bstore v0.0.2 h1:4fdpIOY/+Dv1dBHyzdqa4PD90p8Mz86FeyRpI4qcehw=

http/atime.go

@@ -1,9 +1,16 @@
-//go:build !netbsd && !freebsd && !darwin
+//go:build !netbsd && !freebsd && !darwin && !windows
 
 package http
 
-import "syscall"
+import (
+	"fmt"
+	"syscall"
+)
 
-func statAtime(sys *syscall.Stat_t) int64 {
+func statAtime(sys any) (int64, error) {
-	return int64(sys.Atim.Sec)*1000*1000*1000 + int64(sys.Atim.Nsec)
+	x, ok := sys.(*syscall.Stat_t)
+	if !ok {
+		return 0, fmt.Errorf("sys is a %T, expected *syscall.Stat_t", sys)
+	}
+	return int64(x.Atim.Sec)*1000*1000*1000 + int64(x.Atim.Nsec), nil
 }

http/atime_bsd.go

@@ -2,8 +2,15 @@
 
 package http
 
-import "syscall"
+import (
+	"fmt"
+	"syscall"
+)
 
-func statAtime(sys *syscall.Stat_t) int64 {
+func statAtime(sys any) (int64, error) {
-	return int64(sys.Atimespec.Sec)*1000*1000*1000 + int64(sys.Atimespec.Nsec)
+	x, ok := sys.(*syscall.Stat_t)
+	if !ok {
+		return 0, fmt.Errorf("stat sys is a %T, expected *syscall.Stat_t", sys)
+	}
+	return int64(x.Atimespec.Sec)*1000*1000*1000 + int64(x.Atimespec.Nsec), nil
 }

http/atime_windows.go

@@ -0,0 +1,16 @@
+//go:build windows
+
+package http
+
+import (
+	"fmt"
+	"syscall"
+)
+
+func statAtime(sys any) (int64, error) {
+	x, ok := sys.(*syscall.Win32FileAttributeData)
+	if !ok {
+		return 0, fmt.Errorf("sys is a %T, expected *syscall.Win32FileAttributeData", sys)
+	}
+	return x.LastAccessTime.Nanoseconds(), nil
+}

http/gzcache.go

@@ -14,7 +14,6 @@ import (
 	"strconv"
 	"strings"
 	"sync"
-	"syscall"
 	"time"
 
 	"github.com/mjl-/mox/mlog"
@@ -109,11 +108,7 @@ func loadStaticGzipCache(dir string, maxSize int64) {
 		}
 		var atime int64
 		if err == nil {
-			if sys, sysok := fi.Sys().(*syscall.Stat_t); !sysok {
+			atime, err = statAtime(fi.Sys())
-				err = fmt.Errorf("FileInfo.Sys not a *syscall.Stat_t but %T", fi.Sys())
-			} else {
-				atime = statAtime(sys)
-			}
 		}
 		if err != nil {
 			xlog.Infox("removing unusable/unrecognized file in static gzip cache dir", err)

http/web_test.go

@@ -15,7 +15,7 @@ import (
 
 func TestServeHTTP(t *testing.T) {
 	os.RemoveAll("../testdata/web/data")
-	mox.ConfigStaticPath = "../testdata/web/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/web/mox.conf")
 	mox.ConfigDynamicPath = filepath.Join(filepath.Dir(mox.ConfigStaticPath), "domains.conf")
 	mox.MustLoadConfig(true, false)
 

http/webserver_test.go

@@ -27,7 +27,7 @@ func tcheck(t *testing.T, err error, msg string) {
 
 func TestWebserver(t *testing.T) {
 	os.RemoveAll("../testdata/webserver/data")
-	mox.ConfigStaticPath = "../testdata/webserver/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/webserver/mox.conf")
 	mox.ConfigDynamicPath = filepath.Join(filepath.Dir(mox.ConfigStaticPath), "domains.conf")
 	mox.MustLoadConfig(true, false)
 
@@ -158,7 +158,7 @@ func TestWebserver(t *testing.T) {
 
 func TestWebsocket(t *testing.T) {
 	os.RemoveAll("../testdata/websocket/data")
-	mox.ConfigStaticPath = "../testdata/websocket/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/websocket/mox.conf")
 	mox.ConfigDynamicPath = filepath.Join(filepath.Dir(mox.ConfigStaticPath), "domains.conf")
 	mox.MustLoadConfig(true, false)
 

imapserver/fuzz_test.go

@@ -7,6 +7,7 @@ import (
 	"io"
 	"net"
 	"os"
+	"path/filepath"
 	"testing"
 	"time"
 
@@ -58,7 +59,7 @@ func FuzzServer(f *testing.F) {
 	}
 
 	mox.Context = ctxbg
-	mox.ConfigStaticPath = "../testdata/imapserverfuzz/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/imapserverfuzz/mox.conf")
 	mox.MustLoadConfig(true, false)
 	dataDir := mox.ConfigDirPath(mox.Conf.Static.DataDir)
 	os.RemoveAll(dataDir)

imapserver/list.go

@@ -2,7 +2,7 @@ package imapserver
 
 import (
 	"fmt"
-	"path/filepath"
+	"path"
 	"sort"
 	"strings"
 
@@ -133,7 +133,7 @@ func (c *conn) cmdList(tag, cmd string, p *parser) {
 			err := q.ForEach(func(mb store.Mailbox) error {
 				names[mb.Name] = info{mailbox: &mb}
 				nameList = append(nameList, mb.Name)
-				for p := filepath.Dir(mb.Name); p != "."; p = filepath.Dir(p) {
+				for p := path.Dir(mb.Name); p != "."; p = path.Dir(p) {
 					hasChild[p] = true
 				}
 				return nil
@@ -148,7 +148,7 @@ func (c *conn) cmdList(tag, cmd string, p *parser) {
 				if !ok {
 					nameList = append(nameList, sub.Name)
 				}
-				for p := filepath.Dir(sub.Name); p != "."; p = filepath.Dir(p) {
+				for p := path.Dir(sub.Name); p != "."; p = path.Dir(p) {
 					hasSubscribedChild[p] = true
 				}
 				return nil

imapserver/server.go

@@ -50,6 +50,7 @@ import (
 	"math"
 	"net"
 	"os"
+	"path"
 	"path/filepath"
 	"regexp"
 	"runtime/debug"
@@ -895,7 +896,7 @@ func xmailboxPatternMatcher(ref string, patterns []string) matchStringer {
 
 		s := pat
 		if ref != "" {
-			s = filepath.Join(ref, pat)
+			s = path.Join(ref, pat)
 		}
 
 		// Fix casing for all Inbox paths.
@@ -2481,7 +2482,7 @@ func (c *conn) cmdLsub(tag, cmd string, p *parser) {
 		for _, sub := range subscriptions {
 			name := sub.Name
 			if ispercent {
-				for p := filepath.Dir(name); p != "."; p = filepath.Dir(p) {
+				for p := path.Dir(name); p != "."; p = path.Dir(p) {
 					subscribedKids[p] = true
 				}
 			}
@@ -2675,12 +2676,11 @@ func (c *conn) cmdAppend(tag, cmd string, p *parser) {
 	msgFile, err := store.CreateMessageTemp("imap-append")
 	xcheckf(err, "creating temp file for message")
 	defer func() {
-		if msgFile != nil {
+		p := msgFile.Name()
-			err := os.Remove(msgFile.Name())
+		err := msgFile.Close()
-			c.xsanity(err, "removing APPEND temporary file")
-			err = msgFile.Close()
 		c.xsanity(err, "closing APPEND temporary file")
-		}
+		err = os.Remove(p)
+		c.xsanity(err, "removing APPEND temporary file")
 	}()
 	defer c.xtrace(mlog.LevelTracedata)()
 	mw := message.NewWriter(msgFile)
@@ -2740,7 +2740,7 @@ func (c *conn) cmdAppend(tag, cmd string, p *parser) {
 			err = tx.Update(&mb)
 			xcheckf(err, "updating mailbox counts")
 
-			err := c.account.DeliverMessage(c.log, tx, &m, msgFile, true, true, false, false)
+			err := c.account.DeliverMessage(c.log, tx, &m, msgFile, true, false, false)
 			xcheckf(err, "delivering message")
 		})
 
@@ -2754,10 +2754,6 @@ func (c *conn) cmdAppend(tag, cmd string, p *parser) {
 		c.broadcast(changes)
 	})
 
-	err = msgFile.Close()
-	c.log.Check(err, "closing appended file")
-	msgFile = nil
-
 	if c.mailboxID == mb.ID {
 		c.applyChanges(pendingChanges, false)
 		c.uidAppend(m.UID)

imapserver/server_test.go

@@ -10,6 +10,7 @@ import (
 	"math/big"
 	"net"
 	"os"
+	"path/filepath"
 	"reflect"
 	"strings"
 	"testing"
@@ -338,7 +339,7 @@ func startArgs(t *testing.T, first, isTLS, allowLoginWithoutTLS bool) *testconn
 		os.RemoveAll("../testdata/imap/data")
 	}
 	mox.Context = ctxbg
-	mox.ConfigStaticPath = "../testdata/imap/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/imap/mox.conf")
 	mox.MustLoadConfig(true, false)
 	acc, err := store.OpenAccount("mjl")
 	tcheck(t, err, "open account")

import.go

@@ -276,11 +276,10 @@ func importctl(ctx context.Context, ctl *ctl, mbox bool) {
 	xdeliver := func(m *store.Message, mf *os.File) {
 		// todo: possibly set dmarcdomain to the domain of the from address? at least for non-spams that have been seen. otherwise user would start without any reputations. the assumption would be that the user has accepted email and deemed it legit, coming from the indicated sender.
 
-		const consumeFile = true
 		const sync = false
 		const notrain = true
 		const nothreads = true
-		err := a.DeliverMessage(ctl.log, tx, m, mf, consumeFile, sync, notrain, nothreads)
+		err := a.DeliverMessage(ctl.log, tx, m, mf, sync, notrain, nothreads)
 		ctl.xcheck(err, "delivering message")
 		deliveredIDs = append(deliveredIDs, m.ID)
 		ctl.log.Debug("delivered message", mlog.Field("id", m.ID))
@@ -313,13 +312,11 @@ func importctl(ctx context.Context, ctl *ctl, mbox bool) {
 
 		process := func(m *store.Message, msgf *os.File, origPath string) {
 			defer func() {
-				if msgf == nil {
+				name := msgf.Name()
-					return
+				err := msgf.Close()
-				}
-				err := os.Remove(msgf.Name())
-				ctl.log.Check(err, "removing temporary message after failing to import")
-				err = msgf.Close()
 				ctl.log.Check(err, "closing temporary message after failing to import")
+				err = os.Remove(name)
+				ctl.log.Check(err, "removing temporary message after failing to import", mlog.Field("path", name))
 			}()
 
 			for _, kw := range m.Keywords {
@@ -373,9 +370,6 @@ func importctl(ctx context.Context, ctl *ctl, mbox bool) {
 			m.CreateSeq = modseq
 			m.ModSeq = modseq
 			xdeliver(m, msgf)
-			err = msgf.Close()
-			ctl.log.Check(err, "closing message after delivery")
-			msgf = nil
 
 			n++
 			if n%1000 == 0 {

junk.go

@@ -20,6 +20,7 @@ import (
 	"log"
 	mathrand "math/rand"
 	"os"
+	"path/filepath"
 	"sort"
 	"time"
 
@@ -157,7 +158,7 @@ func cmdJunkTest(c *cmd) {
 		files, err := os.ReadDir(dir)
 		xcheckf(err, "readdir %q", dir)
 		for _, fi := range files {
-			path := dir + "/" + fi.Name()
+			path := filepath.Join(dir, fi.Name())
 			prob, _, _, _, err := f.ClassifyMessagePath(context.Background(), path)
 			if err != nil {
 				log.Printf("classify message %q: %s", path, err)
@@ -249,7 +250,7 @@ messages are shuffled, with optional random seed.`
 
 	testDir := func(dir string, files []string, ham bool) (ok, bad, malformed int) {
 		for _, name := range files {
-			path := dir + "/" + name
+			path := filepath.Join(dir, name)
 			prob, _, _, _, err := f.ClassifyMessagePath(context.Background(), path)
 			if err != nil {
 				// log.Infof("%s: %s", path, err)
@@ -313,7 +314,7 @@ func cmdJunkPlay(c *cmd) {
 
 	scanDir := func(dir string, ham, sent bool) {
 		for _, name := range listDir(dir) {
-			path := dir + "/" + name
+			path := filepath.Join(dir, name)
 			mf, err := os.Open(path)
 			xcheckf(err, "open %q", path)
 			fi, err := mf.Stat()
@@ -366,7 +367,7 @@ func cmdJunkPlay(c *cmd) {
 
 	play := func(msg msg) {
 		var words map[string]struct{}
-		path := msg.dir + "/" + msg.filename
+		path := filepath.Join(msg.dir, msg.filename)
 		if !msg.sent {
 			var prob float64
 			var err error

junk/filter.go

@@ -17,6 +17,7 @@ import (
 	"io"
 	"math"
 	"os"
+	"path/filepath"
 	"sort"
 	"time"
 
@@ -644,7 +645,7 @@ func (f *Filter) TrainDir(dir string, files []string, ham bool) (n, malformed ui
 	}
 
 	for _, name := range files {
-		p := fmt.Sprintf("%s/%s", dir, name)
+		p := filepath.Join(dir, name)
 		valid, words, err := f.tokenizeMail(p)
 		if err != nil {
 			// f.log.Infox("tokenizing mail", err, mlog.Field("path", p))

junk/filter_test.go

@@ -42,8 +42,8 @@ func TestFilter(t *testing.T) {
 		IgnoreWords: 0.1,
 		RareWords:   1,
 	}
-	dbPath := "../testdata/junk/filter.db"
+	dbPath := filepath.FromSlash("../testdata/junk/filter.db")
-	bloomPath := "../testdata/junk/filter.bloom"
+	bloomPath := filepath.FromSlash("../testdata/junk/filter.bloom")
 	os.Remove(dbPath)
 	os.Remove(bloomPath)
 	f, err := NewFilter(ctxbg, log, params, dbPath, bloomPath)
@@ -59,8 +59,8 @@ func TestFilter(t *testing.T) {
 	os.MkdirAll("../testdata/train/ham", 0770)
 	os.MkdirAll("../testdata/train/spam", 0770)
 
-	hamdir := "../testdata/train/ham"
+	hamdir := filepath.FromSlash("../testdata/train/ham")
-	spamdir := "../testdata/train/spam"
+	spamdir := filepath.FromSlash("../testdata/train/spam")
 	hamfiles := tlistdir(t, hamdir)
 	if len(hamfiles) > 100 {
 		hamfiles = hamfiles[:100]

junk/parse_test.go

@@ -2,6 +2,7 @@ package junk
 
 import (
 	"os"
+	"path/filepath"
 	"testing"
 )
 
@@ -14,12 +15,12 @@ func FuzzParseMessage(f *testing.F) {
 		}
 		f.Add(string(buf))
 	}
-	add("../testdata/junk/parse.eml")
+	add(filepath.FromSlash("../testdata/junk/parse.eml"))
-	add("../testdata/junk/parse2.eml")
+	add(filepath.FromSlash("../testdata/junk/parse2.eml"))
-	add("../testdata/junk/parse3.eml")
+	add(filepath.FromSlash("../testdata/junk/parse3.eml"))
 
-	dbPath := "../testdata/junk/parse.db"
+	dbPath := filepath.FromSlash("../testdata/junk/parse.db")
-	bloomPath := "../testdata/junk/parse.bloom"
+	bloomPath := filepath.FromSlash("../testdata/junk/parse.bloom")
 	os.Remove(dbPath)
 	os.Remove(bloomPath)
 	params := Params{Twograms: true}

localserve.go

@@ -321,11 +321,15 @@ func writeLocalConfig(log *mlog.Log, dir, ip string) (rerr error) {
 	local.WebserverHTTPS.Enabled = true
 	local.WebserverHTTPS.Port = 1443
 
+	uid := os.Getuid()
+	if uid < 0 {
+		uid = 1 // For windows.
+	}
 	static := config.Static{
 		DataDir:           ".",
 		LogLevel:          "traceauth",
 		Hostname:          "localhost",
-		User:              fmt.Sprintf("%d", os.Getuid()),
+		User:              fmt.Sprintf("%d", uid),
 		AdminPasswordFile: "adminpasswd",
 		Pedantic:          true,
 		Listeners: map[string]config.Listener{

main.go

@@ -428,7 +428,7 @@ func main() {
 		return
 	}
 
-	flag.StringVar(&mox.ConfigStaticPath, "config", envString("MOXCONF", "config/mox.conf"), "configuration file, other config files are looked up in the same directory, defaults to $MOXCONF with a fallback to mox.conf")
+	flag.StringVar(&mox.ConfigStaticPath, "config", envString("MOXCONF", filepath.FromSlash("config/mox.conf")), "configuration file, other config files are looked up in the same directory, defaults to $MOXCONF with a fallback to mox.conf")
 	flag.StringVar(&loglevel, "loglevel", "", "if non-empty, this log level is set early in startup")
 	flag.BoolVar(&pedantic, "pedantic", false, "protocol violations result in errors instead of accepting/working around them")
 	flag.BoolVar(&store.CheckConsistencyOnClose, "checkconsistency", false, "dangerous option for testing only, enables data checks that abort/panic when inconsistencies are found")
@@ -1050,7 +1050,7 @@ for a domain and create the TLSA DNS records it suggests to enable DANE.
 			p := mox.DataDirPath(filepath.Join("acme", "keycerts", l.TLS.ACME, filename))
 			privKey := xtryLoadPrivateKey(kt, p)
 
-			relPath := fmt.Sprintf("hostkeys/%s.%s.%s.privatekey.pkcs8.pem", host.Name(), timestamp, kind)
+			relPath := filepath.Join("hostkeys", fmt.Sprintf("%s.%s.%s.privatekey.pkcs8.pem", host.Name(), timestamp, kind))
 			destPath := mox.ConfigDirPath(relPath)
 			err := writeHostPrivateKey(privKey, destPath)
 			xcheckf(err, "writing host private key file to %s: %v", destPath, err)

mox-/admin.go

@@ -178,10 +178,10 @@ func MakeDomainConfig(ctx context.Context, domain, hostname dns.Domain, accountN
 		}
 		defer func() {
 			if f != nil {
-				err := os.Remove(path)
+				err := f.Close()
-				log.Check(err, "removing file after error")
-				err = f.Close()
 				log.Check(err, "closing file after error")
+				err = os.Remove(path)
+				log.Check(err, "removing file after error", mlog.Field("path", path))
 			}
 		}()
 		if _, err := f.Write(data); err != nil {

mox-/config.go

@@ -471,8 +471,7 @@ func PrepareStaticConfig(ctx context.Context, configFile string, conf *Config, c
 		c.User = "mox"
 	}
 	u, err := user.Lookup(c.User)
-	var userErr user.UnknownUserError
+	if err != nil {
-	if err != nil && errors.As(err, &userErr) {
 		uid, err := strconv.ParseUint(c.User, 10, 32)
 		if err != nil {
 			addErrorf("parsing unknown user %s as uid: %v (hint: add user mox with \"useradd -d $PWD mox\" or specify a different username on the quickstart command-line)", c.User, err)
@@ -481,8 +480,6 @@ func PrepareStaticConfig(ctx context.Context, configFile string, conf *Config, c
 			c.UID = uint32(uid)
 			c.GID = uint32(uid)
 		}
-	} else if err != nil {
-		addErrorf("looking up user: %v", err)
 	} else {
 		if uid, err := strconv.ParseUint(u.Uid, 10, 32); err != nil {
 			addErrorf("parsing uid %s: %v", u.Uid, err)

mox-/forkexec_unix.go

@@ -0,0 +1,74 @@
+//go:build unix
+
+package mox
+
+import (
+	"os"
+	"os/signal"
+	"strings"
+	"syscall"
+
+	"github.com/mjl-/mox/mlog"
+)
+
+// Fork and exec as unprivileged user.
+//
+// We don't use just setuid because it is hard to guarantee that no other
+// privileged go worker processes have been started before we get here. E.g. init
+// functions in packages can start goroutines.
+func ForkExecUnprivileged() {
+	prog, err := os.Executable()
+	if err != nil {
+		xlog.Fatalx("finding executable for exec", err)
+	}
+
+	files := []*os.File{os.Stdin, os.Stdout, os.Stderr}
+	var addrs []string
+	for addr, f := range passedListeners {
+		files = append(files, f)
+		addrs = append(addrs, addr)
+	}
+	var paths []string
+	for path, fl := range passedFiles {
+		for _, f := range fl {
+			files = append(files, f)
+			paths = append(paths, path)
+		}
+	}
+	env := os.Environ()
+	env = append(env, "MOX_SOCKETS="+strings.Join(addrs, ","), "MOX_FILES="+strings.Join(paths, ","))
+
+	p, err := os.StartProcess(prog, os.Args, &os.ProcAttr{
+		Env:   env,
+		Files: files,
+		Sys: &syscall.SysProcAttr{
+			Credential: &syscall.Credential{
+				Uid: Conf.Static.UID,
+				Gid: Conf.Static.GID,
+			},
+		},
+	})
+	if err != nil {
+		xlog.Fatalx("fork and exec", err)
+	}
+	CleanupPassedFiles()
+
+	// If we get a interrupt/terminate signal, pass it on to the child. For interrupt,
+	// the child probably already got it.
+	// todo: see if we tie up child and root process so a kill -9 of the root process
+	// kills the child process too.
+	sigc := make(chan os.Signal, 1)
+	signal.Notify(sigc, os.Interrupt, syscall.SIGTERM)
+	go func() {
+		sig := <-sigc
+		p.Signal(sig)
+	}()
+
+	st, err := p.Wait()
+	if err != nil {
+		xlog.Fatalx("wait", err)
+	}
+	code := st.ExitCode()
+	xlog.Print("stopping after child exit", mlog.Field("exitcode", code))
+	os.Exit(code)
+}

mox-/forkexec_windows.go

@@ -0,0 +1,9 @@
+package mox
+
+// Fork and exec as unprivileged user.
+//
+// Not implemented yet on windows. Would need to understand its security model
+// first.
+func ForkExecUnprivileged() {
+	xlog.Fatal("fork and exec to unprivileged user not yet implemented on windows")
+}

mox-/lifecycle.go

@@ -5,18 +5,14 @@ import (
 	"fmt"
 	"net"
 	"os"
-	"os/signal"
 	"runtime"
 	"runtime/debug"
 	"strings"
 	"sync"
-	"syscall"
 	"time"
 
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promauto"
-
-	"github.com/mjl-/mox/mlog"
 )
 
 // We start up as root, bind to sockets, open private key/cert files and fork and
@@ -56,68 +52,6 @@ func RestorePassedFiles() {
 	}
 }
 
-// Fork and exec as unprivileged user.
-//
-// We don't use just setuid because it is hard to guarantee that no other
-// privileged go worker processes have been started before we get here. E.g. init
-// functions in packages can start goroutines.
-func ForkExecUnprivileged() {
-	prog, err := os.Executable()
-	if err != nil {
-		xlog.Fatalx("finding executable for exec", err)
-	}
-
-	files := []*os.File{os.Stdin, os.Stdout, os.Stderr}
-	var addrs []string
-	for addr, f := range passedListeners {
-		files = append(files, f)
-		addrs = append(addrs, addr)
-	}
-	var paths []string
-	for path, fl := range passedFiles {
-		for _, f := range fl {
-			files = append(files, f)
-			paths = append(paths, path)
-		}
-	}
-	env := os.Environ()
-	env = append(env, "MOX_SOCKETS="+strings.Join(addrs, ","), "MOX_FILES="+strings.Join(paths, ","))
-
-	p, err := os.StartProcess(prog, os.Args, &os.ProcAttr{
-		Env:   env,
-		Files: files,
-		Sys: &syscall.SysProcAttr{
-			Credential: &syscall.Credential{
-				Uid: Conf.Static.UID,
-				Gid: Conf.Static.GID,
-			},
-		},
-	})
-	if err != nil {
-		xlog.Fatalx("fork and exec", err)
-	}
-	CleanupPassedFiles()
-
-	// If we get a interrupt/terminate signal, pass it on to the child. For interrupt,
-	// the child probably already got it.
-	// todo: see if we tie up child and root process so a kill -9 of the root process
-	// kills the child process too.
-	sigc := make(chan os.Signal, 1)
-	signal.Notify(sigc, os.Interrupt, syscall.SIGTERM)
-	go func() {
-		sig := <-sigc
-		p.Signal(sig)
-	}()
-
-	st, err := p.Wait()
-	if err != nil {
-		xlog.Fatalx("wait", err)
-	}
-	code := st.ExitCode()
-	xlog.Print("stopping after child exit", mlog.Field("exitcode", code))
-	os.Exit(code)
-}
-
 // CleanupPassedFiles closes the listening socket file descriptors and files passed
 // in by the parent process. To be called by the unprivileged child after listeners
 // have been recreated (they dup the file descriptor), and by the privileged
@@ -164,6 +98,9 @@ func Listen(network, addr string) (net.Listener, error) {
 	if err != nil {
 		return nil, err
 	}
+	// On windows, we cannot duplicate a socket. We don't need to for mox localserve
+	// with FilesImmediate.
+	if !FilesImmediate {
 		tcpln, ok := ln.(*net.TCPListener)
 		if !ok {
 			return nil, fmt.Errorf("listener not a tcp listener, but %T, for network %s, address %s", ln, network, addr)
@@ -173,6 +110,7 @@ func Listen(network, addr string) (net.Listener, error) {
 			return nil, fmt.Errorf("dup listener: %v", err)
 		}
 		passedListeners[addr] = f
+	}
 	return ln, err
 }
 

moxio/linkcopy.go

@@ -45,10 +45,10 @@ func LinkOrCopy(log *mlog.Log, dst, src string, srcReaderOpt io.Reader, sync boo
 	}
 	defer func() {
 		if df != nil {
-			err = os.Remove(dst)
+			err := df.Close()
-			log.Check(err, "removing partial destination file")
-			err = df.Close()
 			log.Check(err, "closing partial destination file")
+			err = os.Remove(dst)
+			log.Check(err, "removing partial destination file", mlog.Field("path", dst))
 		}
 	}()
 
@@ -64,7 +64,7 @@ func LinkOrCopy(log *mlog.Log, dst, src string, srcReaderOpt io.Reader, sync boo
 	df = nil
 	if err != nil {
 		err := os.Remove(dst)
-		log.Check(err, "removing partial destination file")
+		log.Check(err, "removing partial destination file", mlog.Field("path", dst))
 		return err
 	}
 	return nil

moxio/linkcopy_test.go

@@ -26,6 +26,7 @@ func TestLinkOrCopy(t *testing.T) {
 	f, err := os.Create(src)
 	tcheckf(t, err, "creating test file")
 	defer os.Remove(src)
+	defer f.Close()
 	err = LinkOrCopy(log, "linkorcopytest-dst.txt", src, nil, false)
 	tcheckf(t, err, "linking file")
 	err = os.Remove("linkorcopytest-dst.txt")

moxio/syncdir.go

@@ -1,3 +1,5 @@
+//go:build !windows
+
 package moxio
 
 import (

moxio/syncdir_windows.go

@@ -0,0 +1,8 @@
+package moxio
+
+// SyncDir opens a directory and syncs its contents to disk.
+// SyncDir is a no-op on Windows.
+func SyncDir(dir string) error {
+	// todo: how to sync a directory on windows?
+	return nil
+}

mtastsdb/db_test.go

@@ -24,7 +24,7 @@ func tcheckf(t *testing.T, err error, format string, args ...any) {
 
 func TestDB(t *testing.T) {
 	mox.Shutdown = ctxbg
-	mox.ConfigStaticPath = "../testdata/mtasts/fake.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/mtasts/fake.conf")
 	mox.Conf.Static.DataDir = "."
 
 	dbpath := mox.DataDirPath("mtasts.db")

mtastsdb/refresh_test.go

@@ -30,7 +30,7 @@ var ctxbg = context.Background()
 
 func TestRefresh(t *testing.T) {
 	mox.Shutdown = ctxbg
-	mox.ConfigStaticPath = "../testdata/mtasts/fake.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/mtasts/fake.conf")
 	mox.Conf.Static.DataDir = "."
 
 	dbpath := mox.DataDirPath("mtasts.db")

queue/dsn.go

@@ -154,12 +154,11 @@ func queueDSN(log *mlog.Log, m Msg, remoteMTA dsn.NameIP, secodeOpt, errmsg stri
 		return
 	}
 	defer func() {
-		if msgFile != nil {
+		name := msgFile.Name()
-			err := os.Remove(msgFile.Name())
+		err := msgFile.Close()
-			log.Check(err, "removing message file", mlog.Field("path", msgFile.Name()))
-			err = msgFile.Close()
 		log.Check(err, "closing message file")
-		}
+		err = os.Remove(name)
+		log.Check(err, "removing message file", mlog.Field("path", name))
 	}()
 
 	msgWriter := message.NewWriter(msgFile)
@@ -174,12 +173,9 @@ func queueDSN(log *mlog.Log, m Msg, remoteMTA dsn.NameIP, secodeOpt, errmsg stri
 		MsgPrefix: []byte{},
 	}
 	acc.WithWLock(func() {
-		if err := acc.DeliverMailbox(log, mailbox, msg, msgFile, true); err != nil {
+		if err := acc.DeliverMailbox(log, mailbox, msg, msgFile); err != nil {
 			qlog("delivering dsn to mailbox", err)
 			return
 		}
 	})
-	err = msgFile.Close()
-	log.Check(err, "closing dsn file")
-	msgFile = nil
 }

queue/queue.go

@@ -118,7 +118,7 @@ func (m Msg) MessagePath() string {
 
 // Init opens the queue database without starting delivery.
 func Init() error {
-	qpath := mox.DataDirPath("queue/index.db")
+	qpath := mox.DataDirPath(filepath.FromSlash("queue/index.db"))
 	os.MkdirAll(filepath.Dir(qpath), 0770)
 	isNew := false
 	if _, err := os.Stat(qpath); err != nil && os.IsNotExist(err) {
@@ -176,14 +176,11 @@ func Count(ctx context.Context) (int, error) {
 // Add a new message to the queue. The queue is kicked immediately to start a
 // first delivery attempt.
 //
-// If consumeFile is true, it is removed as part of delivery (by rename or copy
-// and remove). msgFile is never closed by Add.
-//
 // dnsutf8Opt is a utf8-version of the message, to be used only for DNSs. If set,
 // this data is used as the message when delivering the DSN and the remote SMTP
 // server supports SMTPUTF8. If the remote SMTP server does not support SMTPUTF8,
 // the regular non-utf8 message is delivered.
-func Add(ctx context.Context, log *mlog.Log, senderAccount string, mailFrom, rcptTo smtp.Path, has8bit, smtputf8 bool, size int64, messageID string, msgPrefix []byte, msgFile *os.File, dsnutf8Opt []byte, consumeFile bool) (int64, error) {
+func Add(ctx context.Context, log *mlog.Log, senderAccount string, mailFrom, rcptTo smtp.Path, has8bit, smtputf8 bool, size int64, messageID string, msgPrefix []byte, msgFile *os.File, dsnutf8Opt []byte) (int64, error) {
 	// todo: Add should accept multiple rcptTo if they are for the same domain. so we can queue them for delivery in one (or just a few) session(s), transferring the data only once. ../rfc/5321:3759
 
 	if Localserve {
@@ -202,7 +199,7 @@ func Add(ctx context.Context, log *mlog.Log, senderAccount string, mailFrom, rcp
 		conf, _ := acc.Conf()
 		dest := conf.Destinations[mailFrom.String()]
 		acc.WithWLock(func() {
-			err = acc.DeliverDestination(log, dest, &m, msgFile, consumeFile)
+			err = acc.DeliverDestination(log, dest, &m, msgFile)
 		})
 		if err != nil {
 			return 0, fmt.Errorf("delivering message: %v", err)
@@ -239,12 +236,7 @@ func Add(ctx context.Context, log *mlog.Log, senderAccount string, mailFrom, rcp
 	}()
 	dstDir := filepath.Dir(dst)
 	os.MkdirAll(dstDir, 0770)
-	if consumeFile {
+	if err := moxio.LinkOrCopy(log, dst, msgFile.Name(), nil, true); err != nil {
-		if err := os.Rename(msgFile.Name(), dst); err != nil {
-			// Could be due to cross-filesystem rename. Users shouldn't configure their systems that way.
-			return 0, fmt.Errorf("move message into queue dir: %w", err)
-		}
-	} else if err := moxio.LinkOrCopy(log, dst, msgFile.Name(), nil, true); err != nil {
 		return 0, fmt.Errorf("linking/copying message to new file: %s", err)
 	} else if err := moxio.SyncDir(dstDir); err != nil {
 		return 0, fmt.Errorf("sync directory: %v", err)

queue/queue_test.go

@@ -13,6 +13,7 @@ import (
 	"math/big"
 	"net"
 	"os"
+	"path/filepath"
 	"strings"
 	"testing"
 	"time"
@@ -40,7 +41,7 @@ func setup(t *testing.T) (*store.Account, func()) {
 	// Prepare config so email can be delivered to mjl@mox.example.
 	os.RemoveAll("../testdata/queue/data")
 	mox.Context = ctxbg
-	mox.ConfigStaticPath = "../testdata/queue/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/queue/mox.conf")
 	mox.MustLoadConfig(true, false)
 	acc, err := store.OpenAccount("mjl")
 	tcheck(t, err, "open account")
@@ -86,13 +87,15 @@ func TestQueue(t *testing.T) {
 	}
 
 	path := smtp.Path{Localpart: "mjl", IPDomain: dns.IPDomain{Domain: dns.Domain{ASCII: "mox.example"}}}
-	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, prepareFile(t), nil, true)
+	mf := prepareFile(t)
+	defer os.Remove(mf.Name())
+	defer mf.Close()
+
+	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 
-	mf2 := prepareFile(t)
+	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, mf, nil)
-	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, mf2, nil, false)
 	tcheck(t, err, "add message to queue for delivery")
-	os.Remove(mf2.Name())
 
 	msgs, err = List(ctxbg)
 	tcheck(t, err, "listing queue")
@@ -385,7 +388,7 @@ func TestQueue(t *testing.T) {
 
 	// Add a message to be delivered with submit because of its route.
 	topath := smtp.Path{Localpart: "mjl", IPDomain: dns.IPDomain{Domain: dns.Domain{ASCII: "submit.example"}}}
-	_, err = Add(ctxbg, xlog, "mjl", path, topath, false, false, int64(len(testmsg)), "<test@localhost>", nil, prepareFile(t), nil, true)
+	_, err = Add(ctxbg, xlog, "mjl", path, topath, false, false, int64(len(testmsg)), "<test@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	wasNetDialer = testDeliver(fakeSubmitServer)
 	if !wasNetDialer {
@@ -393,7 +396,7 @@ func TestQueue(t *testing.T) {
 	}
 
 	// Add a message to be delivered with submit because of explicitly configured transport, that uses TLS.
-	msgID, err := Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, prepareFile(t), nil, true)
+	msgID, err := Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	transportSubmitTLS := "submittls"
 	n, err = Kick(ctxbg, msgID, "", "", &transportSubmitTLS)
@@ -417,7 +420,7 @@ func TestQueue(t *testing.T) {
 	}
 
 	// Add a message to be delivered with socks.
-	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<socks@localhost>", nil, prepareFile(t), nil, true)
+	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<socks@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	transportSocks := "socks"
 	n, err = Kick(ctxbg, msgID, "", "", &transportSocks)
@@ -431,7 +434,7 @@ func TestQueue(t *testing.T) {
 	}
 
 	// Add message to be delivered with opportunistic TLS verification.
-	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<opportunistictls@localhost>", nil, prepareFile(t), nil, true)
+	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<opportunistictls@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	n, err = Kick(ctxbg, msgID, "", "", nil)
 	tcheck(t, err, "kick queue")
@@ -441,7 +444,7 @@ func TestQueue(t *testing.T) {
 	testDeliver(fakeSMTPSTARTTLSServer)
 
 	// Test fallback to plain text with TLS handshake fails.
-	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<badtls@localhost>", nil, prepareFile(t), nil, true)
+	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<badtls@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	n, err = Kick(ctxbg, msgID, "", "", nil)
 	tcheck(t, err, "kick queue")
@@ -457,7 +460,7 @@ func TestQueue(t *testing.T) {
 			{Usage: adns.TLSAUsageDANEEE, Selector: adns.TLSASelectorSPKI, MatchType: adns.TLSAMatchTypeFull, CertAssoc: moxCert.Leaf.RawSubjectPublicKeyInfo},
 		},
 	}
-	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<dane@localhost>", nil, prepareFile(t), nil, true)
+	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<dane@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	n, err = Kick(ctxbg, msgID, "", "", nil)
 	tcheck(t, err, "kick queue")
@@ -472,7 +475,7 @@ func TestQueue(t *testing.T) {
 			{},
 		},
 	}
-	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<daneunusable@localhost>", nil, prepareFile(t), nil, true)
+	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<daneunusable@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	n, err = Kick(ctxbg, msgID, "", "", nil)
 	tcheck(t, err, "kick queue")
@@ -489,7 +492,7 @@ func TestQueue(t *testing.T) {
 			{Usage: adns.TLSAUsageDANEEE, Selector: adns.TLSASelectorSPKI, MatchType: adns.TLSAMatchTypeFull, CertAssoc: make([]byte, sha256.Size)},
 		},
 	}
-	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<daneinsecure@localhost>", nil, prepareFile(t), nil, true)
+	msgID, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<daneinsecure@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	n, err = Kick(ctxbg, msgID, "", "", nil)
 	tcheck(t, err, "kick queue")
@@ -504,7 +507,7 @@ func TestQueue(t *testing.T) {
 	resolver.TLSA = nil
 
 	// Add another message that we'll fail to deliver entirely.
-	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, prepareFile(t), nil, true)
+	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 
 	msgs, err = List(ctxbg)
@@ -660,7 +663,10 @@ func TestQueueStart(t *testing.T) {
 	}
 
 	path := smtp.Path{Localpart: "mjl", IPDomain: dns.IPDomain{Domain: dns.Domain{ASCII: "mox.example"}}}
-	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, prepareFile(t), nil, true)
+	mf := prepareFile(t)
+	defer os.Remove(mf.Name())
+	defer mf.Close()
+	_, err = Add(ctxbg, xlog, "mjl", path, path, false, false, int64(len(testmsg)), "<test@localhost>", nil, mf, nil)
 	tcheck(t, err, "add message to queue for delivery")
 	checkDialed(true)
 

quickstart.go

@@ -592,7 +592,7 @@ many authentication failures).
 
 	dc := config.Dynamic{}
 	sc := config.Static{
-		DataDir:           "../data",
+		DataDir:           filepath.FromSlash("../data"),
 		User:              user,
 		LogLevel:          "debug", // Help new users, they'll bring it back to info when it all works.
 		Hostname:          dnshostname.Name(),
@@ -625,9 +625,9 @@ many authentication failures).
 	public.IMAPS.Enabled = true
 
 	if existingWebserver {
-		hostbase := fmt.Sprintf("path/to/%s", dnshostname.Name())
+		hostbase := filepath.FromSlash("path/to/" + dnshostname.Name())
-		mtastsbase := fmt.Sprintf("path/to/mta-sts.%s", domain.Name())
+		mtastsbase := filepath.FromSlash("path/to/mta-sts." + domain.Name())
-		autoconfigbase := fmt.Sprintf("path/to/autoconfig.%s", domain.Name())
+		autoconfigbase := filepath.FromSlash("path/to/autoconfig." + domain.Name())
 		public.TLS = &config.TLS{
 			KeyCerts: []config.KeyCert{
 				{CertFile: hostbase + "-chain.crt.pem", KeyFile: hostbase + ".key.pem"},
@@ -657,8 +657,8 @@ and check the admin page for the needed DNS records.`)
 		}
 		now := time.Now()
 		timestamp := now.Format("20060102T150405")
-		hostRSAPrivateKeyFile := fmt.Sprintf("hostkeys/%s.%s.%s.privatekey.pkcs8.pem", dnshostname.Name(), timestamp, "rsa2048")
+		hostRSAPrivateKeyFile := filepath.Join("hostkeys", fmt.Sprintf("%s.%s.%s.privatekey.pkcs8.pem", dnshostname.Name(), timestamp, "rsa2048"))
-		hostECDSAPrivateKeyFile := fmt.Sprintf("hostkeys/%s.%s.%s.privatekey.pkcs8.pem", dnshostname.Name(), timestamp, "ecdsap256")
+		hostECDSAPrivateKeyFile := filepath.Join("hostkeys", fmt.Sprintf("%s.%s.%s.privatekey.pkcs8.pem", dnshostname.Name(), timestamp, "ecdsap256"))
 		xwritehostkeyfile := func(path string, key crypto.Signer) {
 			buf, err := x509.MarshalPKCS8PrivateKey(key)
 			if err != nil {
@@ -727,8 +727,8 @@ and check the admin page for the needed DNS records.`)
 	sc.Postmaster.Account = accountName
 	sc.Postmaster.Mailbox = "Postmaster"
 
-	mox.ConfigStaticPath = "config/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("config/mox.conf")
-	mox.ConfigDynamicPath = "config/domains.conf"
+	mox.ConfigDynamicPath = filepath.FromSlash("config/domains.conf")
 
 	mox.Conf.DynamicLastCheck = time.Now() // Prevent error logging by Make calls below.
 
@@ -759,7 +759,7 @@ and check the admin page for the needed DNS records.`)
 	for _, bl := range public.SMTP.DNSBLs {
 		confstr = strings.ReplaceAll(confstr, "- "+bl+"\n", "#- "+bl+"\n")
 	}
-	xwritefile("config/mox.conf", []byte(confstr), 0660)
+	xwritefile(filepath.FromSlash("config/mox.conf"), []byte(confstr), 0660)
 
 	// Generate domains config, and add a commented out example for delivery to a mailing list.
 	var db bytes.Buffer
@@ -798,11 +798,11 @@ and check the admin page for the needed DNS records.`)
 		ndests += "#\t\t" + line + "\n"
 	}
 	dconfstr := strings.ReplaceAll(db.String(), odests, ndests)
-	xwritefile("config/domains.conf", []byte(dconfstr), 0660)
+	xwritefile(filepath.FromSlash("config/domains.conf"), []byte(dconfstr), 0660)
 
 	// Verify config.
 	loadTLSKeyCerts := !existingWebserver
-	mc, errs := mox.ParseConfig(context.Background(), "config/mox.conf", true, loadTLSKeyCerts, false)
+	mc, errs := mox.ParseConfig(context.Background(), filepath.FromSlash("config/mox.conf"), true, loadTLSKeyCerts, false)
 	if len(errs) > 0 {
 		if len(errs) > 1 {
 			log.Printf("checking generated config, multiple errors:")

sendmail.go

@@ -220,13 +220,7 @@ binary should be setgid that group:
 		os.Mkdir(maildir, 0700)
 		f, err := os.CreateTemp(maildir, "newmsg.")
 		xcheckf(err, "creating temp file for storing message after failed delivery")
-		defer func() {
+		// note: not removing the partial file if writing/closing below fails.
-			if f != nil {
-				if err := os.Remove(f.Name()); err != nil {
-					log.Printf("removing temp file after failure storing failed delivery: %v", err)
-				}
-			}
-		}()
 		_, err = f.Write([]byte(msg))
 		xcheckf(err, "writing message to temp file after failed delivery")
 		name := f.Name()

serve.go

@@ -1,393 +1,23 @@
 package main
 
 import (
-	"context"
-	cryptorand "crypto/rand"
 	"fmt"
-	"io/fs"
-	"net"
 	"os"
-	"os/signal"
-	"path/filepath"
-	"runtime"
-	"runtime/debug"
-	"strings"
-	"sync"
-	"syscall"
 	"time"
 
-	"github.com/prometheus/client_golang/prometheus"
-	"github.com/prometheus/client_golang/prometheus/promauto"
-
 	"github.com/mjl-/mox/dmarcdb"
 	"github.com/mjl-/mox/dns"
-	"github.com/mjl-/mox/dnsbl"
 	"github.com/mjl-/mox/http"
 	"github.com/mjl-/mox/imapserver"
-	"github.com/mjl-/mox/message"
-	"github.com/mjl-/mox/metrics"
 	"github.com/mjl-/mox/mlog"
 	"github.com/mjl-/mox/mox-"
-	"github.com/mjl-/mox/moxvar"
 	"github.com/mjl-/mox/mtastsdb"
 	"github.com/mjl-/mox/queue"
 	"github.com/mjl-/mox/smtpserver"
 	"github.com/mjl-/mox/store"
 	"github.com/mjl-/mox/tlsrptdb"
-	"github.com/mjl-/mox/updates"
 )
 
-func monitorDNSBL(log *mlog.Log) {
-	defer func() {
-		// On error, don't bring down the entire server.
-		x := recover()
-		if x != nil {
-			log.Error("monitordnsbl panic", mlog.Field("panic", x))
-			debug.PrintStack()
-			metrics.PanicInc(metrics.Serve)
-		}
-	}()
-
-	l, ok := mox.Conf.Static.Listeners["public"]
-	if !ok {
-		log.Info("no listener named public, not monitoring our ips at dnsbls")
-		return
-	}
-
-	var zones []dns.Domain
-	for _, zone := range l.SMTP.DNSBLs {
-		d, err := dns.ParseDomain(zone)
-		if err != nil {
-			log.Fatalx("parsing dnsbls zone", err, mlog.Field("zone", zone))
-		}
-		zones = append(zones, d)
-	}
-	if len(zones) == 0 {
-		return
-	}
-
-	type key struct {
-		zone dns.Domain
-		ip   string
-	}
-	metrics := map[key]prometheus.GaugeFunc{}
-	var statusMutex sync.Mutex
-	statuses := map[key]bool{}
-
-	resolver := dns.StrictResolver{Pkg: "dnsblmonitor"}
-	var sleep time.Duration // No sleep on first iteration.
-	for {
-		time.Sleep(sleep)
-		sleep = 3 * time.Hour
-
-		ips, err := mox.IPs(mox.Context, false)
-		if err != nil {
-			log.Errorx("listing ips for dnsbl monitor", err)
-			continue
-		}
-		for _, ip := range ips {
-			if ip.IsLoopback() || ip.IsPrivate() {
-				continue
-			}
-
-			for _, zone := range zones {
-				status, expl, err := dnsbl.Lookup(mox.Context, resolver, zone, ip)
-				if err != nil {
-					log.Errorx("dnsbl monitor lookup", err, mlog.Field("ip", ip), mlog.Field("zone", zone), mlog.Field("expl", expl), mlog.Field("status", status))
-				}
-				k := key{zone, ip.String()}
-
-				statusMutex.Lock()
-				statuses[k] = status == dnsbl.StatusPass
-				statusMutex.Unlock()
-
-				if _, ok := metrics[k]; !ok {
-					metrics[k] = promauto.NewGaugeFunc(
-						prometheus.GaugeOpts{
-							Name: "mox_dnsbl_ips_success",
-							Help: "DNSBL lookups to configured DNSBLs of our IPs.",
-							ConstLabels: prometheus.Labels{
-								"zone": zone.LogString(),
-								"ip":   k.ip,
-							},
-						},
-						func() float64 {
-							statusMutex.Lock()
-							defer statusMutex.Unlock()
-							if statuses[k] {
-								return 1
-							}
-							return 0
-						},
-					)
-				}
-				time.Sleep(time.Second)
-			}
-		}
-	}
-}
-
-// also see localserve.go, code is similar or even shared.
-func cmdServe(c *cmd) {
-	c.help = `Start mox, serving SMTP/IMAP/HTTPS.
-
-Incoming email is accepted over SMTP. Email can be retrieved by users using
-IMAP. HTTP listeners are started for the admin/account web interfaces, and for
-automated TLS configuration. Missing essential TLS certificates are immediately
-requested, other TLS certificates are requested on demand.
-`
-	args := c.Parse()
-	if len(args) != 0 {
-		c.Usage()
-	}
-
-	// Set debug logging until config is fully loaded.
-	mlog.Logfmt = true
-	mox.Conf.Log[""] = mlog.LevelDebug
-	mlog.SetConfig(mox.Conf.Log)
-
-	checkACMEHosts := os.Getuid() != 0
-
-	log := mlog.New("serve")
-
-	if os.Getuid() == 0 {
-		mox.MustLoadConfig(true, checkACMEHosts)
-
-		// No need to potentially start and keep multiple processes. As root, we just need
-		// to start the child process.
-		runtime.GOMAXPROCS(1)
-
-		moxconf, err := filepath.Abs(mox.ConfigStaticPath)
-		log.Check(err, "finding absolute mox.conf path")
-		domainsconf, err := filepath.Abs(mox.ConfigDynamicPath)
-		log.Check(err, "finding absolute domains.conf path")
-
-		log.Print("starting as root, initializing network listeners", mlog.Field("version", moxvar.Version), mlog.Field("pid", os.Getpid()), mlog.Field("moxconf", moxconf), mlog.Field("domainsconf", domainsconf))
-		if os.Getenv("MOX_SOCKETS") != "" {
-			log.Fatal("refusing to start as root with $MOX_SOCKETS set")
-		}
-		if os.Getenv("MOX_FILES") != "" {
-			log.Fatal("refusing to start as root with $MOX_FILES set")
-		}
-
-		if !mox.Conf.Static.NoFixPermissions {
-			// Fix permissions now that we have privilege to do so. Useful for update of v0.0.1
-			// that was running directly as mox-user.
-			workdir, err := os.Getwd()
-			if err != nil {
-				log.Printx("get working dir, continuing without potentially fixing up permissions", err)
-			} else {
-				configdir := filepath.Dir(mox.ConfigStaticPath)
-				datadir := mox.DataDirPath(".")
-				err := fixperms(log, workdir, configdir, datadir, mox.Conf.Static.UID, mox.Conf.Static.GID)
-				if err != nil {
-					log.Fatalx("fixing permissions", err)
-				}
-			}
-		}
-	} else {
-		mox.RestorePassedFiles()
-		mox.MustLoadConfig(true, checkACMEHosts)
-		log.Print("starting as unprivileged user", mlog.Field("user", mox.Conf.Static.User), mlog.Field("uid", mox.Conf.Static.UID), mlog.Field("gid", mox.Conf.Static.GID), mlog.Field("pid", os.Getpid()))
-	}
-
-	syscall.Umask(syscall.Umask(007) | 007)
-
-	// Initialize key and random buffer for creating opaque SMTP
-	// transaction IDs based on "cid"s.
-	recvidpath := mox.DataDirPath("receivedid.key")
-	recvidbuf, err := os.ReadFile(recvidpath)
-	if err != nil || len(recvidbuf) != 16+8 {
-		recvidbuf = make([]byte, 16+8)
-		if _, err := cryptorand.Read(recvidbuf); err != nil {
-			log.Fatalx("reading random recvid data", err)
-		}
-		if err := os.WriteFile(recvidpath, recvidbuf, 0660); err != nil {
-			log.Fatalx("writing recvidpath", err, mlog.Field("path", recvidpath))
-		}
-		err := os.Chown(recvidpath, int(mox.Conf.Static.UID), 0)
-		log.Check(err, "chown receveidid.key", mlog.Field("path", recvidpath), mlog.Field("uid", mox.Conf.Static.UID), mlog.Field("gid", 0))
-		err = os.Chmod(recvidpath, 0640)
-		log.Check(err, "chmod receveidid.key to 0640", mlog.Field("path", recvidpath))
-	}
-	if err := mox.ReceivedIDInit(recvidbuf[:16], recvidbuf[16:]); err != nil {
-		log.Fatalx("init receivedid", err)
-	}
-
-	// Start mox. If running as root, this will bind/listen on network sockets, and
-	// fork and exec itself as unprivileged user, then waits for the child to stop and
-	// exit. When running as root, this function never returns. But the new
-	// unprivileged user will get here again, with network sockets prepared.
-	//
-	// We listen to the unix domain ctl socket afterwards, which we always remove
-	// before listening. We need to do that because we may not have cleaned up our
-	// control socket during unexpected shutdown. We don't want to remove and listen on
-	// the unix domain socket first. If we would, we would make the existing instance
-	// unreachable over its ctl socket, and then fail because the network addresses are
-	// taken.
-	const mtastsdbRefresher = true
-	const skipForkExec = false
-	if err := start(mtastsdbRefresher, skipForkExec); err != nil {
-		log.Fatalx("start", err)
-	}
-	log.Print("ready to serve")
-
-	if mox.Conf.Static.CheckUpdates {
-		checkUpdates := func() time.Duration {
-			next := 24 * time.Hour
-			current, lastknown, mtime, err := mox.LastKnown()
-			if err != nil {
-				log.Infox("determining own version before checking for updates, trying again in 24h", err)
-				return next
-			}
-
-			// We don't want to check for updates at every startup. So we sleep based on file
-			// mtime. But file won't exist initially.
-			if !mtime.IsZero() && time.Since(mtime) < 24*time.Hour {
-				d := 24*time.Hour - time.Since(mtime)
-				log.Debug("sleeping for next check for updates", mlog.Field("sleep", d))
-				time.Sleep(d)
-				next = 0
-			}
-			now := time.Now()
-			if err := os.Chtimes(mox.DataDirPath("lastknownversion"), now, now); err != nil {
-				if !os.IsNotExist(err) {
-					log.Infox("setting mtime on lastknownversion file, continuing", err)
-				}
-			}
-
-			log.Debug("checking for updates", mlog.Field("lastknown", lastknown))
-			updatesctx, updatescancel := context.WithTimeout(mox.Context, time.Minute)
-			latest, _, changelog, err := updates.Check(updatesctx, dns.StrictResolver{}, dns.Domain{ASCII: changelogDomain}, lastknown, changelogURL, changelogPubKey)
-			updatescancel()
-			if err != nil {
-				log.Infox("checking for updates", err, mlog.Field("latest", latest))
-				return next
-			}
-			if !latest.After(lastknown) {
-				log.Debug("no new version available")
-				return next
-			}
-			if len(changelog.Changes) == 0 {
-				log.Info("new version available, but changelog is empty, ignoring", mlog.Field("latest", latest))
-				return next
-			}
-
-			var cl string
-			for _, c := range changelog.Changes {
-				cl += "----\n\n" + strings.TrimSpace(c.Text) + "\n\n"
-			}
-			cl += "----"
-
-			a, err := store.OpenAccount(mox.Conf.Static.Postmaster.Account)
-			if err != nil {
-				log.Infox("open account for postmaster changelog delivery", err)
-				return next
-			}
-			defer func() {
-				err := a.Close()
-				log.Check(err, "closing account")
-			}()
-			f, err := store.CreateMessageTemp("changelog")
-			if err != nil {
-				log.Infox("making temporary message file for changelog delivery", err)
-				return next
-			}
-			defer func() {
-				if f != nil {
-					err := os.Remove(f.Name())
-					log.Check(err, "removing temp changelog file")
-					err = f.Close()
-					log.Check(err, "closing temp changelog file")
-				}
-			}()
-			m := &store.Message{
-				Received: time.Now(),
-				Flags:    store.Flags{Flagged: true},
-			}
-			n, err := fmt.Fprintf(f, "Date: %s\r\nSubject: mox %s available\r\nContent-Type: text/plain; charset=utf-8\r\nContent-Transfer-Encoding: 8-bit\r\n\r\nHi!\r\n\r\nVersion %s of mox is available, this install is at %s.\r\n\r\nChanges:\r\n\r\n%s\r\n\r\nRemember to make a backup with \"mox backup\" before upgrading.\r\nPlease report any issues at https://github.com/mjl-/mox, thanks!\r\n\r\nCheers,\r\nmox\r\n", time.Now().Format(message.RFC5322Z), latest, latest, current, strings.ReplaceAll(cl, "\n", "\r\n"))
-			if err != nil {
-				log.Infox("writing temporary message file for changelog delivery", err)
-				return next
-			}
-			m.Size = int64(n)
-			if err := a.DeliverMailbox(log, mox.Conf.Static.Postmaster.Mailbox, m, f, true); err != nil {
-				log.Errorx("changelog delivery", err)
-				return next
-			}
-			f = nil
-			log.Info("delivered changelog", mlog.Field("current", current), mlog.Field("lastknown", lastknown), mlog.Field("latest", latest))
-			if err := mox.StoreLastKnown(latest); err != nil {
-				// This will be awkward, we'll keep notifying the postmaster once every 24h...
-				log.Infox("updating last known version", err)
-			}
-			return next
-		}
-
-		go func() {
-			for {
-				next := checkUpdates()
-				time.Sleep(next)
-			}
-		}()
-	}
-
-	go monitorDNSBL(log)
-
-	ctlpath := mox.DataDirPath("ctl")
-	_ = os.Remove(ctlpath)
-	ctl, err := net.Listen("unix", ctlpath)
-	if err != nil {
-		log.Fatalx("listen on ctl unix domain socket", err)
-	}
-	go func() {
-		for {
-			conn, err := ctl.Accept()
-			if err != nil {
-				log.Printx("accept for ctl", err)
-				continue
-			}
-			cid := mox.Cid()
-			ctx := context.WithValue(mox.Context, mlog.CidKey, cid)
-			go servectl(ctx, log.WithCid(cid), conn, func() { shutdown(log) })
-		}
-	}()
-
-	// Remove old temporary files that somehow haven't been cleaned up.
-	tmpdir := mox.DataDirPath("tmp")
-	os.MkdirAll(tmpdir, 0770)
-	tmps, err := os.ReadDir(tmpdir)
-	if err != nil {
-		log.Errorx("listing files in tmpdir", err)
-	} else {
-		now := time.Now()
-		for _, e := range tmps {
-			if fi, err := e.Info(); err != nil {
-				log.Errorx("stat tmp file", err, mlog.Field("filename", e.Name()))
-			} else if now.Sub(fi.ModTime()) > 7*24*time.Hour && !fi.IsDir() {
-				p := filepath.Join(tmpdir, e.Name())
-				if err := os.Remove(p); err != nil {
-					log.Errorx("removing stale temporary file", err, mlog.Field("path", p))
-				} else {
-					log.Info("removed stale temporary file", mlog.Field("path", p))
-				}
-			}
-		}
-	}
-
-	// Graceful shutdown.
-	sigc := make(chan os.Signal, 1)
-	signal.Notify(sigc, os.Interrupt, syscall.SIGTERM)
-	sig := <-sigc
-	log.Print("shutting down, waiting max 3s for existing connections", mlog.Field("signal", sig))
-	shutdown(log)
-	if num, ok := sig.(syscall.Signal); ok {
-		os.Exit(int(num))
-	} else {
-		os.Exit(1)
-	}
-}
-
 func shutdown(log *mlog.Log) {
 	// We indicate we are shutting down. Causes new connections and new SMTP commands
 	// to be rejected. Should stop active connections pretty quickly.
@@ -420,168 +50,6 @@ func shutdown(log *mlog.Log) {
 	log.Check(err, "removing ctl unix domain socket during shutdown")
 }
 
-// Set correct permissions for mox working directory, binary, config and data and service file.
-//
-// We require being able to stat the basic non-optional paths. Then we'll try to
-// fix up permissions. If an error occurs when fixing permissions, we log and
-// continue (could not be an actual problem).
-func fixperms(log *mlog.Log, workdir, configdir, datadir string, moxuid, moxgid uint32) (rerr error) {
-	type fserr struct{ Err error }
-	defer func() {
-		x := recover()
-		if x == nil {
-			return
-		}
-		e, ok := x.(fserr)
-		if ok {
-			rerr = e.Err
-		} else {
-			panic(x)
-		}
-	}()
-
-	checkf := func(err error, format string, args ...any) {
-		if err != nil {
-			panic(fserr{fmt.Errorf(format, args...)})
-		}
-	}
-
-	// Changes we have to make. We collect them first, then apply.
-	type change struct {
-		path           string
-		uid, gid       *uint32
-		olduid, oldgid uint32
-		mode           *fs.FileMode
-		oldmode        fs.FileMode
-	}
-	var changes []change
-
-	ensure := func(p string, uid, gid uint32, perm fs.FileMode) bool {
-		fi, err := os.Stat(p)
-		checkf(err, "stat %s", p)
-
-		st, ok := fi.Sys().(*syscall.Stat_t)
-		if !ok {
-			checkf(fmt.Errorf("got %T", st), "stat sys, expected syscall.Stat_t")
-		}
-
-		var ch change
-		if st.Uid != uid || st.Gid != gid {
-			ch.uid = &uid
-			ch.gid = &gid
-			ch.olduid = st.Uid
-			ch.oldgid = st.Gid
-		}
-		if perm != fi.Mode()&(fs.ModeSetgid|0777) {
-			ch.mode = &perm
-			ch.oldmode = fi.Mode() & (fs.ModeSetgid | 0777)
-		}
-		var zerochange change
-		if ch == zerochange {
-			return false
-		}
-		ch.path = p
-		changes = append(changes, ch)
-		return true
-	}
-
-	xexists := func(p string) bool {
-		_, err := os.Stat(p)
-		if err != nil && !os.IsNotExist(err) {
-			checkf(err, "stat %s", p)
-		}
-		return err == nil
-	}
-
-	// We ensure these permissions:
-	//
-	//	$workdir root:mox 0751
-	//	$configdir mox:root 0750 + setgid, and recursively (but files 0640)
-	//	$datadir mox:root 0750 + setgid, and recursively (but files 0640)
-	//	$workdir/mox (binary, optional) root:mox 0750
-	//	$workdir/mox.service (systemd service file, optional) root:root 0644
-
-	const root = 0
-	ensure(workdir, root, moxgid, 0751)
-	fixconfig := ensure(configdir, moxuid, 0, fs.ModeSetgid|0750)
-	fixdata := ensure(datadir, moxuid, 0, fs.ModeSetgid|0750)
-
-	// Binary and systemd service file do not exist (there) when running under docker.
-	binary := filepath.Join(workdir, "mox")
-	if xexists(binary) {
-		ensure(binary, root, moxgid, 0750)
-	}
-	svc := filepath.Join(workdir, "mox.service")
-	if xexists(svc) {
-		ensure(svc, root, root, 0644)
-	}
-
-	if len(changes) == 0 {
-		return
-	}
-
-	// Apply changes.
-	log.Print("fixing up permissions, will continue on errors")
-	for _, ch := range changes {
-		if ch.uid != nil {
-			err := os.Chown(ch.path, int(*ch.uid), int(*ch.gid))
-			log.Printx("chown, fixing uid/gid", err, mlog.Field("path", ch.path), mlog.Field("olduid", ch.olduid), mlog.Field("oldgid", ch.oldgid), mlog.Field("newuid", *ch.uid), mlog.Field("newgid", *ch.gid))
-		}
-		if ch.mode != nil {
-			err := os.Chmod(ch.path, *ch.mode)
-			log.Printx("chmod, fixing permissions", err, mlog.Field("path", ch.path), mlog.Field("oldmode", fmt.Sprintf("%03o", ch.oldmode)), mlog.Field("newmode", fmt.Sprintf("%03o", *ch.mode)))
-		}
-	}
-
-	walkchange := func(dir string) {
-		err := filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error {
-			if err != nil {
-				log.Printx("walk error, continuing", err, mlog.Field("path", path))
-				return nil
-			}
-			fi, err := d.Info()
-			if err != nil {
-				log.Printx("stat during walk, continuing", err, mlog.Field("path", path))
-				return nil
-			}
-			st, ok := fi.Sys().(*syscall.Stat_t)
-			if !ok {
-				log.Printx("syscall stat during walk, continuing", err, mlog.Field("path", path))
-				return nil
-			}
-			if st.Uid != moxuid || st.Gid != root {
-				err := os.Chown(path, int(moxuid), root)
-				log.Printx("walk chown, fixing uid/gid", err, mlog.Field("path", path), mlog.Field("olduid", st.Uid), mlog.Field("oldgid", st.Gid), mlog.Field("newuid", moxuid), mlog.Field("newgid", root))
-			}
-			omode := fi.Mode() & (fs.ModeSetgid | 0777)
-			var nmode fs.FileMode
-			if fi.IsDir() {
-				nmode = fs.ModeSetgid | 0750
-			} else {
-				nmode = 0640
-			}
-			if omode != nmode {
-				err := os.Chmod(path, nmode)
-				log.Printx("walk chmod, fixing permissions", err, mlog.Field("path", path), mlog.Field("oldmode", fmt.Sprintf("%03o", omode)), mlog.Field("newmode", fmt.Sprintf("%03o", nmode)))
-			}
-			return nil
-		})
-		log.Check(err, "walking dir to fix permissions", mlog.Field("dir", dir))
-	}
-
-	// If config or data dir needed fixing, also set uid/gid and mode and files/dirs
-	// inside, recursively. We don't always recurse, data probably contains many files.
-	if fixconfig {
-		log.Print("fixing permissions in config dir", mlog.Field("configdir", configdir))
-		walkchange(configdir)
-	}
-	if fixdata {
-		log.Print("fixing permissions in data dir", mlog.Field("configdir", configdir))
-		walkchange(datadir)
-	}
-	return nil
-}
-
 // start initializes all packages, starts all listeners and the switchboard
 // goroutine, then returns.
 func start(mtastsdbRefresher, skipForkExec bool) error {

serve_unix.go

@@ -0,0 +1,546 @@
+//go:build !windows
+
+package main
+
+import (
+	"context"
+	cryptorand "crypto/rand"
+	"fmt"
+	"io/fs"
+	"net"
+	"os"
+	"os/signal"
+	"path/filepath"
+	"runtime"
+	"runtime/debug"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promauto"
+
+	"github.com/mjl-/mox/dns"
+	"github.com/mjl-/mox/dnsbl"
+	"github.com/mjl-/mox/message"
+	"github.com/mjl-/mox/metrics"
+	"github.com/mjl-/mox/mlog"
+	"github.com/mjl-/mox/mox-"
+	"github.com/mjl-/mox/moxvar"
+	"github.com/mjl-/mox/store"
+	"github.com/mjl-/mox/updates"
+)
+
+func monitorDNSBL(log *mlog.Log) {
+	defer func() {
+		// On error, don't bring down the entire server.
+		x := recover()
+		if x != nil {
+			log.Error("monitordnsbl panic", mlog.Field("panic", x))
+			debug.PrintStack()
+			metrics.PanicInc(metrics.Serve)
+		}
+	}()
+
+	l, ok := mox.Conf.Static.Listeners["public"]
+	if !ok {
+		log.Info("no listener named public, not monitoring our ips at dnsbls")
+		return
+	}
+
+	var zones []dns.Domain
+	for _, zone := range l.SMTP.DNSBLs {
+		d, err := dns.ParseDomain(zone)
+		if err != nil {
+			log.Fatalx("parsing dnsbls zone", err, mlog.Field("zone", zone))
+		}
+		zones = append(zones, d)
+	}
+	if len(zones) == 0 {
+		return
+	}
+
+	type key struct {
+		zone dns.Domain
+		ip   string
+	}
+	metrics := map[key]prometheus.GaugeFunc{}
+	var statusMutex sync.Mutex
+	statuses := map[key]bool{}
+
+	resolver := dns.StrictResolver{Pkg: "dnsblmonitor"}
+	var sleep time.Duration // No sleep on first iteration.
+	for {
+		time.Sleep(sleep)
+		sleep = 3 * time.Hour
+
+		ips, err := mox.IPs(mox.Context, false)
+		if err != nil {
+			log.Errorx("listing ips for dnsbl monitor", err)
+			continue
+		}
+		for _, ip := range ips {
+			if ip.IsLoopback() || ip.IsPrivate() {
+				continue
+			}
+
+			for _, zone := range zones {
+				status, expl, err := dnsbl.Lookup(mox.Context, resolver, zone, ip)
+				if err != nil {
+					log.Errorx("dnsbl monitor lookup", err, mlog.Field("ip", ip), mlog.Field("zone", zone), mlog.Field("expl", expl), mlog.Field("status", status))
+				}
+				k := key{zone, ip.String()}
+
+				statusMutex.Lock()
+				statuses[k] = status == dnsbl.StatusPass
+				statusMutex.Unlock()
+
+				if _, ok := metrics[k]; !ok {
+					metrics[k] = promauto.NewGaugeFunc(
+						prometheus.GaugeOpts{
+							Name: "mox_dnsbl_ips_success",
+							Help: "DNSBL lookups to configured DNSBLs of our IPs.",
+							ConstLabels: prometheus.Labels{
+								"zone": zone.LogString(),
+								"ip":   k.ip,
+							},
+						},
+						func() float64 {
+							statusMutex.Lock()
+							defer statusMutex.Unlock()
+							if statuses[k] {
+								return 1
+							}
+							return 0
+						},
+					)
+				}
+				time.Sleep(time.Second)
+			}
+		}
+	}
+}
+
+// also see localserve.go, code is similar or even shared.
+func cmdServe(c *cmd) {
+	c.help = `Start mox, serving SMTP/IMAP/HTTPS.
+
+Incoming email is accepted over SMTP. Email can be retrieved by users using
+IMAP. HTTP listeners are started for the admin/account web interfaces, and for
+automated TLS configuration. Missing essential TLS certificates are immediately
+requested, other TLS certificates are requested on demand.
+
+Only implemented on unix systems, not Windows.
+`
+	args := c.Parse()
+	if len(args) != 0 {
+		c.Usage()
+	}
+
+	// Set debug logging until config is fully loaded.
+	mlog.Logfmt = true
+	mox.Conf.Log[""] = mlog.LevelDebug
+	mlog.SetConfig(mox.Conf.Log)
+
+	checkACMEHosts := os.Getuid() != 0
+
+	log := mlog.New("serve")
+
+	if os.Getuid() == 0 {
+		mox.MustLoadConfig(true, checkACMEHosts)
+
+		// No need to potentially start and keep multiple processes. As root, we just need
+		// to start the child process.
+		runtime.GOMAXPROCS(1)
+
+		moxconf, err := filepath.Abs(mox.ConfigStaticPath)
+		log.Check(err, "finding absolute mox.conf path")
+		domainsconf, err := filepath.Abs(mox.ConfigDynamicPath)
+		log.Check(err, "finding absolute domains.conf path")
+
+		log.Print("starting as root, initializing network listeners", mlog.Field("version", moxvar.Version), mlog.Field("pid", os.Getpid()), mlog.Field("moxconf", moxconf), mlog.Field("domainsconf", domainsconf))
+		if os.Getenv("MOX_SOCKETS") != "" {
+			log.Fatal("refusing to start as root with $MOX_SOCKETS set")
+		}
+		if os.Getenv("MOX_FILES") != "" {
+			log.Fatal("refusing to start as root with $MOX_FILES set")
+		}
+
+		if !mox.Conf.Static.NoFixPermissions {
+			// Fix permissions now that we have privilege to do so. Useful for update of v0.0.1
+			// that was running directly as mox-user.
+			workdir, err := os.Getwd()
+			if err != nil {
+				log.Printx("get working dir, continuing without potentially fixing up permissions", err)
+			} else {
+				configdir := filepath.Dir(mox.ConfigStaticPath)
+				datadir := mox.DataDirPath(".")
+				err := fixperms(log, workdir, configdir, datadir, mox.Conf.Static.UID, mox.Conf.Static.GID)
+				if err != nil {
+					log.Fatalx("fixing permissions", err)
+				}
+			}
+		}
+	} else {
+		mox.RestorePassedFiles()
+		mox.MustLoadConfig(true, checkACMEHosts)
+		log.Print("starting as unprivileged user", mlog.Field("user", mox.Conf.Static.User), mlog.Field("uid", mox.Conf.Static.UID), mlog.Field("gid", mox.Conf.Static.GID), mlog.Field("pid", os.Getpid()))
+	}
+
+	syscall.Umask(syscall.Umask(007) | 007)
+
+	// Initialize key and random buffer for creating opaque SMTP
+	// transaction IDs based on "cid"s.
+	recvidpath := mox.DataDirPath("receivedid.key")
+	recvidbuf, err := os.ReadFile(recvidpath)
+	if err != nil || len(recvidbuf) != 16+8 {
+		recvidbuf = make([]byte, 16+8)
+		if _, err := cryptorand.Read(recvidbuf); err != nil {
+			log.Fatalx("reading random recvid data", err)
+		}
+		if err := os.WriteFile(recvidpath, recvidbuf, 0660); err != nil {
+			log.Fatalx("writing recvidpath", err, mlog.Field("path", recvidpath))
+		}
+		err := os.Chown(recvidpath, int(mox.Conf.Static.UID), 0)
+		log.Check(err, "chown receveidid.key", mlog.Field("path", recvidpath), mlog.Field("uid", mox.Conf.Static.UID), mlog.Field("gid", 0))
+		err = os.Chmod(recvidpath, 0640)
+		log.Check(err, "chmod receveidid.key to 0640", mlog.Field("path", recvidpath))
+	}
+	if err := mox.ReceivedIDInit(recvidbuf[:16], recvidbuf[16:]); err != nil {
+		log.Fatalx("init receivedid", err)
+	}
+
+	// Start mox. If running as root, this will bind/listen on network sockets, and
+	// fork and exec itself as unprivileged user, then waits for the child to stop and
+	// exit. When running as root, this function never returns. But the new
+	// unprivileged user will get here again, with network sockets prepared.
+	//
+	// We listen to the unix domain ctl socket afterwards, which we always remove
+	// before listening. We need to do that because we may not have cleaned up our
+	// control socket during unexpected shutdown. We don't want to remove and listen on
+	// the unix domain socket first. If we would, we would make the existing instance
+	// unreachable over its ctl socket, and then fail because the network addresses are
+	// taken.
+	const mtastsdbRefresher = true
+	const skipForkExec = false
+	if err := start(mtastsdbRefresher, skipForkExec); err != nil {
+		log.Fatalx("start", err)
+	}
+	log.Print("ready to serve")
+
+	if mox.Conf.Static.CheckUpdates {
+		checkUpdates := func() time.Duration {
+			next := 24 * time.Hour
+			current, lastknown, mtime, err := mox.LastKnown()
+			if err != nil {
+				log.Infox("determining own version before checking for updates, trying again in 24h", err)
+				return next
+			}
+
+			// We don't want to check for updates at every startup. So we sleep based on file
+			// mtime. But file won't exist initially.
+			if !mtime.IsZero() && time.Since(mtime) < 24*time.Hour {
+				d := 24*time.Hour - time.Since(mtime)
+				log.Debug("sleeping for next check for updates", mlog.Field("sleep", d))
+				time.Sleep(d)
+				next = 0
+			}
+			now := time.Now()
+			if err := os.Chtimes(mox.DataDirPath("lastknownversion"), now, now); err != nil {
+				if !os.IsNotExist(err) {
+					log.Infox("setting mtime on lastknownversion file, continuing", err)
+				}
+			}
+
+			log.Debug("checking for updates", mlog.Field("lastknown", lastknown))
+			updatesctx, updatescancel := context.WithTimeout(mox.Context, time.Minute)
+			latest, _, changelog, err := updates.Check(updatesctx, dns.StrictResolver{}, dns.Domain{ASCII: changelogDomain}, lastknown, changelogURL, changelogPubKey)
+			updatescancel()
+			if err != nil {
+				log.Infox("checking for updates", err, mlog.Field("latest", latest))
+				return next
+			}
+			if !latest.After(lastknown) {
+				log.Debug("no new version available")
+				return next
+			}
+			if len(changelog.Changes) == 0 {
+				log.Info("new version available, but changelog is empty, ignoring", mlog.Field("latest", latest))
+				return next
+			}
+
+			var cl string
+			for _, c := range changelog.Changes {
+				cl += "----\n\n" + strings.TrimSpace(c.Text) + "\n\n"
+			}
+			cl += "----"
+
+			a, err := store.OpenAccount(mox.Conf.Static.Postmaster.Account)
+			if err != nil {
+				log.Infox("open account for postmaster changelog delivery", err)
+				return next
+			}
+			defer func() {
+				err := a.Close()
+				log.Check(err, "closing account")
+			}()
+			f, err := store.CreateMessageTemp("changelog")
+			if err != nil {
+				log.Infox("making temporary message file for changelog delivery", err)
+				return next
+			}
+			defer func() {
+				name := f.Name()
+				err = f.Close()
+				log.Check(err, "closing temp changelog file")
+				err := os.Remove(name)
+				log.Check(err, "removing temp changelog file", mlog.Field("path", name))
+			}()
+			m := &store.Message{
+				Received: time.Now(),
+				Flags:    store.Flags{Flagged: true},
+			}
+			n, err := fmt.Fprintf(f, "Date: %s\r\nSubject: mox %s available\r\nContent-Type: text/plain; charset=utf-8\r\nContent-Transfer-Encoding: 8-bit\r\n\r\nHi!\r\n\r\nVersion %s of mox is available, this install is at %s.\r\n\r\nChanges:\r\n\r\n%s\r\n\r\nRemember to make a backup with \"mox backup\" before upgrading.\r\nPlease report any issues at https://github.com/mjl-/mox, thanks!\r\n\r\nCheers,\r\nmox\r\n", time.Now().Format(message.RFC5322Z), latest, latest, current, strings.ReplaceAll(cl, "\n", "\r\n"))
+			if err != nil {
+				log.Infox("writing temporary message file for changelog delivery", err)
+				return next
+			}
+			m.Size = int64(n)
+			if err := a.DeliverMailbox(log, mox.Conf.Static.Postmaster.Mailbox, m, f); err != nil {
+				log.Errorx("changelog delivery", err)
+				return next
+			}
+			log.Info("delivered changelog", mlog.Field("current", current), mlog.Field("lastknown", lastknown), mlog.Field("latest", latest))
+			if err := mox.StoreLastKnown(latest); err != nil {
+				// This will be awkward, we'll keep notifying the postmaster once every 24h...
+				log.Infox("updating last known version", err)
+			}
+			return next
+		}
+
+		go func() {
+			for {
+				next := checkUpdates()
+				time.Sleep(next)
+			}
+		}()
+	}
+
+	go monitorDNSBL(log)
+
+	ctlpath := mox.DataDirPath("ctl")
+	_ = os.Remove(ctlpath)
+	ctl, err := net.Listen("unix", ctlpath)
+	if err != nil {
+		log.Fatalx("listen on ctl unix domain socket", err)
+	}
+	go func() {
+		for {
+			conn, err := ctl.Accept()
+			if err != nil {
+				log.Printx("accept for ctl", err)
+				continue
+			}
+			cid := mox.Cid()
+			ctx := context.WithValue(mox.Context, mlog.CidKey, cid)
+			go servectl(ctx, log.WithCid(cid), conn, func() { shutdown(log) })
+		}
+	}()
+
+	// Remove old temporary files that somehow haven't been cleaned up.
+	tmpdir := mox.DataDirPath("tmp")
+	os.MkdirAll(tmpdir, 0770)
+	tmps, err := os.ReadDir(tmpdir)
+	if err != nil {
+		log.Errorx("listing files in tmpdir", err)
+	} else {
+		now := time.Now()
+		for _, e := range tmps {
+			if fi, err := e.Info(); err != nil {
+				log.Errorx("stat tmp file", err, mlog.Field("filename", e.Name()))
+			} else if now.Sub(fi.ModTime()) > 7*24*time.Hour && !fi.IsDir() {
+				p := filepath.Join(tmpdir, e.Name())
+				if err := os.Remove(p); err != nil {
+					log.Errorx("removing stale temporary file", err, mlog.Field("path", p))
+				} else {
+					log.Info("removed stale temporary file", mlog.Field("path", p))
+				}
+			}
+		}
+	}
+
+	// Graceful shutdown.
+	sigc := make(chan os.Signal, 1)
+	signal.Notify(sigc, os.Interrupt, syscall.SIGTERM)
+	sig := <-sigc
+	log.Print("shutting down, waiting max 3s for existing connections", mlog.Field("signal", sig))
+	shutdown(log)
+	if num, ok := sig.(syscall.Signal); ok {
+		os.Exit(int(num))
+	} else {
+		os.Exit(1)
+	}
+}
+
+// Set correct permissions for mox working directory, binary, config and data and service file.
+//
+// We require being able to stat the basic non-optional paths. Then we'll try to
+// fix up permissions. If an error occurs when fixing permissions, we log and
+// continue (could not be an actual problem).
+func fixperms(log *mlog.Log, workdir, configdir, datadir string, moxuid, moxgid uint32) (rerr error) {
+	type fserr struct{ Err error }
+	defer func() {
+		x := recover()
+		if x == nil {
+			return
+		}
+		e, ok := x.(fserr)
+		if ok {
+			rerr = e.Err
+		} else {
+			panic(x)
+		}
+	}()
+
+	checkf := func(err error, format string, args ...any) {
+		if err != nil {
+			panic(fserr{fmt.Errorf(format, args...)})
+		}
+	}
+
+	// Changes we have to make. We collect them first, then apply.
+	type change struct {
+		path           string
+		uid, gid       *uint32
+		olduid, oldgid uint32
+		mode           *fs.FileMode
+		oldmode        fs.FileMode
+	}
+	var changes []change
+
+	ensure := func(p string, uid, gid uint32, perm fs.FileMode) bool {
+		fi, err := os.Stat(p)
+		checkf(err, "stat %s", p)
+
+		st, ok := fi.Sys().(*syscall.Stat_t)
+		if !ok {
+			checkf(fmt.Errorf("got %T", st), "stat sys, expected syscall.Stat_t")
+		}
+
+		var ch change
+		if st.Uid != uid || st.Gid != gid {
+			ch.uid = &uid
+			ch.gid = &gid
+			ch.olduid = st.Uid
+			ch.oldgid = st.Gid
+		}
+		if perm != fi.Mode()&(fs.ModeSetgid|0777) {
+			ch.mode = &perm
+			ch.oldmode = fi.Mode() & (fs.ModeSetgid | 0777)
+		}
+		var zerochange change
+		if ch == zerochange {
+			return false
+		}
+		ch.path = p
+		changes = append(changes, ch)
+		return true
+	}
+
+	xexists := func(p string) bool {
+		_, err := os.Stat(p)
+		if err != nil && !os.IsNotExist(err) {
+			checkf(err, "stat %s", p)
+		}
+		return err == nil
+	}
+
+	// We ensure these permissions:
+	//
+	//	$workdir root:mox 0751
+	//	$configdir mox:root 0750 + setgid, and recursively (but files 0640)
+	//	$datadir mox:root 0750 + setgid, and recursively (but files 0640)
+	//	$workdir/mox (binary, optional) root:mox 0750
+	//	$workdir/mox.service (systemd service file, optional) root:root 0644
+
+	const root = 0
+	ensure(workdir, root, moxgid, 0751)
+	fixconfig := ensure(configdir, moxuid, 0, fs.ModeSetgid|0750)
+	fixdata := ensure(datadir, moxuid, 0, fs.ModeSetgid|0750)
+
+	// Binary and systemd service file do not exist (there) when running under docker.
+	binary := filepath.Join(workdir, "mox")
+	if xexists(binary) {
+		ensure(binary, root, moxgid, 0750)
+	}
+	svc := filepath.Join(workdir, "mox.service")
+	if xexists(svc) {
+		ensure(svc, root, root, 0644)
+	}
+
+	if len(changes) == 0 {
+		return
+	}
+
+	// Apply changes.
+	log.Print("fixing up permissions, will continue on errors")
+	for _, ch := range changes {
+		if ch.uid != nil {
+			err := os.Chown(ch.path, int(*ch.uid), int(*ch.gid))
+			log.Printx("chown, fixing uid/gid", err, mlog.Field("path", ch.path), mlog.Field("olduid", ch.olduid), mlog.Field("oldgid", ch.oldgid), mlog.Field("newuid", *ch.uid), mlog.Field("newgid", *ch.gid))
+		}
+		if ch.mode != nil {
+			err := os.Chmod(ch.path, *ch.mode)
+			log.Printx("chmod, fixing permissions", err, mlog.Field("path", ch.path), mlog.Field("oldmode", fmt.Sprintf("%03o", ch.oldmode)), mlog.Field("newmode", fmt.Sprintf("%03o", *ch.mode)))
+		}
+	}
+
+	walkchange := func(dir string) {
+		err := filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error {
+			if err != nil {
+				log.Printx("walk error, continuing", err, mlog.Field("path", path))
+				return nil
+			}
+			fi, err := d.Info()
+			if err != nil {
+				log.Printx("stat during walk, continuing", err, mlog.Field("path", path))
+				return nil
+			}
+			st, ok := fi.Sys().(*syscall.Stat_t)
+			if !ok {
+				log.Printx("syscall stat during walk, continuing", err, mlog.Field("path", path))
+				return nil
+			}
+			if st.Uid != moxuid || st.Gid != root {
+				err := os.Chown(path, int(moxuid), root)
+				log.Printx("walk chown, fixing uid/gid", err, mlog.Field("path", path), mlog.Field("olduid", st.Uid), mlog.Field("oldgid", st.Gid), mlog.Field("newuid", moxuid), mlog.Field("newgid", root))
+			}
+			omode := fi.Mode() & (fs.ModeSetgid | 0777)
+			var nmode fs.FileMode
+			if fi.IsDir() {
+				nmode = fs.ModeSetgid | 0750
+			} else {
+				nmode = 0640
+			}
+			if omode != nmode {
+				err := os.Chmod(path, nmode)
+				log.Printx("walk chmod, fixing permissions", err, mlog.Field("path", path), mlog.Field("oldmode", fmt.Sprintf("%03o", omode)), mlog.Field("newmode", fmt.Sprintf("%03o", nmode)))
+			}
+			return nil
+		})
+		log.Check(err, "walking dir to fix permissions", mlog.Field("dir", dir))
+	}
+
+	// If config or data dir needed fixing, also set uid/gid and mode and files/dirs
+	// inside, recursively. We don't always recurse, data probably contains many files.
+	if fixconfig {
+		log.Print("fixing permissions in config dir", mlog.Field("configdir", configdir))
+		walkchange(configdir)
+	}
+	if fixdata {
+		log.Print("fixing permissions in data dir", mlog.Field("configdir", configdir))
+		walkchange(datadir)
+	}
+	return nil
+}

serve_windows.go

@@ -0,0 +1,16 @@
+package main
+
+import (
+	"log"
+)
+
+// also see localserve.go, code is similar or even shared.
+func cmdServe(c *cmd) {
+	c.help = `Start mox, serving SMTP/IMAP/HTTPS. Not implemented on windows.
+`
+	args := c.Parse()
+	if len(args) != 0 {
+		c.Usage()
+	}
+	log.Fatalln("mox serve not implemented on windows yet due to unfamiliarity with the windows security model, other commands including localserve do work")
+}

smtpserver/dsn.go

@@ -6,6 +6,7 @@ import (
 	"os"
 
 	"github.com/mjl-/mox/dsn"
+	"github.com/mjl-/mox/mlog"
 	"github.com/mjl-/mox/queue"
 	"github.com/mjl-/mox/smtp"
 	"github.com/mjl-/mox/store"
@@ -30,12 +31,11 @@ func queueDSN(ctx context.Context, c *conn, rcptTo smtp.Path, m dsn.Message) err
 		return fmt.Errorf("creating temp file: %w", err)
 	}
 	defer func() {
-		if f != nil {
+		name := f.Name()
-			err := os.Remove(f.Name())
-			c.log.Check(err, "removing temporary dsn message file")
 		err = f.Close()
 		c.log.Check(err, "closing temporary dsn message file")
-		}
+		err := os.Remove(name)
+		c.log.Check(err, "removing temporary dsn message file", mlog.Field("path", name))
 	}()
 	if _, err := f.Write([]byte(buf)); err != nil {
 		return fmt.Errorf("writing dsn file: %w", err)
@@ -46,11 +46,8 @@ func queueDSN(ctx context.Context, c *conn, rcptTo smtp.Path, m dsn.Message) err
 	// ../rfc/3464:433
 	const has8bit = false
 	const smtputf8 = false
-	if _, err := queue.Add(ctx, c.log, "", smtp.Path{}, rcptTo, has8bit, smtputf8, int64(len(buf)), m.MessageID, nil, f, bufUTF8, true); err != nil {
+	if _, err := queue.Add(ctx, c.log, "", smtp.Path{}, rcptTo, has8bit, smtputf8, int64(len(buf)), m.MessageID, nil, f, bufUTF8); err != nil {
 		return err
 	}
-	err = f.Close()
-	c.log.Check(err, "closing dsn file")
-	f = nil
 	return nil
 }

smtpserver/fuzz_test.go

@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"net"
 	"os"
+	"path/filepath"
 	"testing"
 	"time"
 
@@ -30,7 +31,7 @@ func FuzzServer(f *testing.F) {
 	f.Add("QUIT")
 
 	mox.Context = ctxbg
-	mox.ConfigStaticPath = "../testdata/smtpserverfuzz/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/smtpserverfuzz/mox.conf")
 	mox.MustLoadConfig(true, false)
 	dataDir := mox.ConfigDirPath(mox.Conf.Static.DataDir)
 	os.RemoveAll(dataDir)

smtpserver/reputation_test.go

@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"net"
 	"os"
+	"path/filepath"
 	"testing"
 	"time"
 
@@ -99,7 +100,7 @@ func TestReputation(t *testing.T) {
 	check := func(m store.Message, history []store.Message, expJunk *bool, expConclusive bool, expMethod reputationMethod) {
 		t.Helper()
 
-		p := "../testdata/smtpserver-reputation.db"
+		p := filepath.FromSlash("../testdata/smtpserver-reputation.db")
 		defer os.Remove(p)
 
 		db, err := bstore.Open(ctxbg, p, &bstore.Options{Timeout: 5 * time.Second}, store.DBTypes...)

smtpserver/server.go

@@ -1514,12 +1514,11 @@ func (c *conn) cmdData(p *parser) {
 		xsmtpServerErrorf(errCodes(smtp.C451LocalErr, smtp.SeSys3Other0, err), "creating temporary file for message: %s", err)
 	}
 	defer func() {
-		if dataFile != nil {
+		name := dataFile.Name()
-			err := os.Remove(dataFile.Name())
+		err := dataFile.Close()
-			c.log.Check(err, "removing temporary message file", mlog.Field("path", dataFile.Name()))
-			err = dataFile.Close()
 		c.log.Check(err, "removing temporary message file")
-		}
+		err = os.Remove(name)
+		c.log.Check(err, "removing temporary message file", mlog.Field("path", name))
 	}()
 	msgWriter := message.NewWriter(dataFile)
 	dr := smtp.NewDataReader(c.r)
@@ -1660,18 +1659,16 @@ func (c *conn) cmdData(p *parser) {
 	// handle it first, and leave the rest of the function for handling wild west
 	// internet traffic.
 	if c.submission {
-		c.submit(cmdctx, recvHdrFor, msgWriter, &dataFile)
+		c.submit(cmdctx, recvHdrFor, msgWriter, dataFile)
 	} else {
-		c.deliver(cmdctx, recvHdrFor, msgWriter, iprevStatus, iprevAuthentic, &dataFile)
+		c.deliver(cmdctx, recvHdrFor, msgWriter, iprevStatus, iprevAuthentic, dataFile)
 	}
 }
 
 // submit is used for mail from authenticated users that we will try to deliver.
-func (c *conn) submit(ctx context.Context, recvHdrFor func(string) string, msgWriter *message.Writer, pdataFile **os.File) {
+func (c *conn) submit(ctx context.Context, recvHdrFor func(string) string, msgWriter *message.Writer, dataFile *os.File) {
 	// Similar between ../smtpserver/server.go:/submit\( and ../webmail/webmail.go:/MessageSubmit\(
 
-	dataFile := *pdataFile
-
 	var msgPrefix []byte
 
 	// Check that user is only sending email as one of its configured identities. Not
@@ -1774,7 +1771,7 @@ func (c *conn) submit(ctx context.Context, recvHdrFor func(string) string, msgWr
 	// We always deliver through the queue. It would be more efficient to deliver
 	// directly, but we don't want to circumvent all the anti-spam measures. Accounts
 	// on a single mox instance should be allowed to block each other.
-	for i, rcptAcc := range c.recipients {
+	for _, rcptAcc := range c.recipients {
 		if Localserve {
 			code, timeout := localserveNeedsError(rcptAcc.rcptTo.Localpart)
 			if timeout {
@@ -1790,7 +1787,7 @@ func (c *conn) submit(ctx context.Context, recvHdrFor func(string) string, msgWr
 		xmsgPrefix := append([]byte(recvHdrFor(rcptAcc.rcptTo.String())), msgPrefix...)
 
 		msgSize := int64(len(xmsgPrefix)) + msgWriter.Size
-		if _, err := queue.Add(ctx, c.log, c.account.Name, *c.mailFrom, rcptAcc.rcptTo, msgWriter.Has8bit, c.smtputf8, msgSize, messageID, xmsgPrefix, dataFile, nil, i == len(c.recipients)-1); err != nil {
+		if _, err := queue.Add(ctx, c.log, c.account.Name, *c.mailFrom, rcptAcc.rcptTo, msgWriter.Has8bit, c.smtputf8, msgSize, messageID, xmsgPrefix, dataFile, nil); err != nil {
 			// Aborting the transaction is not great. But continuing and generating DSNs will
 			// probably result in errors as well...
 			metricSubmission.WithLabelValues("queueerror").Inc()
@@ -1804,10 +1801,6 @@ func (c *conn) submit(ctx context.Context, recvHdrFor func(string) string, msgWr
 		xcheckf(err, "adding outgoing message")
 	}
 
-	err = dataFile.Close()
-	c.log.Check(err, "closing file after submission")
-	*pdataFile = nil
-
 	c.transactionGood++
 	c.transactionBad-- // Compensate for early earlier pessimistic increase.
 
@@ -1866,9 +1859,7 @@ func (c *conn) xlocalserveError(lp smtp.Localpart) {
 
 // deliver is called for incoming messages from external, typically untrusted
 // sources. i.e. not submitted by authenticated users.
-func (c *conn) deliver(ctx context.Context, recvHdrFor func(string) string, msgWriter *message.Writer, iprevStatus iprev.Status, iprevAuthentic bool, pdataFile **os.File) {
+func (c *conn) deliver(ctx context.Context, recvHdrFor func(string) string, msgWriter *message.Writer, iprevStatus iprev.Status, iprevAuthentic bool, dataFile *os.File) {
-	dataFile := *pdataFile
-
 	// todo: in decision making process, if we run into (some) temporary errors, attempt to continue. if we decide to accept, all good. if we decide to reject, we'll make it a temporary reject.
 
 	msgFrom, headers, err := message.From(c.log, false, dataFile)
@@ -2381,7 +2372,7 @@ func (c *conn) deliver(ctx context.Context, recvHdrFor func(string) string, msgW
 						if err != nil {
 							log.Errorx("tidying rejects mailbox", err)
 						} else if hasSpace {
-							if err := acc.DeliverMailbox(log, conf.RejectsMailbox, m, dataFile, false); err != nil {
+							if err := acc.DeliverMailbox(log, conf.RejectsMailbox, m, dataFile); err != nil {
 								log.Errorx("delivering spammy mail to rejects mailbox", err)
 							} else {
 								log.Info("delivered spammy mail to rejects mailbox")
@@ -2456,7 +2447,7 @@ func (c *conn) deliver(ctx context.Context, recvHdrFor func(string) string, msgW
 			}
 		}
 		acc.WithWLock(func() {
-			if err := acc.DeliverMailbox(log, a.mailbox, m, dataFile, false); err != nil {
+			if err := acc.DeliverMailbox(log, a.mailbox, m, dataFile); err != nil {
 				log.Errorx("delivering", err)
 				metricDelivery.WithLabelValues("delivererror", a.reason).Inc()
 				addError(rcptAcc, smtp.C451LocalErr, smtp.SeSys3Other0, false, "error processing")
@@ -2562,12 +2553,6 @@ func (c *conn) deliver(ctx context.Context, recvHdrFor func(string) string, msgW
 		}
 	}
 
-	err = os.Remove(dataFile.Name())
-	c.log.Check(err, "removing file after delivery")
-	err = dataFile.Close()
-	c.log.Check(err, "closing data file after delivery")
-	*pdataFile = nil
-
 	c.transactionGood++
 	c.transactionBad-- // Compensate for early earlier pessimistic increase.
 	c.rset()

smtpserver/server_test.go

@@ -193,7 +193,7 @@ func fakeCert(t *testing.T) tls.Certificate {
 
 // Test submission from authenticated user.
 func TestSubmission(t *testing.T) {
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", dns.MockResolver{})
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), dns.MockResolver{})
 	defer ts.close()
 
 	// Set DKIM signing config.
@@ -254,7 +254,7 @@ func TestDelivery(t *testing.T) {
 		},
 		PTR: map[string][]string{},
 	}
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), resolver)
 	defer ts.close()
 
 	ts.run(func(err error, client *smtpclient.Client) {
@@ -333,9 +333,11 @@ func TestDelivery(t *testing.T) {
 func tinsertmsg(t *testing.T, acc *store.Account, mailbox string, m *store.Message, msg string) {
 	mf, err := store.CreateMessageTemp("queue-dsn")
 	tcheck(t, err, "temp message")
+	defer os.Remove(mf.Name())
+	defer mf.Close()
 	_, err = mf.Write([]byte(msg))
 	tcheck(t, err, "write message")
-	err = acc.DeliverMailbox(xlog, mailbox, m, mf, true)
+	err = acc.DeliverMailbox(xlog, mailbox, m, mf)
 	tcheck(t, err, "deliver message")
 	err = mf.Close()
 	tcheck(t, err, "close message")
@@ -392,7 +394,7 @@ func TestSpam(t *testing.T) {
 			"_dmarc.example.org.": {"v=DMARC1;p=reject"},
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/junk/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/junk/mox.conf"), resolver)
 	defer ts.close()
 
 	// Insert spammy messages. No junkfilter training yet.
@@ -538,7 +540,7 @@ func TestForward(t *testing.T) {
 			rcptTo = "mjl@mox.example" // Without IsForward rule.
 		}
 
-		ts := newTestServer(t, "../testdata/smtp/junk/mox.conf", resolver)
+		ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/junk/mox.conf"), resolver)
 		defer ts.close()
 
 		var msgBad = strings.ReplaceAll(`From: <remote@bad.example>
@@ -645,7 +647,7 @@ func TestDMARCSent(t *testing.T) {
 			"_dmarc.example.org.": {"v=DMARC1;p=reject"},
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/junk/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/junk/mox.conf"), resolver)
 	defer ts.close()
 
 	// Insert spammy messages not related to the test message.
@@ -710,7 +712,7 @@ func TestBlocklistedSubjectpass(t *testing.T) {
 			"127.0.0.10": {"example.org."}, // For iprev check.
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), resolver)
 	ts.dnsbls = []dns.Domain{{ASCII: "dnsbl.example"}}
 	defer ts.close()
 
@@ -776,7 +778,7 @@ func TestDMARCReport(t *testing.T) {
 			"127.0.0.10": {"example.org."}, // For iprev check.
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/dmarcreport/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/dmarcreport/mox.conf"), resolver)
 	defer ts.close()
 
 	run := func(report string, n int) {
@@ -899,7 +901,7 @@ func TestTLSReport(t *testing.T) {
 			"127.0.0.10": {"example.org."}, // For iprev check.
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/tlsrpt/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/tlsrpt/mox.conf"), resolver)
 	defer ts.close()
 
 	run := func(tlsrpt string, n int) {
@@ -939,7 +941,7 @@ func TestTLSReport(t *testing.T) {
 }
 
 func TestRatelimitConnectionrate(t *testing.T) {
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", dns.MockResolver{})
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), dns.MockResolver{})
 	defer ts.close()
 
 	// We'll be creating 300 connections, no TLS and reduce noise.
@@ -965,7 +967,7 @@ func TestRatelimitConnectionrate(t *testing.T) {
 }
 
 func TestRatelimitAuth(t *testing.T) {
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", dns.MockResolver{})
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), dns.MockResolver{})
 	defer ts.close()
 
 	ts.submission = true
@@ -1006,7 +1008,7 @@ func TestRatelimitDelivery(t *testing.T) {
 			"127.0.0.10": {"example.org."},
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), resolver)
 	defer ts.close()
 
 	orig := limitIPMasked1MessagesPerMinute
@@ -1061,7 +1063,7 @@ func TestRatelimitDelivery(t *testing.T) {
 }
 
 func TestNonSMTP(t *testing.T) {
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", dns.MockResolver{})
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), dns.MockResolver{})
 	defer ts.close()
 	ts.cid += 2
 
@@ -1105,7 +1107,7 @@ func TestNonSMTP(t *testing.T) {
 
 // Test limits on outgoing messages.
 func TestLimitOutgoing(t *testing.T) {
-	ts := newTestServer(t, "../testdata/smtp/sendlimit/mox.conf", dns.MockResolver{})
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/sendlimit/mox.conf"), dns.MockResolver{})
 	defer ts.close()
 
 	ts.user = "mjl@mox.example"
@@ -1149,7 +1151,7 @@ func TestCatchall(t *testing.T) {
 			"127.0.0.10": {"other.example."},
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/catchall/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/catchall/mox.conf"), resolver)
 	defer ts.close()
 
 	testDeliver := func(rcptTo string, expErr *smtpclient.Error) {
@@ -1195,7 +1197,7 @@ func TestDKIMSign(t *testing.T) {
 		},
 	}
 
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), resolver)
 	defer ts.close()
 
 	// Set DKIM signing config.
@@ -1287,7 +1289,7 @@ func TestPostmaster(t *testing.T) {
 			"127.0.0.10": {"other.example."},
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/postmaster/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/postmaster/mox.conf"), resolver)
 	defer ts.close()
 
 	testDeliver := func(rcptTo string, expErr *smtpclient.Error) {
@@ -1321,7 +1323,7 @@ func TestEmptylocalpart(t *testing.T) {
 			"127.0.0.10": {"other.example."},
 		},
 	}
-	ts := newTestServer(t, "../testdata/smtp/mox.conf", resolver)
+	ts := newTestServer(t, filepath.FromSlash("../testdata/smtp/mox.conf"), resolver)
 	defer ts.close()
 
 	testDeliver := func(rcptTo string, expErr *smtpclient.Error) {

store/account.go

@@ -1189,9 +1189,6 @@ func (a *Account) WithRLock(fn func()) {
 
 // DeliverMessage delivers a mail message to the account.
 //
-// If consumeFile is set, the original msgFile is moved/renamed or copied and
-// removed as part of delivery.
-//
 // The message, with msg.MsgPrefix and msgFile combined, must have a header
 // section. The caller is responsible for adding a header separator to
 // msg.MsgPrefix if missing from an incoming message.
@@ -1210,7 +1207,7 @@ func (a *Account) WithRLock(fn func()) {
 // Caller must broadcast new message.
 //
 // Caller must update mailbox counts.
-func (a *Account) DeliverMessage(log *mlog.Log, tx *bstore.Tx, m *Message, msgFile *os.File, consumeFile, sync, notrain, nothreads bool) error {
+func (a *Account) DeliverMessage(log *mlog.Log, tx *bstore.Tx, m *Message, msgFile *os.File, sync, notrain, nothreads bool) error {
 	if m.Expunged {
 		return fmt.Errorf("cannot deliver expunged message")
 	}
@@ -1346,12 +1343,7 @@ func (a *Account) DeliverMessage(log *mlog.Log, tx *bstore.Tx, m *Message, msgFi
 		}
 	}
 
-	if consumeFile {
+	if err := moxio.LinkOrCopy(log, msgPath, msgFile.Name(), &moxio.AtReader{R: msgFile}, true); err != nil {
-		if err := os.Rename(msgFile.Name(), msgPath); err != nil {
-			// Could be due to cross-filesystem rename. Users shouldn't configure their systems that way.
-			return fmt.Errorf("moving msg file to destination directory: %w", err)
-		}
-	} else if err := moxio.LinkOrCopy(log, msgPath, msgFile.Name(), &moxio.AtReader{R: msgFile}, true); err != nil {
 		return fmt.Errorf("linking/copying message to new file: %w", err)
 	}
 
@@ -1647,7 +1639,7 @@ ruleset:
 
 // MessagePath returns the file system path of a message.
 func (a *Account) MessagePath(messageID int64) string {
-	return strings.Join(append([]string{a.Dir, "msg"}, messagePathElems(messageID)...), "/")
+	return strings.Join(append([]string{a.Dir, "msg"}, messagePathElems(messageID)...), string(filepath.Separator))
 }
 
 // MessageReader opens a message for reading, transparently combining the
@@ -1661,7 +1653,7 @@ func (a *Account) MessageReader(m Message) *MsgReader {
 // Caller must hold account wlock (mailbox may be created).
 // Message delivery, possible mailbox creation, and updated mailbox counts are
 // broadcasted.
-func (a *Account) DeliverDestination(log *mlog.Log, dest config.Destination, m *Message, msgFile *os.File, consumeFile bool) error {
+func (a *Account) DeliverDestination(log *mlog.Log, dest config.Destination, m *Message, msgFile *os.File) error {
 	var mailbox string
 	rs := MessageRuleset(log, dest, m, m.MsgPrefix, msgFile)
 	if rs != nil {
@@ -1671,7 +1663,7 @@ func (a *Account) DeliverDestination(log *mlog.Log, dest config.Destination, m *
 	} else {
 		mailbox = dest.Mailbox
 	}
-	return a.DeliverMailbox(log, mailbox, m, msgFile, consumeFile)
+	return a.DeliverMailbox(log, mailbox, m, msgFile)
 }
 
 // DeliverMailbox delivers an email to the specified mailbox.
@@ -1679,7 +1671,7 @@ func (a *Account) DeliverDestination(log *mlog.Log, dest config.Destination, m *
 // Caller must hold account wlock (mailbox may be created).
 // Message delivery, possible mailbox creation, and updated mailbox counts are
 // broadcasted.
-func (a *Account) DeliverMailbox(log *mlog.Log, mailbox string, m *Message, msgFile *os.File, consumeFile bool) error {
+func (a *Account) DeliverMailbox(log *mlog.Log, mailbox string, m *Message, msgFile *os.File) error {
 	var changes []Change
 	err := a.DB.Write(context.TODO(), func(tx *bstore.Tx) error {
 		mb, chl, err := a.MailboxEnsure(tx, mailbox, true)
@@ -1696,7 +1688,7 @@ func (a *Account) DeliverMailbox(log *mlog.Log, mailbox string, m *Message, msgF
 			return fmt.Errorf("updating mailbox for delivery: %w", err)
 		}
 
-		if err := a.DeliverMessage(log, tx, m, msgFile, consumeFile, true, false, false); err != nil {
+		if err := a.DeliverMessage(log, tx, m, msgFile, true, false, false); err != nil {
 			return err
 		}
 
@@ -1977,10 +1969,11 @@ func OpenEmail(email string) (*Account, config.Destination, error) {
 // 64 characters, must be power of 2 for MessagePath
 const msgDirChars = "abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ-_"
 
-// MessagePath returns the filename of the on-disk filename, relative to the containing directory such as <account>/msg or queue.
+// MessagePath returns the filename of the on-disk filename, relative to the
+// containing directory such as <account>/msg or queue.
 // Returns names like "AB/1".
 func MessagePath(messageID int64) string {
-	return strings.Join(messagePathElems(messageID), "/")
+	return strings.Join(messagePathElems(messageID), string(filepath.Separator))
 }
 
 // messagePathElems returns the elems, for a single join without intermediate

store/account_test.go

@@ -3,6 +3,7 @@ package store
 import (
 	"context"
 	"os"
+	"path/filepath"
 	"regexp"
 	"strings"
 	"testing"
@@ -28,7 +29,7 @@ func tcheck(t *testing.T, err error, msg string) {
 
 func TestMailbox(t *testing.T) {
 	os.RemoveAll("../testdata/store/data")
-	mox.ConfigStaticPath = "../testdata/store/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/store/mox.conf")
 	mox.MustLoadConfig(true, false)
 	acc, err := OpenAccount("mjl")
 	tcheck(t, err, "open account")
@@ -44,6 +45,7 @@ func TestMailbox(t *testing.T) {
 	if err != nil {
 		t.Fatalf("creating temp msg file: %s", err)
 	}
+	defer os.Remove(msgFile.Name())
 	defer msgFile.Close()
 	msgWriter := message.NewWriter(msgFile)
 	if _, err := msgWriter.Write([]byte(" message")); err != nil {
@@ -70,7 +72,7 @@ func TestMailbox(t *testing.T) {
 	}
 	acc.WithWLock(func() {
 		conf, _ := acc.Conf()
-		err := acc.DeliverDestination(xlog, conf.Destinations["mjl"], &m, msgFile, false)
+		err := acc.DeliverDestination(xlog, conf.Destinations["mjl"], &m, msgFile)
 		tcheck(t, err, "deliver without consume")
 
 		err = acc.DB.Write(ctxbg, func(tx *bstore.Tx) error {
@@ -79,7 +81,7 @@ func TestMailbox(t *testing.T) {
 			tcheck(t, err, "sent mailbox")
 			msent.MailboxID = mbsent.ID
 			msent.MailboxOrigID = mbsent.ID
-			err = acc.DeliverMessage(xlog, tx, &msent, msgFile, false, true, false, false)
+			err = acc.DeliverMessage(xlog, tx, &msent, msgFile, true, false, false)
 			tcheck(t, err, "deliver message")
 			if !msent.ThreadMuted || !msent.ThreadCollapsed {
 				t.Fatalf("thread muted & collapsed should have been copied from parent (duplicate message-id) m")
@@ -95,7 +97,7 @@ func TestMailbox(t *testing.T) {
 			tcheck(t, err, "insert rejects mailbox")
 			mreject.MailboxID = mbrejects.ID
 			mreject.MailboxOrigID = mbrejects.ID
-			err = acc.DeliverMessage(xlog, tx, &mreject, msgFile, false, true, false, false)
+			err = acc.DeliverMessage(xlog, tx, &mreject, msgFile, true, false, false)
 			tcheck(t, err, "deliver message")
 
 			err = tx.Get(&mbrejects)
@@ -108,7 +110,7 @@ func TestMailbox(t *testing.T) {
 		})
 		tcheck(t, err, "deliver as sent and rejects")
 
-		err = acc.DeliverDestination(xlog, conf.Destinations["mjl"], &mconsumed, msgFile, true)
+		err = acc.DeliverDestination(xlog, conf.Destinations["mjl"], &mconsumed, msgFile)
 		tcheck(t, err, "deliver with consume")
 
 		err = acc.DB.Write(ctxbg, func(tx *bstore.Tx) error {
@@ -251,9 +253,11 @@ func TestMailbox(t *testing.T) {
 }
 
 func TestMessageRuleset(t *testing.T) {
-	f, err := os.Open("/dev/null")
+	f, err := CreateMessageTemp("msgruleset")
-	tcheck(t, err, "open")
+	tcheck(t, err, "creating temp msg file")
+	defer os.Remove(f.Name())
 	defer f.Close()
+
 	msgBuf := []byte(strings.ReplaceAll(`List-ID:  <test.mox.example>
 
 test

store/export.go

@@ -82,10 +82,11 @@ type DirArchiver struct {
 }
 
 // Create create name in the file system, in dir.
+// name must always use forwarded slashes.
 func (a DirArchiver) Create(name string, size int64, mtime time.Time) (io.WriteCloser, error) {
 	isdir := strings.HasSuffix(name, "/")
 	name = strings.TrimSuffix(name, "/")
-	p := filepath.Join(a.Dir, name)
+	p := filepath.Join(a.Dir, filepath.FromSlash(name))
 	os.MkdirAll(filepath.Dir(p), 0770)
 	if isdir {
 		return nil, os.Mkdir(p, 0770)
@@ -213,8 +214,11 @@ func ExportMessages(ctx context.Context, log *mlog.Log, db *bstore.DB, accountDi
 	var mboxwriter *bufio.Writer
 	defer func() {
 		if mboxtmp != nil {
+			name := mboxtmp.Name()
 			err := mboxtmp.Close()
 			log.Check(err, "closing mbox temp file")
+			err = os.Remove(name)
+			log.Check(err, "removing mbox temp file", mlog.Field("name", name))
 		}
 	}()
 
@@ -287,8 +291,11 @@ func ExportMessages(ctx context.Context, log *mlog.Log, db *bstore.DB, accountDi
 		if err := w.Close(); err != nil {
 			return fmt.Errorf("closing message file: %v", err)
 		}
+		name := mboxtmp.Name()
 		err = mboxtmp.Close()
 		log.Check(err, "closing temporary mbox file")
+		err = os.Remove(name)
+		log.Check(err, "removing temporary mbox file", mlog.Field("path", name))
 		mboxwriter = nil
 		mboxtmp = nil
 		return nil
@@ -524,10 +531,6 @@ func ExportMessages(ctx context.Context, log *mlog.Log, db *bstore.DB, accountDi
 				if err != nil {
 					return fmt.Errorf("creating temp mbox file: %v", err)
 				}
-				// Remove file immediately, so we are sure we don't leave it around.
-				if err := os.Remove(mboxtmp.Name()); err != nil {
-					return fmt.Errorf("removing temp file just created: %v", err)
-				}
 				mboxwriter = bufio.NewWriter(mboxtmp)
 			}
 		}

store/export_test.go

@@ -20,7 +20,7 @@ func TestExport(t *testing.T) {
 	// and maildir/mbox. check there are 2 files in the repo, no errors.txt.
 
 	os.RemoveAll("../testdata/store/data")
-	mox.ConfigStaticPath = "../testdata/store/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/store/mox.conf")
 	mox.MustLoadConfig(true, false)
 	acc, err := OpenAccount("mjl")
 	tcheck(t, err, "open account")
@@ -32,16 +32,17 @@ func TestExport(t *testing.T) {
 	msgFile, err := CreateMessageTemp("mox-test-export")
 	tcheck(t, err, "create temp")
 	defer os.Remove(msgFile.Name()) // To be sure.
+	defer msgFile.Close()
 	const msg = "test: test\r\n\r\ntest\r\n"
 	_, err = msgFile.Write([]byte(msg))
 	tcheck(t, err, "write message")
 
 	m := Message{Received: time.Now(), Size: int64(len(msg))}
-	err = acc.DeliverMailbox(xlog, "Inbox", &m, msgFile, false)
+	err = acc.DeliverMailbox(xlog, "Inbox", &m, msgFile)
 	tcheck(t, err, "deliver")
 
 	m = Message{Received: time.Now(), Size: int64(len(msg))}
-	err = acc.DeliverMailbox(xlog, "Trash", &m, msgFile, true)
+	err = acc.DeliverMailbox(xlog, "Trash", &m, msgFile)
 	tcheck(t, err, "deliver")
 
 	var maildirZip, maildirTar, mboxZip, mboxTar bytes.Buffer
@@ -61,8 +62,8 @@ func TestExport(t *testing.T) {
 	archive(ZipArchiver{zip.NewWriter(&mboxZip)}, false)
 	archive(TarArchiver{tar.NewWriter(&maildirTar)}, true)
 	archive(TarArchiver{tar.NewWriter(&mboxTar)}, false)
-	archive(DirArchiver{"../testdata/exportmaildir"}, true)
+	archive(DirArchiver{filepath.FromSlash("../testdata/exportmaildir")}, true)
-	archive(DirArchiver{"../testdata/exportmbox"}, false)
+	archive(DirArchiver{filepath.FromSlash("../testdata/exportmbox")}, false)
 
 	if r, err := zip.NewReader(bytes.NewReader(maildirZip.Bytes()), int64(maildirZip.Len())); err != nil {
 		t.Fatalf("reading maildir zip: %v", err)
@@ -115,6 +116,6 @@ func TestExport(t *testing.T) {
 		}
 	}
 
-	checkDirFiles("../testdata/exportmaildir", 2)
+	checkDirFiles(filepath.FromSlash("../testdata/exportmaildir"), 2)
-	checkDirFiles("../testdata/exportmbox", 2)
+	checkDirFiles(filepath.FromSlash("../testdata/exportmbox"), 2)
 }

store/import.go

@@ -84,10 +84,11 @@ func (mr *MboxReader) Next() (*Message, *os.File, string, error) {
 	}
 	defer func() {
 		if f != nil {
-			err := os.Remove(f.Name())
+			name := f.Name()
-			mr.log.Check(err, "removing temporary message file after mbox read error", mlog.Field("path", f.Name()))
+			err := f.Close()
-			err = f.Close()
 			mr.log.Check(err, "closing temporary message file after mbox read error")
+			err = os.Remove(name)
+			mr.log.Check(err, "removing temporary message file after mbox read error", mlog.Field("path", name))
 		}
 	}()
 
@@ -272,10 +273,11 @@ func (mr *MaildirReader) Next() (*Message, *os.File, string, error) {
 	}
 	defer func() {
 		if f != nil {
-			err := os.Remove(f.Name())
+			name := f.Name()
-			mr.log.Check(err, "removing temporary message file after maildir read error", mlog.Field("path", f.Name()))
+			err := f.Close()
-			err = f.Close()
 			mr.log.Check(err, "closing temporary message file after maildir read error")
+			err = os.Remove(name)
+			mr.log.Check(err, "removing temporary message file after maildir read error", mlog.Field("path", name))
 		}
 	}()
 

store/import_test.go

@@ -24,15 +24,15 @@ func TestMboxReader(t *testing.T) {
 	if err != nil {
 		t.Fatalf("next mbox message: %v", err)
 	}
-	defer mf0.Close()
 	defer os.Remove(mf0.Name())
+	defer mf0.Close()
 
 	_, mf1, _, err := mr.Next()
 	if err != nil {
 		t.Fatalf("next mbox message: %v", err)
 	}
-	defer mf1.Close()
 	defer os.Remove(mf1.Name())
+	defer mf1.Close()
 
 	_, _, _, err = mr.Next()
 	if err != io.EOF {
@@ -62,15 +62,15 @@ func TestMaildirReader(t *testing.T) {
 	if err != nil {
 		t.Fatalf("next maildir message: %v", err)
 	}
-	defer mf0.Close()
 	defer os.Remove(mf0.Name())
+	defer mf0.Close()
 
 	_, mf1, _, err := mr.Next()
 	if err != nil {
 		t.Fatalf("next maildir message: %v", err)
 	}
-	defer mf1.Close()
 	defer os.Remove(mf1.Name())
+	defer mf1.Close()
 
 	_, _, _, err = mr.Next()
 	if err != io.EOF {

store/msgreader_test.go

@@ -12,7 +12,13 @@ func TestMsgreader(t *testing.T) {
 		t.Fatalf("expected error for non-existing file, got %s", err)
 	}
 
-	if buf, err := io.ReadAll(&MsgReader{prefix: []byte("hello"), path: "/dev/null", size: int64(len("hello"))}); err != nil {
+	if err := os.WriteFile("emptyfile_test.txt", []byte{}, 0660); err != nil {
+		t.Fatalf("writing emptyfile_test.txt: %s", err)
+	}
+	defer os.Remove("emptyfile_test.txt")
+	mr := &MsgReader{prefix: []byte("hello"), path: "emptyfile_test.txt", size: int64(len("hello"))}
+	defer mr.Close()
+	if buf, err := io.ReadAll(mr); err != nil {
 		t.Fatalf("readall: %s", err)
 	} else if string(buf) != "hello" {
 		t.Fatalf("got %q, expected %q", buf, "hello")
@@ -22,7 +28,8 @@ func TestMsgreader(t *testing.T) {
 		t.Fatalf("writing msgreader_test.txt: %s", err)
 	}
 	defer os.Remove("msgreader_test.txt")
-	mr := &MsgReader{prefix: []byte("hello"), path: "msgreader_test.txt", size: int64(len("hello world"))}
+	mr = &MsgReader{prefix: []byte("hello"), path: "msgreader_test.txt", size: int64(len("hello world"))}
+	defer mr.Close()
 	if buf, err := io.ReadAll(mr); err != nil {
 		t.Fatalf("readall: %s", err)
 	} else if string(buf) != "hello world" {

store/threads_test.go

@@ -2,6 +2,7 @@ package store
 
 import (
 	"os"
+	"path/filepath"
 	"reflect"
 	"strings"
 	"testing"
@@ -15,7 +16,7 @@ import (
 
 func TestThreadingUpgrade(t *testing.T) {
 	os.RemoveAll("../testdata/store/data")
-	mox.ConfigStaticPath = "../testdata/store/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/store/mox.conf")
 	mox.MustLoadConfig(true, false)
 	acc, err := OpenAccount("mjl")
 	tcheck(t, err, "open account")
@@ -32,6 +33,7 @@ func TestThreadingUpgrade(t *testing.T) {
 		t.Helper()
 		f, err := CreateMessageTemp("account-test")
 		tcheck(t, err, "temp file")
+		defer os.Remove(f.Name())
 		defer f.Close()
 
 		s = strings.ReplaceAll(s, "\n", "\r\n")
@@ -40,7 +42,7 @@ func TestThreadingUpgrade(t *testing.T) {
 			MsgPrefix: []byte(s),
 			Received:  recv,
 		}
-		err = acc.DeliverMailbox(log, "Inbox", &m, f, true)
+		err = acc.DeliverMailbox(log, "Inbox", &m, f)
 		tcheck(t, err, "deliver")
 		if expThreadID == 0 {
 			expThreadID = m.ID

tlsrptdb/db_test.go

@@ -63,7 +63,7 @@ const reportJSON = `{
 func TestReport(t *testing.T) {
 	mox.Context = ctxbg
 	mox.Shutdown, mox.ShutdownCancel = context.WithCancel(ctxbg)
-	mox.ConfigStaticPath = "../testdata/tlsrpt/fake.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/tlsrpt/fake.conf")
 	mox.Conf.Static.DataDir = "."
 	// Recognize as configured domain.
 	mox.Conf.Dynamic.Domains = map[string]config.Domain{

vendor/github.com/mjl-/adns/Makefile

@@ -27,7 +27,8 @@ buildall:
 	GOOS=illumos GOARCH=amd64 go build
 	GOOS=solaris GOARCH=amd64 go build
 	GOOS=aix GOARCH=ppc64 go build
-	# no windows or plan9 for now
+	GOOS=windows GOARCH=amd64 go build
+	# no plan9 for now
 
 fmt:
 	gofmt -w -s *.go */*/*.go

vendor/github.com/mjl-/adns/dnsconfig_windows.go

@@ -0,0 +1,65 @@
+// Copyright 2022 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package adns
+
+import (
+	"net"
+	"syscall"
+	"time"
+
+	"golang.org/x/sys/windows"
+)
+
+func dnsReadConfig(ignoredFilename string) (conf *dnsConfig) {
+	conf = &dnsConfig{
+		ndots:    1,
+		timeout:  5 * time.Second,
+		attempts: 2,
+	}
+	defer func() {
+		if len(conf.servers) == 0 {
+			conf.servers = defaultNS
+		}
+	}()
+	aas, err := adapterAddresses()
+	if err != nil {
+		return
+	}
+	// TODO(bradfitz): this just collects all the DNS servers on all
+	// the interfaces in some random order. It should order it by
+	// default route, or only use the default route(s) instead.
+	// In practice, however, it mostly works.
+	for _, aa := range aas {
+		for dns := aa.FirstDnsServerAddress; dns != nil; dns = dns.Next {
+			// Only take interfaces whose OperStatus is IfOperStatusUp(0x01) into DNS configs.
+			if aa.OperStatus != windows.IfOperStatusUp {
+				continue
+			}
+			sa, err := dns.Address.Sockaddr.Sockaddr()
+			if err != nil {
+				continue
+			}
+			var ip net.IP
+			switch sa := sa.(type) {
+			case *syscall.SockaddrInet4:
+				ip = net.IPv4(sa.Addr[0], sa.Addr[1], sa.Addr[2], sa.Addr[3])
+			case *syscall.SockaddrInet6:
+				ip = make(net.IP, net.IPv6len)
+				copy(ip, sa.Addr[:])
+				if ip[0] == 0xfe && ip[1] == 0xc0 {
+					// Ignore these fec0/10 ones. Windows seems to
+					// populate them as defaults on its misc rando
+					// interfaces.
+					continue
+				}
+			default:
+				// Unexpected type.
+				continue
+			}
+			conf.servers = append(conf.servers, JoinHostPort(ip.String(), "53"))
+		}
+	}
+	return conf
+}

vendor/github.com/mjl-/adns/interface_windows.go

@@ -0,0 +1,43 @@
+// Copyright 2011 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package adns
+
+import (
+	"os"
+	"syscall"
+	"unsafe"
+
+	"golang.org/x/sys/windows"
+)
+
+// adapterAddresses returns a list of IP adapter and address
+// structures. The structure contains an IP adapter and flattened
+// multiple IP addresses including unicast, anycast and multicast
+// addresses.
+func adapterAddresses() ([]*windows.IpAdapterAddresses, error) {
+	var b []byte
+	l := uint32(15000) // recommended initial size
+	for {
+		b = make([]byte, l)
+		err := windows.GetAdaptersAddresses(syscall.AF_UNSPEC, windows.GAA_FLAG_INCLUDE_PREFIX, 0, (*windows.IpAdapterAddresses)(unsafe.Pointer(&b[0])), &l)
+		if err == nil {
+			if l == 0 {
+				return nil, nil
+			}
+			break
+		}
+		if err.(syscall.Errno) != syscall.ERROR_BUFFER_OVERFLOW {
+			return nil, os.NewSyscallError("getadaptersaddresses", err)
+		}
+		if l <= uint32(len(b)) {
+			return nil, os.NewSyscallError("getadaptersaddresses", err)
+		}
+	}
+	var aas []*windows.IpAdapterAddresses
+	for aa := (*windows.IpAdapterAddresses)(unsafe.Pointer(&b[0])); aa != nil; aa = aa.Next {
+		aas = append(aas, aa)
+	}
+	return aas, nil
+}

vendor/github.com/mjl-/adns/lookup_unix.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build unix || wasip1
+//go:build unix || wasip1 || windows
 
 package adns
 

vendor/github.com/mjl-/adns/port_unix.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build unix || (js && wasm) || wasip1
+//go:build unix || (js && wasm) || wasip1 || windows
 
 // Read system port mappings from /etc/services
 

vendor/modules.txt

@@ -11,7 +11,7 @@ github.com/golang/protobuf/ptypes/timestamp
 # github.com/matttproud/golang_protobuf_extensions v1.0.1
 ## explicit
 github.com/matttproud/golang_protobuf_extensions/pbutil
-# github.com/mjl-/adns v0.0.0-20231009145311-e3834995f16c
+# github.com/mjl-/adns v0.0.0-20231013194548-ea0378d616ab
 ## explicit; go 1.20
 github.com/mjl-/adns
 github.com/mjl-/adns/internal/bytealg

webaccount/account.go

@@ -299,15 +299,13 @@ func Handle(w http.ResponseWriter, r *http.Request) {
 		}
 		defer func() {
 			if tmpf != nil {
+				name := tmpf.Name()
 				err := tmpf.Close()
 				log.Check(err, "closing uploaded file")
+				err = os.Remove(name)
+				log.Check(err, "removing temporary file", mlog.Field("path", name))
 			}
 		}()
-		if err := os.Remove(tmpf.Name()); err != nil {
-			log.Errorx("removing temporary file", err)
-			http.Error(w, "500 - internal server error - "+err.Error(), http.StatusInternalServerError)
-			return
-		}
 		if _, err := io.Copy(tmpf, f); err != nil {
 			log.Errorx("copying import to temporary file", err)
 			http.Error(w, "500 - internal server error - "+err.Error(), http.StatusInternalServerError)
@@ -319,7 +317,7 @@ func Handle(w http.ResponseWriter, r *http.Request) {
 			http.Error(w, "500 - internal server error - "+err.Error(), http.StatusInternalServerError)
 			return
 		}
-		tmpf = nil // importStart is now responsible for closing.
+		tmpf = nil // importStart is now responsible for cleanup.
 
 		w.Header().Set("Content-Type", "application/json")
 		_ = json.NewEncoder(w).Encode(map[string]string{"ImportToken": token})

webaccount/account_test.go

@@ -38,7 +38,7 @@ func tcheck(t *testing.T, err error, msg string) {
 
 func TestAccount(t *testing.T) {
 	os.RemoveAll("../testdata/httpaccount/data")
-	mox.ConfigStaticPath = "../testdata/httpaccount/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/httpaccount/mox.conf")
 	mox.ConfigDynamicPath = filepath.Join(filepath.Dir(mox.ConfigStaticPath), "domains.conf")
 	mox.MustLoadConfig(true, false)
 	acc, err := store.OpenAccount("mjl")
@@ -144,8 +144,8 @@ func TestAccount(t *testing.T) {
 			t.Fatalf("imported %d messages, expected %d", count, expect)
 		}
 	}
-	testImport("../testdata/importtest.mbox.zip", 2)
+	testImport(filepath.FromSlash("../testdata/importtest.mbox.zip"), 2)
-	testImport("../testdata/importtest.maildir.tgz", 2)
+	testImport(filepath.FromSlash("../testdata/importtest.maildir.tgz"), 2)
 
 	// Check there are messages, with the right flags.
 	acc.DB.Read(ctxbg, func(tx *bstore.Tx) error {

webaccount/import.go

@@ -198,12 +198,15 @@ type importStep struct {
 }
 
 // importStart prepare the import and launches the goroutine to actually import.
-// importStart is responsible for closing f.
+// importStart is responsible for closing f and removing f.
 func importStart(log *mlog.Log, accName string, f *os.File, skipMailboxPrefix string) (string, error) {
 	defer func() {
 		if f != nil {
+			name := f.Name()
 			err := f.Close()
 			log.Check(err, "closing uploaded file")
+			err = os.Remove(name)
+			log.Check(err, "removing uploaded file", mlog.Field("name", name))
 		}
 	}()
 
@@ -270,7 +273,7 @@ func importStart(log *mlog.Log, accName string, f *os.File, skipMailboxPrefix st
 
 	log.Info("starting import")
 	go importMessages(ctx, log.WithCid(mox.Cid()), token, acc, tx, zr, tr, f, skipMailboxPrefix)
-	f = nil // importMessages is now responsible for closing.
+	f = nil // importMessages is now responsible for closing and removing.
 
 	return token, nil
 }
@@ -313,8 +316,11 @@ func importMessages(ctx context.Context, log *mlog.Log, token string, acc *store
 	}
 
 	defer func() {
+		name := f.Name()
 		err := f.Close()
 		log.Check(err, "closing uploaded messages file")
+		err = os.Remove(name)
+		log.Check(err, "removing uploaded messages file", mlog.Field("path", name))
 
 		for _, id := range deliveredIDs {
 			p := acc.MessagePath(id)
@@ -482,12 +488,11 @@ func importMessages(ctx context.Context, log *mlog.Log, token string, acc *store
 
 	xdeliver := func(mb store.Mailbox, m *store.Message, f *os.File, pos string) {
 		defer func() {
-			if f != nil {
+			name := f.Name()
-				err := os.Remove(f.Name())
-				log.Check(err, "removing temporary message file for delivery")
 			err = f.Close()
 			log.Check(err, "closing temporary message file for delivery")
-			}
+			err := os.Remove(name)
+			log.Check(err, "removing temporary message file for delivery")
 		}()
 		m.MailboxID = mb.ID
 		m.MailboxOrigID = mb.ID
@@ -542,11 +547,10 @@ func importMessages(ctx context.Context, log *mlog.Log, token string, acc *store
 			trainMessage(m, p, pos)
 		}
 
-		const consumeFile = true
 		const sync = false
 		const notrain = true
 		const nothreads = true
-		if err := acc.DeliverMessage(log, tx, m, f, consumeFile, sync, notrain, nothreads); err != nil {
+		if err := acc.DeliverMessage(log, tx, m, f, sync, notrain, nothreads); err != nil {
 			problemf("delivering message %s: %s (continuing)", pos, err)
 			return
 		}
@@ -557,7 +561,6 @@ func importMessages(ctx context.Context, log *mlog.Log, token string, acc *store
 			prevMailbox = mb.Name
 			sendEvent("count", importCount{mb.Name, messages[mb.Name]})
 		}
-		f = nil
 	}
 
 	ximportMbox := func(mailbox, filename string, r io.Reader) {
@@ -591,10 +594,11 @@ func importMessages(ctx context.Context, log *mlog.Log, token string, acc *store
 		ximportcheckf(err, "creating temp message")
 		defer func() {
 			if f != nil {
-				err := os.Remove(f.Name())
+				name := f.Name()
-				log.Check(err, "removing temporary file for delivery")
 				err = f.Close()
 				log.Check(err, "closing temporary file for delivery")
+				err := os.Remove(name)
+				log.Check(err, "removing temporary file for delivery", mlog.Field("path", name))
 			}
 		}()
 

webmail/api.go

@@ -352,12 +352,11 @@ func (w Webmail) MessageSubmit(ctx context.Context, m SubmitMessage) {
 	dataFile, err := store.CreateMessageTemp("webmail-submit")
 	xcheckf(ctx, err, "creating temporary file for message")
 	defer func() {
-		if dataFile != nil {
+		name := dataFile.Name()
 		err := dataFile.Close()
 		log.Check(err, "closing submit message file")
-			err = os.Remove(dataFile.Name())
+		err = os.Remove(name)
-			log.Check(err, "removing temporary submit message file")
+		log.Check(err, "removing temporary submit message file", mlog.Field("name", name))
-		}
 	}()
 
 	// If writing to the message file fails, we abort immediately.
@@ -669,7 +668,7 @@ func (w Webmail) MessageSubmit(ctx context.Context, m SubmitMessage) {
 			Localpart: rcpt.Localpart,
 			IPDomain:  dns.IPDomain{Domain: rcpt.Domain},
 		}
-		_, err := queue.Add(ctx, log, reqInfo.AccountName, fromPath, toPath, has8bit, smtputf8, msgSize, messageID, []byte(rcptMsgPrefix), dataFile, nil, false)
+		_, err := queue.Add(ctx, log, reqInfo.AccountName, fromPath, toPath, has8bit, smtputf8, msgSize, messageID, []byte(rcptMsgPrefix), dataFile, nil)
 		if err != nil {
 			metricSubmission.WithLabelValues("queueerror").Inc()
 		}
@@ -720,11 +719,6 @@ func (w Webmail) MessageSubmit(ctx context.Context, m SubmitMessage) {
 			sentmb, err := bstore.QueryDB[store.Mailbox](ctx, acc.DB).FilterEqual("Sent", true).Get()
 			if err == bstore.ErrAbsent {
 				// There is no mailbox designated as Sent mailbox, so we're done.
-				err := os.Remove(dataFile.Name())
-				log.Check(err, "removing submitmessage file")
-				err = dataFile.Close()
-				log.Check(err, "closing submitmessage file")
-				dataFile = nil
 				return
 			}
 			xcheckf(ctx, err, "message submitted to queue, adding to Sent mailbox")
@@ -749,7 +743,7 @@ func (w Webmail) MessageSubmit(ctx context.Context, m SubmitMessage) {
 			err = tx.Update(&sentmb)
 			xcheckf(ctx, err, "updating sent mailbox for counts")
 
-			err = acc.DeliverMessage(log, tx, &sentm, dataFile, true, true, false, false)
+			err = acc.DeliverMessage(log, tx, &sentm, dataFile, true, false, false)
 			if err != nil {
 				metricSubmission.WithLabelValues("storesenterror").Inc()
 				metricked = true
@@ -757,10 +751,6 @@ func (w Webmail) MessageSubmit(ctx context.Context, m SubmitMessage) {
 			xcheckf(ctx, err, "message submitted to queue, appending message to Sent mailbox")
 
 			changes = append(changes, sentm.ChangeAddUID(), sentmb.ChangeCounts())
-
-			err = dataFile.Close()
-			log.Check(err, "closing submit message file")
-			dataFile = nil
 		})
 
 		store.BroadcastChanges(acc, changes)

webmail/api_test.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"net/http"
 	"os"
+	"path/filepath"
 	"runtime/debug"
 	"testing"
 
@@ -43,7 +44,7 @@ func TestAPI(t *testing.T) {
 	mox.LimitersInit()
 	os.RemoveAll("../testdata/webmail/data")
 	mox.Context = ctxbg
-	mox.ConfigStaticPath = "../testdata/webmail/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/webmail/mox.conf")
 	mox.MustLoadConfig(true, false)
 	defer store.Switchboard()()
 

webmail/view_test.go

@@ -12,6 +12,7 @@ import (
 	"net/http/httptest"
 	"net/url"
 	"os"
+	"path/filepath"
 	"reflect"
 	"testing"
 	"time"
@@ -23,7 +24,7 @@ import (
 func TestView(t *testing.T) {
 	os.RemoveAll("../testdata/webmail/data")
 	mox.Context = ctxbg
-	mox.ConfigStaticPath = "../testdata/webmail/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/webmail/mox.conf")
 	mox.MustLoadConfig(true, false)
 	defer store.Switchboard()()
 

webmail/webmail.go

@@ -160,8 +160,8 @@ type merged struct {
 var webmail = &merged{
 	fallbackHTML: webmailHTML,
 	fallbackJS:   webmailJS,
-	htmlPath:     "webmail/webmail.html",
+	htmlPath:     filepath.FromSlash("webmail/webmail.html"),
-	jsPath:       "webmail/webmail.js",
+	jsPath:       filepath.FromSlash("webmail/webmail.js"),
 }
 
 // fallbackMtime returns a time to use for the Last-Modified header in case we
@@ -733,7 +733,7 @@ func handle(apiHandler http.Handler, w http.ResponseWriter, r *http.Request) {
 		h.Set("Content-Type", "text/html; charset=utf-8")
 		h.Set("Cache-Control", "no-cache, max-age=0")
 
-		path := "webmail/msg.html"
+		path := filepath.FromSlash("webmail/msg.html")
 		fallback := webmailmsgHTML
 		serveContentFallback(log, w, r, path, fallback)
 
@@ -800,7 +800,7 @@ func handle(apiHandler http.Handler, w http.ResponseWriter, r *http.Request) {
 
 		// We typically return the embedded file, but during development it's handy to load
 		// from disk.
-		path := "webmail/text.html"
+		path := filepath.FromSlash("webmail/text.html")
 		fallback := webmailtextHTML
 		serveContentFallback(log, w, r, path, fallback)
 

webmail/webmail_test.go

@@ -12,6 +12,7 @@ import (
 	"net/http/httptest"
 	"net/textproto"
 	"os"
+	"path/filepath"
 	"reflect"
 	"strings"
 	"testing"
@@ -253,10 +254,12 @@ type testmsg struct {
 func tdeliver(t *testing.T, acc *store.Account, tm *testmsg) {
 	msgFile, err := store.CreateMessageTemp("webmail-test")
 	tcheck(t, err, "create message temp")
+	defer os.Remove(msgFile.Name())
+	defer msgFile.Close()
 	size, err := msgFile.Write(tm.msg.Marshal(t))
 	tcheck(t, err, "write message temp")
 	m := store.Message{Flags: tm.Flags, Keywords: tm.Keywords, Size: int64(size)}
-	err = acc.DeliverMailbox(xlog, tm.Mailbox, &m, msgFile, true)
+	err = acc.DeliverMailbox(xlog, tm.Mailbox, &m, msgFile)
 	tcheck(t, err, "deliver test message")
 	err = msgFile.Close()
 	tcheck(t, err, "closing test message")
@@ -272,7 +275,7 @@ func TestWebmail(t *testing.T) {
 	mox.LimitersInit()
 	os.RemoveAll("../testdata/webmail/data")
 	mox.Context = ctxbg
-	mox.ConfigStaticPath = "../testdata/webmail/mox.conf"
+	mox.ConfigStaticPath = filepath.FromSlash("../testdata/webmail/mox.conf")
 	mox.MustLoadConfig(true, false)
 	defer store.Switchboard()()