improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config

- make builtin http handlers serve on specific domains, such as for mta-sts, so
  e.g. /.well-known/mta-sts.txt isn't served on all domains.
- add logging of a few more fields in access logging.
- small tweaks/bug fixes in webserver request handling.
- add config option for redirecting entire domains to another (common enough).
- split httpserver metric into two: one for duration until writing header (i.e.
  performance of server), another for duration until full response is sent to
  client (i.e. performance as perceived by users).
- add admin ui, a new page for managing the configs. after making changes
  and hitting "save", the changes take effect immediately. the page itself
  doesn't look very well-designed (many input fields, makes it look messy). i
  have an idea to improve it (explained in admin.html as todo) by making the
  layout look just like the config file. not urgent though.

i've already changed my websites/webapps over.

the idea of adding a webserver is to take away a (the) reason for folks to want
to complicate their mox setup by running an other webserver on the same machine.
i think the current webserver implementation can already serve most common use
cases. with a few more tweaks (feedback needed!) we should be able to get to 95%
of the use cases. the reverse proxy can take care of the remaining 5%.
nevertheless, a next step is still to change the quickstart to make it easier
for folks to run with an existing webserver, with existing tls certs/keys.
that's how this relates to issue #5.

.jshintrc

@@ -3,6 +3,7 @@
 	"asi": true,
 	"strict": "implied",
 	"globals": {
+		"self": true,
 		"window": true,
 		"console": true,
 		"document": true,

README.md

@@ -31,6 +31,8 @@ See Quickstart below to get started.
   accounts/domains, and modifying the configuration file.
 - Autodiscovery (with SRV records, Microsoft-style and Thunderbird-style) for
   easy account setup (though not many clients support it).
+- Webserver with serving static files and forwarding requests (reverse
+  proxy), so port 443 can also be used to serve websites.
 - Prometheus metrics and structured logging for operational insight.
 
 Mox is available under the MIT-license and was created by Mechiel Lukkien,
@@ -54,7 +56,7 @@ Verify you have a working mox binary:
 
 	./mox version
 
-Note: Mox only compiles/works on unix systems, not on Plan 9 or Windows.
+Note: Mox only compiles for/works on unix systems, not on Plan 9 or Windows.
 
 You can also run mox with docker image "docker.io/moxmail/mox", with tags like
 "latest", "0.0.1" and "0.0.1-go1.20.1-alpine3.17.2", etc. See docker-compose.yml
@@ -66,8 +68,9 @@ in this repository for instructions on starting.
 The easiest way to get started with serving email for your domain is to get a
 vm/machine dedicated to serving email, name it [host].[domain] (e.g.
 mail.example.com), login as root, create user "mox" and its homedir by running
-"useradd -d /home/mox mox && mkdir /home/mox", download mox to that directory,
+`useradd -d /home/mox mox && mkdir /home/mox` (or pick another directory),
-and generate a configuration for your desired email address at your domain:
+download mox to that directory, and generate a configuration for your desired
+email address at your domain:
 
 	./mox quickstart you@example.com
 
@@ -75,13 +78,10 @@ This creates an account, generates a password and configuration files, prints
 the DNS records you need to manually create and prints commands to start mox and
 optionally install mox as a service.
 
-If you already have email configured for your domain, or if you are already
-sending email for your domain from other machines/services, you should modify
-the suggested configuration and/or DNS records.
-
 A dedicated machine is highly recommended because modern email requires HTTPS,
-and mox currently needs it for automatic TLS.  You can combine mox with an
+and mox currently needs it for automatic TLS.  You could combine mox with an
-existing webserver, but it requires more configuration.
+existing webserver, but it requires more configuration. If you want to serve
+websites on the same machine, use the webserver built into mox.
 
 After starting, you can access the admin web interface on internal IPs.
 
@@ -109,7 +109,6 @@ The code is heavily cross-referenced with the RFCs for readability/maintainabili
 - DANE and DNSSEC.
 - Sending DMARC and TLS reports (currently only receiving).
 - OAUTH2 support, for single sign on.
-- Basic reverse proxy, so port 443 can be used for regular web serving too.
 - Using mox as backup MX.
 - ACME verification over HTTP (in addition to current tls-alpn01).
 - Add special IMAP mailbox ("Queue?") that contains queued but
@@ -182,7 +181,7 @@ and receive emails through it with your favourite email clients, and file an
 issue if you encounter a problem or would like to see a feature/functionality
 implemented.
 
-Instead of switching your email for your domain over to mox, you could simply
+Instead of switching email for your domain over to mox, you could simply
 configure mox for a subdomain, e.g. [you]@moxtest.[yourdomain].
 
 If you have experience with how the email protocols are used in the wild, e.g.
@@ -212,17 +211,17 @@ The admin password can be changed with "mox setadminpassword".
 Unfortunately, mox does not yet provide an option for that. Mox does spam
 filtering based on reputation of received messages. It will take a good amount
 of work to share that information with a backup MX. Without that information,
-spammer could use a backup MX to get their spam accepted. Until mox has a
+spammers could use a backup MX to get their spam accepted. Until mox has a
 proper solution, you can simply run a single SMTP server.
 
 ## How do I stay up to date?
 
-Please set "CheckUpdates: true" in mox.conf. It will check for a new version
+Please set "CheckUpdates: true" in mox.conf. Mox will check for a new version
-through a DNS TXT request at `_updates.xmox.nl` once per 24h. Only if a new
+through a DNS TXT request for `_updates.xmox.nl` once per 24h. Only if a new
-version is published, will the changelog be fetched and delivered to the
+version is published will the changelog be fetched and delivered to the
 postmaster mailbox.
 
-The changelog is at https://updates.xmox.nl/changelog
+The changelog is at https://updates.xmox.nl/changelog.
 
 You could also monitor newly added tags on this repository, or for the docker
 image, but update instructions are in the changelog.
@@ -241,6 +240,6 @@ to mechiel@ueber.net.
 
 ## I'm now running an email server, but how does email work?
 
-Congrats and welcome to the club! Running an email server brings some
+Congrats and welcome to the club! Running an email server on the internet comes
-responsibility so you should understand how it works. See
+with some responsibilities so you should understand how it works. See
 https://explained-from-first-principles.com/email/ for a thorough explanation.

config/config.go

@@ -70,9 +70,12 @@ type Static struct {
 
 // Dynamic is the parsed form of domains.conf, and is automatically reloaded when changed.
 type Dynamic struct {
-	Domains     map[string]Domain  `sconf-doc:"Domains for which email is accepted. For internationalized domains, use their IDNA names in UTF-8."`
+	Domains            map[string]Domain  `sconf-doc:"Domains for which email is accepted. For internationalized domains, use their IDNA names in UTF-8."`
-	Accounts    map[string]Account `sconf-doc:"Accounts to which email can be delivered. An account can accept email for multiple domains, for multiple localparts, and deliver to multiple mailboxes."`
+	Accounts           map[string]Account `sconf-doc:"Accounts to which email can be delivered. An account can accept email for multiple domains, for multiple localparts, and deliver to multiple mailboxes."`
-	WebHandlers []WebHandler       `sconf:"optional" sconf-doc:"Handle webserver requests by serving static files, redirecting or reverse-proxying HTTP(s). The first matching WebHandler will handle the request. Built-in handlers for autoconfig and mta-sts always run first. If no handler matches, the response status code is file not found (404). If functionality you need is missng, simply forward the requests to an application that can provide the needed functionality."`
+	WebDomainRedirects map[string]string  `sconf:"optional" sconf-doc:"Redirect all requests from domain (key) to domain (value). Always redirects to HTTPS. For plain HTTP redirects, use a WebHandler with a WebRedirect."`
+	WebHandlers        []WebHandler       `sconf:"optional" sconf-doc:"Handle webserver requests by serving static files, redirecting or reverse-proxying HTTP(s). The first matching WebHandler will handle the request. Built-in handlers for autoconfig and mta-sts always run first. If no handler matches, the response status code is file not found (404). If functionality you need is missng, simply forward the requests to an application that can provide the needed functionality."`
+
+	WebDNSDomainRedirects map[dns.Domain]dns.Domain `sconf:"-"`
 }
 
 type ACME struct {
@@ -308,10 +311,9 @@ type TLS struct {
 }
 
 type WebHandler struct {
-	LogName    string `sconf:"optional" sconf-doc:"Name to use in logging and metrics."`
+	LogName               string       `sconf:"optional" sconf-doc:"Name to use in logging and metrics."`
-	Domain     string `sconf-doc:"Both Domain and PathRegexp must match for this WebHandler to match a request. Exactly one of WebStatic, WebRedirect, WebForward must be set."`
+	Domain                string       `sconf-doc:"Both Domain and PathRegexp must match for this WebHandler to match a request. Exactly one of WebStatic, WebRedirect, WebForward must be set."`
-	PathRegexp string `sconf-doc:"Regular expression matched against request path, must always start with ^ to ensure matching from the start of the path. The matching prefix can optionally be stripped by WebForward. The regular expression does not have to end with $."`
+	PathRegexp            string       `sconf-doc:"Regular expression matched against request path, must always start with ^ to ensure matching from the start of the path. The matching prefix can optionally be stripped by WebForward. The regular expression does not have to end with $."`
-
 	DontRedirectPlainHTTP bool         `sconf:"optional" sconf-doc:"If set, plain HTTP requests are not automatically permanently redirected (308) to HTTPS. If you don't have a HTTPS webserver configured, set this to true."`
 	WebStatic             *WebStatic   `sconf:"optional" sconf-doc:"Serve static files."`
 	WebRedirect           *WebRedirect `sconf:"optional" sconf-doc:"Redirect requests to configured URL."`
@@ -322,6 +324,37 @@ type WebHandler struct {
 	Path      *regexp.Regexp `sconf:"-" json:"-"`
 }
 
+// Equal returns if wh and o are equal, only looking at fields in the configuration file, not the derived fields.
+func (wh WebHandler) Equal(o WebHandler) bool {
+	clean := func(x WebHandler) WebHandler {
+		x.Name = ""
+		x.DNSDomain = dns.Domain{}
+		x.Path = nil
+		x.WebStatic = nil
+		x.WebRedirect = nil
+		x.WebForward = nil
+		return x
+	}
+	cwh := clean(wh)
+	co := clean(o)
+	if cwh != co {
+		return false
+	}
+	if (wh.WebStatic == nil) != (o.WebStatic == nil) || (wh.WebRedirect == nil) != (o.WebRedirect == nil) || (wh.WebForward == nil) != (o.WebForward == nil) {
+		return false
+	}
+	if wh.WebStatic != nil {
+		return reflect.DeepEqual(wh.WebStatic, o.WebStatic)
+	}
+	if wh.WebRedirect != nil {
+		return wh.WebRedirect.equal(*o.WebRedirect)
+	}
+	if wh.WebForward != nil {
+		return wh.WebForward.equal(*o.WebForward)
+	}
+	return true
+}
+
 type WebStatic struct {
 	StripPrefix      string            `sconf:"optional" sconf-doc:"Path to strip from the request URL before evaluating to a local path. If the requested URL path does not start with this prefix and ContinueNotFound it is considered non-matching and next WebHandlers are tried. If ContinueNotFound is not set, a file not found (404) is returned in that case."`
 	Root             string            `sconf-doc:"Directory to serve files from for this handler. Keep in mind that relative paths are relative to the working directory of mox."`
@@ -331,7 +364,7 @@ type WebStatic struct {
 }
 
 type WebRedirect struct {
-	BaseURL        string `sconf:"optional" sconf-doc:"Base URL to redirect to. The path must be empty and will be replaced, either by the request URL path, or byOrigPathRegexp/ReplacePath. Scheme, host, port and fragment stay intact, and query strings are combined. If empty, the response redirects to a different path through OrigPathRegexp and ReplacePath, which must then be set. Use a URL without scheme to redirect without changing the protocol, e.g. //newdomain/."`
+	BaseURL        string `sconf:"optional" sconf-doc:"Base URL to redirect to. The path must be empty and will be replaced, either by the request URL path, or by OrigPathRegexp/ReplacePath. Scheme, host, port and fragment stay intact, and query strings are combined. If empty, the response redirects to a different path through OrigPathRegexp and ReplacePath, which must then be set. Use a URL without scheme to redirect without changing the protocol, e.g. //newdomain/."`
 	OrigPathRegexp string `sconf:"optional" sconf-doc:"Regular expression for matching path. If set and path does not match, a 404 is returned. The HTTP path used for matching always starts with a slash."`
 	ReplacePath    string `sconf:"optional" sconf-doc:"Replacement path for destination URL based on OrigPathRegexp. Implemented with Go's Regexp.ReplaceAllString: $1 is replaced with the text of the first submatch, etc. If both OrigPathRegexp and ReplacePath are empty, BaseURL must be set and all paths are redirected unaltered."`
 	StatusCode     int    `sconf:"optional" sconf-doc:"Status code to use in redirect, e.g. 307. By default, a permanent redirect (308) is returned."`
@@ -340,6 +373,14 @@ type WebRedirect struct {
 	OrigPath *regexp.Regexp `sconf:"-" json:"-"`
 }
 
+func (wr WebRedirect) equal(o WebRedirect) bool {
+	wr.URL = nil
+	wr.OrigPath = nil
+	o.URL = nil
+	o.OrigPath = nil
+	return reflect.DeepEqual(wr, o)
+}
+
 type WebForward struct {
 	StripPath       bool              `sconf:"optional" sconf-doc:"Strip the matching WebHandler path from the WebHandler before forwarding the request."`
 	URL             string            `sconf-doc:"URL to forward HTTP requests to, e.g. http://127.0.0.1:8123/base. If StripPath is false the full request path is added to the URL. Host headers are sent unmodified. New X-Forwarded-{For,Host,Proto} headers are set. Any query string in the URL is ignored. Requests are made using Go's net/http.DefaultTransport that takes environment variables HTTP_PROXY and HTTPS_PROXY into account."`
@@ -347,3 +388,9 @@ type WebForward struct {
 
 	TargetURL *url.URL `sconf:"-" json:"-"`
 }
+
+func (wf WebForward) equal(o WebForward) bool {
+	wf.TargetURL = nil
+	o.TargetURL = nil
+	return reflect.DeepEqual(wf, o)
+}

config/doc.go

@@ -557,6 +557,11 @@ describe-static" and "mox config describe-domains":
 					# in calculating probability reduced. E.g. 1 or 2. (optional)
 					RareWords: 0
 
+	# Redirect all requests from domain (key) to domain (value). Always redirects to
+	# HTTPS. For plain HTTP redirects, use a WebHandler with a WebRedirect. (optional)
+	WebDomainRedirects:
+		x:
+
 	# Handle webserver requests by serving static files, redirecting or
 	# reverse-proxying HTTP(s). The first matching WebHandler will handle the request.
 	# Built-in handlers for autoconfig and mta-sts always run first. If no handler
@@ -622,7 +627,7 @@ describe-static" and "mox config describe-domains":
 			WebRedirect:
 
 				# Base URL to redirect to. The path must be empty and will be replaced, either by
-				# the request URL path, or byOrigPathRegexp/ReplacePath. Scheme, host, port and
+				# the request URL path, or by OrigPathRegexp/ReplacePath. Scheme, host, port and
 				# fragment stay intact, and query strings are combined. If empty, the response
 				# redirects to a different path through OrigPathRegexp and ReplacePath, which must
 				# then be set. Use a URL without scheme to redirect without changing the protocol,
@@ -670,14 +675,20 @@ examples with "mox examples", and print a specific example with "mox examples
 
 # Example webhandlers
 
-	# Snippet of domains.conf to configure WebHandlers.
+	# Snippet of domains.conf to configure WebDomainRedirects and WebHandlers.
+
+	# Redirect all requests for mox.example to https://www.mox.example.
+	WebDomainRedirects:
+		mox.example: www.mox.example
+
+	# Each request is matched against these handlers until one matches and serves it.
 	WebHandlers:
 		-
 			# The name of the handler, used in logging and metrics.
 			LogName: staticmjl
 			# With ACME configured, each configured domain will automatically get a TLS
 			# certificate on first request.
-			Domain: mox.example
+			Domain: www.mox.example
 			PathRegexp: ^/who/mjl/
 			WebStatic:
 				StripPrefix: /who/mjl
@@ -690,7 +701,7 @@ examples with "mox examples", and print a specific example with "mox examples
 					X-Mox: hi
 		-
 			LogName: redir
-			Domain: mox.example
+			Domain: www.mox.example
 			PathRegexp: ^/redir/a/b/c
 			# Don't redirect from plain HTTP to HTTPS.
 			DontRedirectPlainHTTP: true
@@ -703,7 +714,7 @@ examples with "mox examples", and print a specific example with "mox examples
 				StatusCode: 307
 		-
 			LogName: oldnew
-			Domain: mox.example
+			Domain: www.mox.example
 			PathRegexp: ^/old/
 			WebRedirect:
 				# Replace path, leaving rest of URL intact.
@@ -711,7 +722,7 @@ examples with "mox examples", and print a specific example with "mox examples
 				ReplacePath: /new/$1
 		-
 			LogName: app
-			Domain: mox.example
+			Domain: www.mox.example
 			PathRegexp: ^/app/
 			WebForward:
 				# Strip the path matched by PathRegexp before forwarding the request. So original

docker-compose.yml

@@ -20,6 +20,9 @@ services:
     volumes:
       - ./config:/mox/config
       - ./data:/mox/data
+      # web is optional but recommended to bind in, useful for serving static files with
+      # the webserver.
+      - ./web:/mox/web
     working_dir: /mox
     restart: on-failure
     healthcheck:

http/account.go

@@ -76,7 +76,7 @@ func checkAccountAuth(ctx context.Context, log *mlog.Log, w http.ResponseWriter,
 	}
 	if addr != nil && !mox.LimiterFailedAuth.Add(addr.IP, start, 1) {
 		metrics.AuthenticationRatelimitedInc("httpaccount")
-		http.Error(w, "http 429 - too many auth attempts", http.StatusTooManyRequests)
+		http.Error(w, "429 - too many auth attempts", http.StatusTooManyRequests)
 		return ""
 	}
 

http/admin.go

@@ -16,6 +16,7 @@ import (
 	"net"
 	"net/http"
 	"os"
+	"reflect"
 	"runtime/debug"
 	"sort"
 	"strings"
@@ -31,6 +32,7 @@ import (
 	"github.com/mjl-/sherpadoc"
 	"github.com/mjl-/sherpaprom"
 
+	"github.com/mjl-/mox/config"
 	"github.com/mjl-/mox/dkim"
 	"github.com/mjl-/mox/dmarc"
 	"github.com/mjl-/mox/dmarcdb"
@@ -136,7 +138,7 @@ func checkAdminAuth(ctx context.Context, passwordfile string, w http.ResponseWri
 	}
 	if addr != nil && !mox.LimiterFailedAuth.Add(addr.IP, start, 1) {
 		metrics.AuthenticationRatelimitedInc("httpadmin")
-		http.Error(w, "http 429 - too many auth attempts", http.StatusTooManyRequests)
+		http.Error(w, "429 - too many auth attempts", http.StatusTooManyRequests)
 		return false
 	}
 
@@ -1525,3 +1527,73 @@ func (Admin) LogLevelRemove(ctx context.Context, pkg string) {
 func (Admin) CheckUpdatesEnabled(ctx context.Context) bool {
 	return mox.Conf.Static.CheckUpdates
 }
+
+// WebserverConfig is the combination of WebDomainRedirects and WebHandlers
+// from the domains.conf configuration file.
+type WebserverConfig struct {
+	WebDNSDomainRedirects [][2]dns.Domain // From server to frontend.
+	WebDomainRedirects    [][2]string     // From frontend to server, it's not convenient to create dns.Domain in the frontend.
+	WebHandlers           []config.WebHandler
+}
+
+// WebserverConfig returns the current webserver config
+func (Admin) WebserverConfig(ctx context.Context) (conf WebserverConfig) {
+	conf = webserverConfig()
+	conf.WebDomainRedirects = nil
+	return conf
+}
+
+func webserverConfig() WebserverConfig {
+	r, l := mox.Conf.WebServer()
+	x := make([][2]dns.Domain, 0, len(r))
+	xs := make([][2]string, 0, len(r))
+	for k, v := range r {
+		x = append(x, [2]dns.Domain{k, v})
+		xs = append(xs, [2]string{k.Name(), v.Name()})
+	}
+	sort.Slice(x, func(i, j int) bool {
+		return x[i][0].ASCII < x[j][0].ASCII
+	})
+	sort.Slice(xs, func(i, j int) bool {
+		return xs[i][0] < xs[j][0]
+	})
+	return WebserverConfig{x, xs, l}
+}
+
+// WebserverConfigSave saves a new webserver config. If oldConf is not equal to
+// the current config, an error is returned.
+func (Admin) WebserverConfigSave(ctx context.Context, oldConf, newConf WebserverConfig) (savedConf WebserverConfig) {
+	current := webserverConfig()
+	webhandlersEqual := func() bool {
+		if len(current.WebHandlers) != len(oldConf.WebHandlers) {
+			return false
+		}
+		for i, wh := range current.WebHandlers {
+			if !wh.Equal(oldConf.WebHandlers[i]) {
+				return false
+			}
+		}
+		return true
+	}
+	if !reflect.DeepEqual(oldConf.WebDNSDomainRedirects, current.WebDNSDomainRedirects) || !webhandlersEqual() {
+		xcheckf(ctx, errors.New("config has changed"), "comparing old/current config")
+	}
+
+	// Convert to map, check that there are no duplicates here. The canonicalized
+	// dns.Domain are checked again for uniqueness when parsing the config before
+	// storing.
+	domainRedirects := map[string]string{}
+	for _, x := range newConf.WebDomainRedirects {
+		if _, ok := domainRedirects[x[0]]; ok {
+			xcheckf(ctx, errors.New("already present"), "checking redirect %s", x[0])
+		}
+		domainRedirects[x[0]] = x[1]
+	}
+
+	err := mox.WebserverConfigSet(ctx, domainRedirects, newConf.WebHandlers)
+	xcheckf(ctx, err, "saving webserver config")
+
+	savedConf = webserverConfig()
+	savedConf.WebDomainRedirects = nil
+	return savedConf
+}

http/admin.html

@@ -14,6 +14,9 @@ h3, h4 { font-size: 1rem; }
 ul { padding-left: 1rem; }
 .literal { background-color: #fdfdfd; padding: .5em 1em; border: 1px solid #eee; border-radius: 4px; white-space: pre-wrap; font-family: monospace; font-size: 15px; tab-size: 4; }
 table td, table th { padding: .2em .5em; }
+table table td, table table th { padding: 0 0.1em; }
+table.long >tbody >tr >td { padding: 1em .5em; }
+table.long td { vertical-align: top; }
 table > tbody > tr:nth-child(odd) { background-color: #f8f8f8; }
 .text { max-width: 50em; }
 p { margin-bottom: 1em; max-width: 50em; }
@@ -267,7 +270,8 @@ const index = async () => {
 		dom.div(dom.a('DNSBL status', attr({href: '#dnsbl'}))),
 		dom.br(),
 		dom.h2('Configuration'),
-		dom.div(dom.a('See configuration', attr({href: '#config'}))),
+		dom.div(dom.a('Webserver', attr({href: '#webserver'}))),
+		dom.div(dom.a('Files', attr({href: '#config'}))),
 		dom.div(dom.a('Log levels', attr({href: '#loglevels'}))),
 		footer,
 	)
@@ -1561,6 +1565,541 @@ const queueList = async () => {
 	)
 }
 
+const webserver = async () => {
+	let conf = await api.WebserverConfig()
+
+	// We disable this while saving the form.
+	let fieldset
+
+	// Keep track of redirects. Rows are objects that hold both the DOM and allows
+	// retrieving the visible (modified) data to construct a config for saving.
+	let redirectRows = []
+	let redirectsTbody
+	let noredirect
+
+	// Similar to redirects, but for web handlers.
+	let handlerRows = []
+	let handlersTbody
+	let nohandler
+
+	// Make a new redirect rows, adding it to the list. The caller typically uses this
+	// while building the DOM, the element is added because this object has it as
+	// "root" field.
+	const redirectRow = (t) => {
+		const row = {}
+		row.root = dom.tr(
+			dom.td(
+				row.from=dom.input(attr({required: '', value: domainName(t[0])})),
+			),
+			dom.td(
+				row.to=dom.input(attr({required: '', value: domainName(t[1])})),
+			),
+			dom.td(
+				dom.button('Remove', attr({type: 'button'}), function click(e) {
+					redirectRows = redirectRows.filter(r => r !== row)
+					row.root.remove()
+					noredirect.style.display = redirectRows.length ? 'none' : ''
+				}),
+			),
+		)
+		// "get" is the common function to retrieve the data from an object with a root field as DOM element.
+		row.get = () => [row.from.value, row.to.value]
+		redirectRows.push(row)
+		return row
+	}
+
+	// Reusable component for managing headers. Just a table with a header key and
+	// value. We can remove existing rows, and add new rows, and edit existing.
+	const makeHeaders = (h) => {
+		const r = {
+			rows: [],
+		}
+		let tbody, norow
+		const headerRow = (k, v) => {
+			const row = {}
+			row.root = dom.tr(
+				dom.td(
+					row.key=dom.input(attr({required: '', value: k})),
+				),
+				dom.td(
+					row.value=dom.input(attr({required: '', value: v})),
+				),
+				dom.td(
+					dom.button('Remove', attr({type: 'button'}), function click(e) {
+						r.rows = r.rows.filter(x => x !== row)
+						row.root.remove()
+						norow.style.display = r.rows.length ? 'none' : ''
+					})
+				),
+			)
+			r.rows.push(row)
+			row.get = () => [row.key.value, row.value.value]
+			return row
+		}
+		r.add = dom.button('Add', attr({type: 'button'}), function click(e) {
+			const row = headerRow('', '')
+			tbody.appendChild(row.root)
+			norow.style.display = r.rows.length ? 'none' : ''
+		})
+		r.root = dom.table(
+			tbody=dom.tbody(
+				Object.entries(h).sort().map(t => headerRow(t[0], t[1])),
+				norow=dom.tr(
+					style({display: r.rows.length ? 'none' : ''}),
+					dom.td(attr({colspan: 3}), 'None added.'),
+				)
+			),
+		)
+		r.get = () => Object.fromEntries(r.rows.map(row => row.get()))
+		return r
+	}
+
+	// todo: make a mechanism to get the ../config/config.go sconf-doc struct tags
+	// here. So we can use them for the titles, as documentation. Instead of current
+	// approach of copy/pasting those texts, inevitably will get out of date.
+
+	// todo: perhaps lay these out in the same way as in the config file? will help admins mentally map between the two. will take a bit more vertical screen space, but current approach looks messy/garbled. we could use that mechanism for more parts of the configuration file. we can even show the same sconf-doc struct tags. the html admin page will then just be a glorified guided text editor!
+
+	// Make a handler row. This is more complicated, since it can be one of the three
+	// types (static, redirect, forward), and can change between those types.
+	const handlerRow = (wh) => {
+		// We make and remember components for headers, possibly not used.
+		const row = {
+			staticHeaders: makeHeaders((wh.WebStatic || {}).ResponseHeaders || {}),
+			forwardHeaders: makeHeaders((wh.WebForward || {}).ResponseHeaders || {}),
+		}
+
+		const makeWebStatic = () => {
+			const ws = wh.WebStatic || {}
+			row.getDetails = () => {
+				return {
+					StripPrefix: row.StripPrefix.value,
+					Root: row.Root.value,
+					ListFiles: row.ListFiles.checked,
+					ContinueNotFound: row.ContinueNotFound.checked,
+					ResponseHeaders: row.staticHeaders.get(),
+				}
+			}
+			return dom.table(
+				dom.tr(
+					dom.td('Type'),
+					dom.td(
+						'StripPrefix',
+						attr({title: 'Path to strip from the request URL before evaluating to a local path. If the requested URL path does not start with this prefix and ContinueNotFound it is considered non-matching and next WebHandlers are tried. If ContinueNotFound is not set, a file not found (404) is returned in that case.'}),
+					),
+					dom.td(
+						'Root',
+						attr({title: 'Directory to serve files from for this handler. Keep in mind that relative paths are relative to the working directory of mox.'}),
+					),
+					dom.td(
+						'ListFiles',
+						attr({title: 'If set, and a directory is requested, and no index.html is present that can be served, a file listing is returned. Results in 403 if ListFiles is not set. If a directory is requested and the URL does not end with a slash, the response is a redirect to the path with trailing slash.'}),
+					),
+					dom.td(
+						'ContinueNotFound',
+						attr({title: "If a requested URL does not exist, don't return a file not found (404) response, but consider this handler non-matching and continue attempts to serve with later WebHandlers, which may be a reverse proxy generating dynamic content, possibly even writing a static file for a next request to serve statically. If ContinueNotFound is set, HTTP requests other than GET and HEAD do not match. This mechanism can be used to implement the equivalent of 'try_files' in other webservers."}),
+					),
+					dom.td(
+						dom.span(
+							'Response headers',
+							attr({title: 'Headers to add to the response. Useful for cache-control, content-type, etc. By default, Content-Type headers are automatically added for recognized file types, unless added explicitly through this setting. For directory listings, a content-type header is skipped.'}),
+						),
+						' ',
+						row.staticHeaders.add,
+					),
+				),
+				dom.tr(
+					dom.td(
+						row.type=dom.select(
+							attr({required: ''}),
+							dom.option('Static', attr({selected: ''})),
+							dom.option('Redirect'),
+							dom.option('Forward'),
+							function change(e) {
+								makeType(e.target.value)
+							},
+						),
+					),
+					dom.td(
+						row.StripPrefix=dom.input(attr({value: ws.StripPrefix || ''})),
+					),
+					dom.td(
+						row.Root=dom.input(attr({required: '', placeholder: 'web/...', value: ws.Root || ''})),
+					),
+					dom.td(
+						row.ListFiles=dom.input(attr({type: 'checkbox'}), ws.ListFiles ? attr({checked: ''}) : []),
+					),
+					dom.td(
+						row.ContinueNotFound=dom.input(attr({type: 'checkbox'}), ws.ContinueNotFound ? attr({checked: ''}) : []),
+					),
+					dom.td(
+						row.staticHeaders,
+					),
+				)
+			)
+		}
+
+		const makeWebRedirect = () => {
+			const wr = wh.WebRedirect || {}
+			row.getDetails = () => {
+				return {
+					BaseURL: row.BaseURL.value,
+					OrigPathRegexp: row.OrigPathRegexp.value,
+					ReplacePath: row.ReplacePath.value,
+					StatusCode: row.StatusCode.value ? parseInt(row.StatusCode.value) : 0,
+				}
+			}
+			return dom.table(
+				dom.tr(
+					dom.td('Type'),
+					dom.td(
+						'BaseURL',
+						attr({title: 'Base URL to redirect to. The path must be empty and will be replaced, either by the request URL path, or by OrigPathRegexp/ReplacePath. Scheme, host, port and fragment stay intact, and query strings are combined. If empty, the response redirects to a different path through OrigPathRegexp and ReplacePath, which must then be set. Use a URL without scheme to redirect without changing the protocol, e.g. //newdomain/.'}),
+					),
+					dom.td(
+						'OrigPathRegexp',
+						attr({title: 'Regular expression for matching path. If set and path does not match, a 404 is returned. The HTTP path used for matching always starts with a slash.'}),
+					),
+					dom.td(
+						'ReplacePath',
+						attr({title: "Replacement path for destination URL based on OrigPathRegexp. Implemented with Go's Regexp.ReplaceAllString: $1 is replaced with the text of the first submatch, etc. If both OrigPathRegexp and ReplacePath are empty, BaseURL must be set and all paths are redirected unaltered."}),
+					),
+					dom.td(
+						'StatusCode',
+						attr({title: 'Status code to use in redirect, e.g. 307. By default, a permanent redirect (308) is returned.'}),
+					),
+				),
+				dom.tr(
+					dom.td(
+						row.type=dom.select(
+							attr({required: ''}),
+							dom.option('Static'),
+							dom.option('Redirect', attr({selected: ''})),
+							dom.option('Forward'),
+							function change(e) {
+								makeType(e.target.value)
+							},
+						),
+					),
+					dom.td(
+						row.BaseURL=dom.input(attr({placeholder: 'empty or https://target/path?q=1#frag or //target/...', value: wr.BaseURL || ''})),
+					),
+					dom.td(
+						row.OrigPathRegexp=dom.input(attr({placeholder: '^/old/(.*)', value: wr.OrigPathRegexp || ''})),
+					),
+					dom.td(
+						row.ReplacePath=dom.input(attr({placeholder: '/new/$1', value: wr.ReplacePath || ''})),
+					),
+					dom.td(
+						row.StatusCode=dom.input(style({width: '4em'}), attr({type: 'number', value: wr.StatusCode || '', min: 300, max: 399})),
+					),
+				),
+			)
+		}
+
+		const makeWebForward = () => {
+			const wf = wh.WebForward || {}
+			row.getDetails = () => {
+				return {
+					StripPath: row.StripPath.checked,
+					URL: row.URL.value,
+					ResponseHeaders: row.forwardHeaders.get(),
+				}
+			}
+			return dom.table(
+				dom.tr(
+					dom.td('Type'),
+					dom.td(
+						'StripPath',
+						attr({title: 'Strip the matching WebHandler path from the WebHandler before forwarding the request.'}),
+					),
+					dom.td(
+						'URL',
+						attr({title: "URL to forward HTTP requests to, e.g. http://127.0.0.1:8123/base. If StripPath is false the full request path is added to the URL. Host headers are sent unmodified. New X-Forwarded-{For,Host,Proto} headers are set. Any query string in the URL is ignored. Requests are made using Go's net/http.DefaultTransport that takes environment variables HTTP_PROXY and HTTPS_PROXY into account."}),
+					),
+					dom.td(
+						dom.span(
+							'Response headers',
+							attr({title: 'Headers to add to the response. Useful for adding security- and cache-related headers.'}),
+						),
+						' ',
+						row.forwardHeaders.add,
+					),
+				),
+				dom.tr(
+					dom.td(
+						row.type=dom.select(
+							attr({required: ''}),
+							dom.option('Static', ),
+							dom.option('Redirect'),
+							dom.option('Forward', attr({selected: ''})),
+							function change(e) {
+								makeType(e.target.value)
+							},
+						),
+					),
+					dom.td(
+						row.StripPath=dom.input(attr({type: 'checkbox'}), wf.StripPath || wf.StripPath === undefined ? attr({checked: ''}) : []),
+					),
+					dom.td(
+						row.URL=dom.input(attr({required: '', placeholder: 'http://127.0.0.1:8888', value: wf.URL || ''})),
+					),
+					dom.td(
+						row.forwardHeaders,
+					),
+				),
+			)
+		}
+
+		// Transform the input fields to match the type of WebHandler.
+		const makeType = (s) => {
+			let details
+			if (s === 'Static') {
+				details = makeWebStatic()
+			} else if (s === 'Redirect') {
+				details = makeWebRedirect()
+			} else if (s === 'Forward') {
+				details = makeWebForward()
+			} else {
+				throw new Error('unknown handler type')
+			}
+			row.details.replaceWith(details)
+			row.details = details
+		}
+
+		// Remove row from oindex, insert it in nindex. Both in handlerRows and in the DOM.
+		const moveHandler = (row, oindex, nindex) => {
+			row.root.remove()
+			handlersTbody.insertBefore(row.root, handlersTbody.children[nindex])
+			handlerRows.splice(oindex, 1)
+			handlerRows.splice(nindex, 0, row)
+		}
+
+		// Row that starts starts with two tables: one for the fields all WebHandlers have
+		// (in common). And one for the details, i.e. WebStatic, WebRedirect, WebForward.
+		row.root = dom.tr(
+			dom.td(
+				dom.table(
+					dom.tr(
+						dom.td('LogName', attr({title: 'Name used during logging for requests matching this handler. If empty, the index of the handler in the list is used.'})),
+						dom.td('Domain', attr({title: 'Request must be for this domain to match this handler.'})),
+						dom.td('Path Regexp', attr({title: 'Request must match this path regular expression to match this handler. Must start with with a ^.'})),
+						dom.td('To HTTPS', attr({title: 'Redirect plain HTTP (non-TLS) requests to HTTPS'})),
+					),
+					dom.tr(
+						dom.td(
+							row.LogName=dom.input(attr({value: wh.LogName || ''})),
+						),
+						dom.td(
+							row.Domain=dom.input(attr({required: '', placeholder: 'example.org', value: domainName(wh.DNSDomain)})),
+						),
+						dom.td(
+							row.PathRegexp=dom.input(attr({required: '', placeholder: '^/', value: wh.PathRegexp || ''})),
+						),
+						dom.td(
+							row.ToHTTPS=dom.input(attr({type: 'checkbox', title: 'Redirect plain HTTP (non-TLS) requests to HTTPS'}), !wh.DontRedirectPlainHTTP ? attr({checked: ''}) : []),
+						),
+					),
+				),
+				// Replaced with a call to makeType, below (and later when switching types).
+				row.details=dom.table(),
+			),
+			dom.td(
+				dom.td(
+					dom.button('Remove', attr({type: 'button'}), function click(e) {
+						handlerRows = handlerRows.filter(r => r !== row)
+						row.root.remove()
+						nohandler.style.display = handlerRows.length ? 'none' : ''
+					}),
+					' ',
+					// We show/hide the buttons to move when clicking the Move button.
+					row.moveButtons=dom.span(
+						style({display: 'none'}),
+						dom.button('↑↑', attr({type: 'button', title: 'Move to top.'}), function click(e) {
+							const index = handlerRows.findIndex(r => r === row)
+							if (index > 0) {
+								moveHandler(row, index, 0)
+							}
+						}),
+						' ',
+						dom.button('↑', attr({type: 'button', title: 'Move one up.'}), function click(e) {
+							const index = handlerRows.findIndex(r => r === row)
+							if (index > 0) {
+								moveHandler(row, index, index-1)
+							}
+						}),
+						' ',
+						dom.button('↓', attr({type: 'button', title: 'Move one down.'}), function click(e) {
+							const index = handlerRows.findIndex(r => r === row)
+							if (index+1 < handlerRows.length) {
+								moveHandler(row, index, index+1)
+							}
+						}),
+						' ',
+						dom.button('↓↓', attr({type: 'button', title: 'Move to bottom.'}), function click(e) {
+							const index = handlerRows.findIndex(r => r === row)
+							if (index+1 < handlerRows.length) {
+								moveHandler(row, index, handlerRows.length-1)
+							}
+						}),
+					),
+				),
+			),
+		)
+
+		// Final "get" that returns a WebHandler that reflects the UI.
+		row.get = () => {
+			const wh = {
+				LogName: row.LogName.value,
+				Domain: row.Domain.value,
+				PathRegexp: row.PathRegexp.value,
+				DontRedirectPlainHTTP: !row.ToHTTPS.checked,
+			}
+			const s = row.type.value
+			const details = row.getDetails()
+			if (s === 'Static') {
+				wh.WebStatic = details
+			} else if (s === 'Redirect') {
+				wh.WebRedirect = details
+			} else if (s === 'Forward') {
+				wh.WebForward = details
+			}
+			return wh
+		}
+
+		// Initialize one of the Web* types.
+		let s
+		if (wh.WebStatic) {
+			s = 'Static'
+		} else if (wh.WebRedirect) {
+			s = 'Redirect'
+		} else if (wh.WebForward) {
+			s = 'Forward'
+		}
+		makeType(s)
+
+		handlerRows.push(row)
+		return row
+	}
+
+	// Return webserver config to store.
+	const gatherConf = () => {
+		return {
+			WebDomainRedirects: redirectRows.map(row => row.get()),
+			WebHandlers: handlerRows.map(row => row.get()),
+		}
+	}
+
+	// Add and move buttons, both above and below the table for quick access, hence a function.
+	const handlerActions = () => {
+		return [
+			'Action ',
+			dom.button('Add', attr({type: 'button'}), function click(e) {
+				// New WebHandler added as WebForward. Good chance this is what the user wants. And
+				// it has the least fields. (;
+				const nwh = {
+					LogName: '',
+					DNSDomain: {ASCII: ''},
+					PathRegexp: '^/',
+					DontRedirectPlainHTTP: false,
+					WebForward: {
+						StripPath: true,
+						URL: '',
+					},
+				}
+				const row = handlerRow(nwh)
+				handlersTbody.appendChild(row.root)
+				nohandler.style.display = handlerRows.length ? 'none' : ''
+			}),
+			' ',
+			dom.button('Move', attr({type: 'button'}), function click(e) {
+				for(const row of handlerRows) {
+					row.moveButtons.style.display = row.moveButtons.style.display === 'none' ? '' : 'none'
+				}
+			}),
+		]
+	}
+
+	const page = document.getElementById('page')
+	dom._kids(page,
+		crumbs(
+			crumblink('Mox Admin', '#'),
+			'Webserver config',
+		),
+		dom.form(
+			fieldset=dom.fieldset(
+				dom.h2('Domain redirects', attr({title: 'Corresponds with WebDomainRedirects in domains.conf'})),
+				dom.p('Incoming requests for these domains are redirected to the target domain, with HTTPS.'),
+				dom.table(
+					dom.thead(
+						dom.tr(
+							dom.th('From'),
+							dom.th('To'),
+							dom.th(
+								'Action ',
+								dom.button('Add', attr({type: 'button'}), function click(e) {
+									const row = redirectRow([{ASCII: ''}, {ASCII: ''}])
+									redirectsTbody.appendChild(row.root)
+									noredirect.style.display = redirectRows.length ? 'none' : ''
+								}),
+							),
+						),
+					),
+					redirectsTbody=dom.tbody(
+						(conf.WebDNSDomainRedirects || []).sort().map(t => redirectRow(t)),
+						noredirect=dom.tr(
+							style({display: redirectRows.length ? 'none' : ''}),
+							dom.td(attr({colspan: 3}), 'No redirects.'),
+						),
+					),
+				),
+				dom.br(),
+				dom.h2('Handlers', attr({title: 'Corresponds with WebHandlers in domains.conf'})),
+				dom.p('Each incoming request is check against these handlers, in order. The first matching handler serves the request.'),
+				dom('table.long',
+					dom.thead(
+						dom.tr(
+							dom.th(),
+							dom.th(handlerActions()),
+						),
+					),
+					handlersTbody=dom.tbody(
+						(conf.WebHandlers || []).map(wh => handlerRow(wh)),
+						nohandler=dom.tr(
+							style({display: handlerRows.length ? 'none' : ''}),
+							dom.td(attr({colspan: 2}), 'No handlers.'),
+						),
+					),
+					dom.tfoot(
+						dom.tr(
+							dom.th(),
+							dom.th(handlerActions()),
+						),
+					),
+				),
+				dom.br(),
+				dom.button('Save', attr({type: 'submit'}), attr({title: 'Save config. If the configuration has changed since this page was loaded, an error will be returned. After saving, the changes take effect immediately.'})),
+			),
+			async function submit(e) {
+				e.preventDefault()
+				e.stopPropagation()
+
+				fieldset.disabled = true
+				try {
+					const newConf = gatherConf()
+					const savedConf = await api.WebserverConfigSave(conf, newConf)
+					conf = savedConf
+				} catch (err) {
+					console.log({err})
+					window.alert('Error: ' + err.message)
+				} finally {
+					fieldset.disabled = false
+				}
+			}
+		),
+	)
+}
+
 const init = async () => {
 	let curhash
 
@@ -1611,6 +2150,8 @@ const init = async () => {
 				await mtasts()
 			} else if (h === 'dnsbl') {
 				await dnsbl()
+			} else if (h === 'webserver') {
+				await webserver()
 			} else {
 				dom._kids(page, 'page not found')
 			}

http/adminapi.json

@@ -649,6 +649,45 @@
 					]
 				}
 			]
+		},
+		{
+			"Name": "WebserverConfig",
+			"Docs": "WebserverConfig returns the current webserver config",
+			"Params": [],
+			"Returns": [
+				{
+					"Name": "conf",
+					"Typewords": [
+						"WebserverConfig"
+					]
+				}
+			]
+		},
+		{
+			"Name": "WebserverConfigSave",
+			"Docs": "WebserverConfigSave saves a new webserver config. If oldConf is not equal to\nthe current config, an error is returned.",
+			"Params": [
+				{
+					"Name": "oldConf",
+					"Typewords": [
+						"WebserverConfig"
+					]
+				},
+				{
+					"Name": "newConf",
+					"Typewords": [
+						"WebserverConfig"
+					]
+				}
+			],
+			"Returns": [
+				{
+					"Name": "savedConf",
+					"Typewords": [
+						"WebserverConfig"
+					]
+				}
+			]
 		}
 	],
 	"Sections": [],
@@ -2898,6 +2937,214 @@
 					]
 				}
 			]
+		},
+		{
+			"Name": "WebserverConfig",
+			"Docs": "WebserverConfig is the combination of WebDomainRedirects and WebHandlers\nfrom the domains.conf configuration file.",
+			"Fields": [
+				{
+					"Name": "WebDNSDomainRedirects",
+					"Docs": "From server to frontend.",
+					"Typewords": [
+						"[]",
+						"[]",
+						"Domain"
+					]
+				},
+				{
+					"Name": "WebDomainRedirects",
+					"Docs": "From frontend to server, it's not convenient to create dns.Domain in the frontend.",
+					"Typewords": [
+						"[]",
+						"[]",
+						"string"
+					]
+				},
+				{
+					"Name": "WebHandlers",
+					"Docs": "",
+					"Typewords": [
+						"[]",
+						"WebHandler"
+					]
+				}
+			]
+		},
+		{
+			"Name": "WebHandler",
+			"Docs": "",
+			"Fields": [
+				{
+					"Name": "LogName",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "Domain",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "PathRegexp",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "DontRedirectPlainHTTP",
+					"Docs": "",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "WebStatic",
+					"Docs": "",
+					"Typewords": [
+						"nullable",
+						"WebStatic"
+					]
+				},
+				{
+					"Name": "WebRedirect",
+					"Docs": "",
+					"Typewords": [
+						"nullable",
+						"WebRedirect"
+					]
+				},
+				{
+					"Name": "WebForward",
+					"Docs": "",
+					"Typewords": [
+						"nullable",
+						"WebForward"
+					]
+				},
+				{
+					"Name": "Name",
+					"Docs": "Either LogName, or numeric index if LogName was empty. Used instead of LogName in logging/metrics.",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "DNSDomain",
+					"Docs": "",
+					"Typewords": [
+						"Domain"
+					]
+				}
+			]
+		},
+		{
+			"Name": "WebStatic",
+			"Docs": "",
+			"Fields": [
+				{
+					"Name": "StripPrefix",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "Root",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "ListFiles",
+					"Docs": "",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "ContinueNotFound",
+					"Docs": "",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "ResponseHeaders",
+					"Docs": "",
+					"Typewords": [
+						"{}",
+						"string"
+					]
+				}
+			]
+		},
+		{
+			"Name": "WebRedirect",
+			"Docs": "",
+			"Fields": [
+				{
+					"Name": "BaseURL",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "OrigPathRegexp",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "ReplacePath",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "StatusCode",
+					"Docs": "",
+					"Typewords": [
+						"int32"
+					]
+				}
+			]
+		},
+		{
+			"Name": "WebForward",
+			"Docs": "",
+			"Fields": [
+				{
+					"Name": "StripPath",
+					"Docs": "",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "URL",
+					"Docs": "",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "ResponseHeaders",
+					"Docs": "",
+					"Typewords": [
+						"{}",
+						"string"
+					]
+				}
+			]
 		}
 	],
 	"Ints": [],

http/web.go

@@ -31,20 +31,41 @@ import (
 
 var xlog = mlog.New("http")
 
-var metricHTTPServer = promauto.NewHistogramVec(
+var (
-	prometheus.HistogramOpts{
+	// metricRequest tracks performance (time to write response header) of server.
-		Name:    "mox_httpserver_request_duration_seconds",
+	metricRequest = promauto.NewHistogramVec(
-		Help:    "HTTP(s) server request with handler name, protocol, method, result codes, and duration in seconds.",
+		prometheus.HistogramOpts{
-		Buckets: []float64{0.001, 0.005, 0.01, 0.05, 0.100, 0.5, 1, 5, 10, 20, 30, 60, 120},
+			Name:    "mox_httpserver_request_duration_seconds",
-	},
+			Help:    "HTTP(s) server request with handler name, protocol, method, result codes, and duration until response status code is written, in seconds.",
-	[]string{
+			Buckets: []float64{0.001, 0.005, 0.01, 0.05, 0.100, 0.5, 1, 5, 10, 20, 30, 60, 120},
-		"handler", // Name from webhandler, can be empty.
+		},
-		"proto",   // "http" or "https"
+		[]string{
-		"method",  // "(unknown)" and otherwise only common verbs
+			"handler", // Name from webhandler, can be empty.
-		"code",
+			"proto",   // "http" or "https"
-	},
+			"method",  // "(unknown)" and otherwise only common verbs
+			"code",
+		},
+	)
+	// metricResponse tracks performance of entire request as experienced by users,
+	// which also depends on their connection speed, so not necessarily something you
+	// could act on.
+	metricResponse = promauto.NewHistogramVec(
+		prometheus.HistogramOpts{
+			Name:    "mox_httpserver_response_duration_seconds",
+			Help:    "HTTP(s) server response with handler name, protocol, method, result codes, and duration of entire response, in seconds.",
+			Buckets: []float64{0.001, 0.005, 0.01, 0.05, 0.100, 0.5, 1, 5, 10, 20, 30, 60, 120},
+		},
+		[]string{
+			"handler", // Name from webhandler, can be empty.
+			"proto",   // "http" or "https"
+			"method",  // "(unknown)" and otherwise only common verbs
+			"code",
+		},
+	)
 )
 
+// todo: automatic gzip on responses, if client supports it, content is not already compressed. in case of static file only if it isn't too large. skip for certain response content-types (image/*, video/*), or file extensions if there is no identifying content-type. if cpu load isn't too high. if first N kb look compressible and come in quickly enough after first byte (e.g. within 100ms). always flush after 100ms to prevent stalled real-time connections.
+
 // http.ResponseWriter that writes access log and tracks metrics at end of response.
 type loggingWriter struct {
 	W     http.ResponseWriter // Calls are forwarded.
@@ -54,16 +75,34 @@ type loggingWriter struct {
 	Handler string // Set by router.
 
 	// Set by handlers.
-	Code     int
+	StatusCode int
-	Size     int64
+	Size       int64
-	WriteErr error
+	WriteErr   error
 }
 
 func (w *loggingWriter) Header() http.Header {
 	return w.W.Header()
 }
 
+func (w *loggingWriter) setStatusCode(statusCode int) {
+	if w.StatusCode != 0 {
+		return
+	}
+
+	w.StatusCode = statusCode
+	method := metricHTTPMethod(w.R.Method)
+	proto := "http"
+	if w.R.TLS != nil {
+		proto = "https"
+	}
+	metricRequest.WithLabelValues(w.Handler, proto, method, fmt.Sprintf("%d", w.StatusCode)).Observe(float64(time.Since(w.Start)) / float64(time.Second))
+}
+
 func (w *loggingWriter) Write(buf []byte) (int, error) {
+	if w.Size == 0 {
+		w.setStatusCode(http.StatusOK)
+	}
+
 	n, err := w.W.Write(buf)
 	if n > 0 {
 		w.Size += int64(n)
@@ -75,9 +114,7 @@ func (w *loggingWriter) Write(buf []byte) (int, error) {
 }
 
 func (w *loggingWriter) WriteHeader(statusCode int) {
-	if w.Code == 0 {
+	w.setStatusCode(statusCode)
-		w.Code = statusCode
-	}
 	w.W.WriteHeader(statusCode)
 }
 
@@ -104,7 +141,7 @@ func (w *loggingWriter) Done() {
 	if w.R.TLS != nil {
 		proto = "https"
 	}
-	metricHTTPServer.WithLabelValues(w.Handler, proto, method, fmt.Sprintf("%d", w.Code)).Observe(float64(time.Since(w.Start)) / float64(time.Second))
+	metricResponse.WithLabelValues(w.Handler, proto, method, fmt.Sprintf("%d", w.StatusCode)).Observe(float64(time.Since(w.Start)) / float64(time.Second))
 
 	tlsinfo := "plain"
 	if w.R.TLS != nil {
@@ -114,7 +151,21 @@ func (w *loggingWriter) Done() {
 			tlsinfo = "(other)"
 		}
 	}
-	xlog.WithContext(w.R.Context()).Debugx("http request", w.WriteErr, mlog.Field("httpaccess", ""), mlog.Field("handler", w.Handler), mlog.Field("url", w.R.URL), mlog.Field("host", w.R.Host), mlog.Field("duration", time.Since(w.Start)), mlog.Field("size", w.Size), mlog.Field("statuscode", w.Code), mlog.Field("proto", strings.ToLower(w.R.Proto)), mlog.Field("remoteaddr", w.R.RemoteAddr), mlog.Field("tlsinfo", tlsinfo))
+	xlog.WithContext(w.R.Context()).Debugx("http request", w.WriteErr,
+		mlog.Field("httpaccess", ""),
+		mlog.Field("handler", w.Handler),
+		mlog.Field("method", method),
+		mlog.Field("url", w.R.URL),
+		mlog.Field("host", w.R.Host),
+		mlog.Field("duration", time.Since(w.Start)),
+		mlog.Field("size", w.Size),
+		mlog.Field("statuscode", w.StatusCode),
+		mlog.Field("proto", strings.ToLower(w.R.Proto)),
+		mlog.Field("remoteaddr", w.R.RemoteAddr),
+		mlog.Field("tlsinfo", tlsinfo),
+		mlog.Field("useragent", w.R.Header.Get("User-Agent")),
+		mlog.Field("referrr", w.R.Header.Get("Referrer")),
+	)
 }
 
 // Set some http headers that should prevent potential abuse. Better safe than sorry.
@@ -131,9 +182,10 @@ func safeHeaders(fn http.HandlerFunc) http.HandlerFunc {
 
 // Built-in handlers, e.g. mta-sts and autoconfig.
 type pathHandler struct {
-	Name string // For logging/metrics.
+	Name      string                    // For logging/metrics.
-	Path string // Path to register, like on http.ServeMux.
+	HostMatch func(dom dns.Domain) bool // If not nil, called to see if domain of requests matches. Only called if requested host is a valid domain.
-	Fn   http.HandlerFunc
+	Path      string                    // Path to register, like on http.ServeMux.
+	Handle    http.HandlerFunc
 }
 type serve struct {
 	Kinds        []string // Type of handler and protocol (http/https).
@@ -142,10 +194,10 @@ type serve struct {
 	Webserver    bool          // Whether serving WebHandler. PathHandlers are always evaluated before WebHandlers.
 }
 
-// HandleFunc registers a named handler for a path. If path ends with a slash, it
+// HandleFunc registers a named handler for a path and optional host. If path ends with a slash, it
-// is used as prefix match, otherwise a full path match is required.
+// is used as prefix match, otherwise a full path match is required. If hostOpt is set, only requests to those host are handled by this handler.
-func (s *serve) HandleFunc(name, path string, fn http.HandlerFunc) {
+func (s *serve) HandleFunc(name string, hostMatch func(dns.Domain) bool, path string, fn http.HandlerFunc) {
-	s.PathHandlers = append(s.PathHandlers, pathHandler{name, path, fn})
+	s.PathHandlers = append(s.PathHandlers, pathHandler{name, hostMatch, path, fn})
 }
 
 var (
@@ -180,10 +232,10 @@ func (s *serve) ServeHTTP(xw http.ResponseWriter, r *http.Request) {
 		if r.TLS != nil {
 			proto = "https"
 		}
-		metricHTTPServer.WithLabelValues("(ratelimited)", proto, method, "429").Observe(0)
+		metricRequest.WithLabelValues("(ratelimited)", proto, method, "429").Observe(0)
 		// No logging, that's just noise.
 
-		http.Error(xw, "http 429 - too many auth attempts", http.StatusTooManyRequests)
+		http.Error(xw, "429 - too many auth attempts", http.StatusTooManyRequests)
 		return
 	}
 
@@ -210,15 +262,27 @@ func (s *serve) ServeHTTP(xw http.ResponseWriter, r *http.Request) {
 		r.URL.Path += "/"
 	}
 
+	var dom dns.Domain
+	host := r.Host
+	nhost, _, err := net.SplitHostPort(host)
+	if err == nil {
+		host = nhost
+	}
+	// host could be an IP, some handles may match, not an error.
+	dom, domErr := dns.ParseDomain(host)
+
 	for _, h := range s.PathHandlers {
+		if h.HostMatch != nil && (domErr != nil || !h.HostMatch(dom)) {
+			continue
+		}
 		if r.URL.Path == h.Path || strings.HasSuffix(h.Path, "/") && strings.HasPrefix(r.URL.Path, h.Path) {
 			nw.Handler = h.Name
-			h.Fn(nw, r)
+			h.Handle(nw, r)
 			return
 		}
 	}
-	if s.Webserver {
+	if s.Webserver && domErr == nil {
-		if WebHandle(nw, r) {
+		if WebHandle(nw, r, dom) {
 			return
 		}
 	}
@@ -260,35 +324,35 @@ func Listen() {
 		if l.AccountHTTP.Enabled {
 			port := config.Port(l.AccountHTTP.Port, 80)
 			srv := ensureServe(false, port, "account-http")
-			srv.HandleFunc("account", "/", safeHeaders(accountHandle))
+			srv.HandleFunc("account", nil, "/", safeHeaders(accountHandle))
 		}
 		if l.AccountHTTPS.Enabled {
 			port := config.Port(l.AccountHTTPS.Port, 443)
 			srv := ensureServe(true, port, "account-https")
-			srv.HandleFunc("account", "/", safeHeaders(accountHandle))
+			srv.HandleFunc("account", nil, "/", safeHeaders(accountHandle))
 		}
 
 		if l.AdminHTTP.Enabled {
 			port := config.Port(l.AdminHTTP.Port, 80)
 			srv := ensureServe(false, port, "admin-http")
 			if !l.AccountHTTP.Enabled {
-				srv.HandleFunc("admin", "/", safeHeaders(adminIndex))
+				srv.HandleFunc("admin", nil, "/", safeHeaders(adminIndex))
 			}
-			srv.HandleFunc("admin", "/admin/", safeHeaders(adminHandle))
+			srv.HandleFunc("admin", nil, "/admin/", safeHeaders(adminHandle))
 		}
 		if l.AdminHTTPS.Enabled {
 			port := config.Port(l.AdminHTTPS.Port, 443)
 			srv := ensureServe(true, port, "admin-https")
 			if !l.AccountHTTPS.Enabled {
-				srv.HandleFunc("admin", "/", safeHeaders(adminIndex))
+				srv.HandleFunc("admin", nil, "/", safeHeaders(adminIndex))
 			}
-			srv.HandleFunc("admin", "/admin/", safeHeaders(adminHandle))
+			srv.HandleFunc("admin", nil, "/admin/", safeHeaders(adminHandle))
 		}
 		if l.MetricsHTTP.Enabled {
 			port := config.Port(l.MetricsHTTP.Port, 8010)
 			srv := ensureServe(false, port, "metrics-http")
-			srv.HandleFunc("metrics", "/metrics", safeHeaders(promhttp.Handler().ServeHTTP))
+			srv.HandleFunc("metrics", nil, "/metrics", safeHeaders(promhttp.Handler().ServeHTTP))
-			srv.HandleFunc("metrics", "/", safeHeaders(func(w http.ResponseWriter, r *http.Request) {
+			srv.HandleFunc("metrics", nil, "/", safeHeaders(func(w http.ResponseWriter, r *http.Request) {
 				if r.URL.Path != "/" {
 					http.NotFound(w, r)
 					return
@@ -303,13 +367,21 @@ func Listen() {
 		if l.AutoconfigHTTPS.Enabled {
 			port := config.Port(l.AutoconfigHTTPS.Port, 443)
 			srv := ensureServe(!l.AutoconfigHTTPS.NonTLS, port, "autoconfig-https")
-			srv.HandleFunc("autoconfig", "/mail/config-v1.1.xml", safeHeaders(autoconfHandle(l)))
+			autoconfigMatch := func(dom dns.Domain) bool {
-			srv.HandleFunc("autodiscover", "/autodiscover/autodiscover.xml", safeHeaders(autodiscoverHandle(l)))
+				// todo: may want to check this against the configured domains, could in theory be just a webserver.
+				return strings.HasPrefix(dom.ASCII, "autoconfig.")
+			}
+			srv.HandleFunc("autoconfig", autoconfigMatch, "/mail/config-v1.1.xml", safeHeaders(autoconfHandle(l)))
+			srv.HandleFunc("autodiscover", autoconfigMatch, "/autodiscover/autodiscover.xml", safeHeaders(autodiscoverHandle(l)))
 		}
 		if l.MTASTSHTTPS.Enabled {
 			port := config.Port(l.MTASTSHTTPS.Port, 443)
 			srv := ensureServe(!l.AutoconfigHTTPS.NonTLS, port, "mtasts-https")
-			srv.HandleFunc("mtasts", "/.well-known/mta-sts.txt", safeHeaders(mtastsPolicyHandle))
+			mtastsMatch := func(dom dns.Domain) bool {
+				// todo: may want to check this against the configured domains, could in theory be just a webserver.
+				return strings.HasPrefix(dom.ASCII, "mta-sts.")
+			}
+			srv.HandleFunc("mtasts", mtastsMatch, "/.well-known/mta-sts.txt", safeHeaders(mtastsPolicyHandle))
 		}
 		if l.PprofHTTP.Enabled {
 			// Importing net/http/pprof registers handlers on the default serve mux.
@@ -319,7 +391,7 @@ func Listen() {
 			}
 			srv := &serve{[]string{"pprof-http"}, nil, nil, false}
 			portServe[port] = srv
-			srv.HandleFunc("pprof", "/", http.DefaultServeMux.ServeHTTP)
+			srv.HandleFunc("pprof", nil, "/", http.DefaultServeMux.ServeHTTP)
 		}
 		if l.WebserverHTTP.Enabled {
 			port := config.Port(l.WebserverHTTP.Port, 80)
@@ -381,17 +453,17 @@ func Listen() {
 		}
 
 		for port, srv := range portServe {
+			sort.Slice(srv.PathHandlers, func(i, j int) bool {
+				a := srv.PathHandlers[i].Path
+				b := srv.PathHandlers[j].Path
+				if len(a) == len(b) {
+					// For consistent order.
+					return a < b
+				}
+				// Longest paths first.
+				return len(a) > len(b)
+			})
 			for _, ip := range l.IPs {
-				sort.Slice(srv.PathHandlers, func(i, j int) bool {
-					a := srv.PathHandlers[i].Path
-					b := srv.PathHandlers[j].Path
-					if len(a) == len(b) {
-						// For consistent order.
-						return a < b
-					}
-					// Longest paths first.
-					return len(a) > len(b)
-				})
 				listen1(ip, port, srv.TLSConfig, name, srv.Kinds, srv)
 			}
 		}

http/web_test.go

@@ -0,0 +1,69 @@
+package http
+
+import (
+	"bytes"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/mjl-/mox/dns"
+	"github.com/mjl-/mox/mox-"
+)
+
+func TestServeHTTP(t *testing.T) {
+	os.RemoveAll("../testdata/web/data")
+	mox.ConfigStaticPath = "../testdata/web/mox.conf"
+	mox.ConfigDynamicPath = filepath.Join(filepath.Dir(mox.ConfigStaticPath), "domains.conf")
+	mox.MustLoadConfig()
+
+	srv := &serve{
+		PathHandlers: []pathHandler{
+			{
+				HostMatch: func(dom dns.Domain) bool {
+					return strings.HasPrefix(dom.ASCII, "mta-sts.")
+				},
+				Path: "/.well-known/mta-sts.txt",
+				Handle: func(w http.ResponseWriter, r *http.Request) {
+					w.Write([]byte("mta-sts!"))
+				},
+			},
+		},
+		Webserver: true,
+	}
+
+	test := func(method, target string, expCode int, expContent string, expHeaders map[string]string) {
+		t.Helper()
+
+		req := httptest.NewRequest(method, target, nil)
+		rw := httptest.NewRecorder()
+		rw.Body = &bytes.Buffer{}
+		srv.ServeHTTP(rw, req)
+		resp := rw.Result()
+		if resp.StatusCode != expCode {
+			t.Fatalf("got statuscode %d, expected %d", resp.StatusCode, expCode)
+		}
+		if expContent != "" {
+			s := rw.Body.String()
+			if s != expContent {
+				t.Fatalf("got response data %q, expected %q", s, expContent)
+			}
+		}
+		for k, v := range expHeaders {
+			if xv := resp.Header.Get(k); xv != v {
+				t.Fatalf("got %q for header %q, expected %q", xv, k, v)
+			}
+		}
+	}
+
+	test("GET", "http://mta-sts.mox.example/.well-known/mta-sts.txt", http.StatusOK, "mta-sts!", nil)
+	test("GET", "http://mox.example/.well-known/mta-sts.txt", http.StatusNotFound, "", nil) // mta-sts endpoint not in this domain.
+	test("GET", "http://mta-sts.mox.example/static/", http.StatusNotFound, "", nil)         // static not served on this domain.
+	test("GET", "http://mta-sts.mox.example/other", http.StatusNotFound, "", nil)
+	test("GET", "http://mox.example/static/", http.StatusOK, "html\n", map[string]string{"X-Test": "mox"}) // index.html is served
+	test("GET", "http://mox.example/static/index.html", http.StatusOK, "html\n", map[string]string{"X-Test": "mox"})
+	test("GET", "http://mox.example/static/dir/", http.StatusOK, "", map[string]string{"X-Test": "mox"}) // Dir listing.
+	test("GET", "http://mox.example/other", http.StatusNotFound, "", nil)
+}

http/webserver.go

@@ -5,7 +5,6 @@ import (
 	htmltemplate "html/template"
 	"io"
 	golog "log"
-	"net"
 	"net/http"
 	"net/http/httputil"
 	"os"
@@ -21,32 +20,28 @@ import (
 	"github.com/mjl-/mox/moxio"
 )
 
-// todo: automatic gzip on responses, if client supports it, and if content looks compressible.
-
 // WebHandle serves an HTTP request by going through the list of WebHandlers,
 // check if there is a domain+path match, and running the handler if so.
 // WebHandle runs after the built-in handlers for mta-sts, autoconfig, etc.
 // If no handler matched, false is returned.
 // WebHandle sets w.Name to that of the matching handler.
-func WebHandle(w *loggingWriter, r *http.Request) (handled bool) {
+func WebHandle(w *loggingWriter, r *http.Request, host dns.Domain) (handled bool) {
-	log := func() *mlog.Log {
+	redirects, handlers := mox.Conf.WebServer()
-		return xlog.WithContext(r.Context())
-	}
 
-	host, _, err := net.SplitHostPort(r.Host)
+	for from, to := range redirects {
-	if err != nil {
+		if host != from {
-		// Common, there often is not port.
+			continue
-		host = r.Host
+		}
-	}
+		u := r.URL
-	dom, err := dns.ParseDomain(host)
+		u.Scheme = "https"
-	if err != nil {
+		u.Host = to.Name()
-		log().Debugx("parsing http request domain", err, mlog.Field("host", host))
+		w.Handler = "(domainredirect)"
-		http.NotFound(w, r)
+		http.Redirect(w, r, u.String(), http.StatusPermanentRedirect)
 		return true
 	}
 
-	for _, h := range mox.Conf.WebHandlers() {
+	for _, h := range handlers {
-		if h.DNSDomain != dom {
+		if host != h.DNSDomain {
 			continue
 		}
 		loc := h.Path.FindStringIndex(r.URL.Path)
@@ -60,7 +55,7 @@ func WebHandle(w *loggingWriter, r *http.Request) (handled bool) {
 		if r.TLS == nil && !h.DontRedirectPlainHTTP {
 			u := *r.URL
 			u.Scheme = "https"
-			u.Host = host
+			u.Host = h.DNSDomain.Name()
 			w.Handler = h.Name
 			http.Redirect(w, r, u.String(), http.StatusPermanentRedirect)
 			return true
@@ -138,7 +133,7 @@ func HandleStatic(h *config.WebStatic, w http.ResponseWriter, r *http.Request) (
 		if cid <= 0 {
 			return ""
 		}
-		return " (requestid " + mox.ReceivedID(cid) + ")"
+		return " (id " + mox.ReceivedID(cid) + ")"
 	}
 
 	if r.Method != "GET" && r.Method != "HEAD" {
@@ -166,6 +161,15 @@ func HandleStatic(h *config.WebStatic, w http.ResponseWriter, r *http.Request) (
 		fspath = filepath.Join(h.Root, r.URL.Path)
 	}
 
+	serveFile := func(name string, mtime time.Time, content *os.File) {
+		// ServeContent only sets a content-type if not already present in the response headers.
+		hdr := w.Header()
+		for k, v := range h.ResponseHeaders {
+			hdr.Add(k, v)
+		}
+		http.ServeContent(w, r, name, mtime, content)
+	}
+
 	f, err := os.Open(fspath)
 	if err != nil {
 		if os.IsNotExist(err) {
@@ -176,6 +180,22 @@ func HandleStatic(h *config.WebStatic, w http.ResponseWriter, r *http.Request) (
 			http.NotFound(w, r)
 			return true
 		} else if os.IsPermission(err) {
+			// If we tried opening a directory, we may not have permission to read it, but
+			// still access files inside it (execute bit), such as index.html. So try to serve it.
+			index, err := os.Open(filepath.Join(fspath, "index.html"))
+			if err == nil {
+				defer index.Close()
+				var ifi os.FileInfo
+				ifi, err = index.Stat()
+				if err != nil {
+					log().Errorx("stat index.html in directory we cannot list", err, mlog.Field("url", r.URL), mlog.Field("fspath", fspath))
+					http.Error(w, "500 - internal server error"+recvid(), http.StatusInternalServerError)
+					return true
+				}
+				w.Header().Set("Content-Type", "text/html; charset=utf-8")
+				serveFile("index.html", ifi.ModTime(), index)
+				return true
+			}
 			http.Error(w, "403 - permission denied", http.StatusForbidden)
 			return true
 		}
@@ -191,23 +211,21 @@ func HandleStatic(h *config.WebStatic, w http.ResponseWriter, r *http.Request) (
 		http.Error(w, "500 - internal server error"+recvid(), http.StatusInternalServerError)
 		return true
 	}
+	// Redirect if the local path is a directory.
 	if fi.IsDir() && !strings.HasSuffix(r.URL.Path, "/") {
 		http.Redirect(w, r, r.URL.Path+"/", http.StatusTemporaryRedirect)
 		return true
 	}
 
-	serveFile := func(name string, mtime time.Time, content *os.File) {
-		// ServeContent only sets a content-type if not already present in the response headers.
-		hdr := w.Header()
-		for k, v := range h.ResponseHeaders {
-			hdr.Add(k, v)
-		}
-		http.ServeContent(w, r, name, mtime, content)
-	}
-
 	if fi.IsDir() {
 		index, err := os.Open(filepath.Join(fspath, "index.html"))
-		if err != nil && os.IsPermission(err) || err != nil && os.IsNotExist(err) && !h.ListFiles {
+		if err != nil && os.IsPermission(err) {
+			http.Error(w, "403 - permission denied", http.StatusForbidden)
+			return true
+		} else if err != nil && os.IsNotExist(err) && !h.ListFiles {
+			if h.ContinueNotFound {
+				return false
+			}
 			http.Error(w, "403 - permission denied", http.StatusForbidden)
 			return true
 		} else if err == nil {
@@ -216,7 +234,7 @@ func HandleStatic(h *config.WebStatic, w http.ResponseWriter, r *http.Request) (
 			ifi, err = index.Stat()
 			if err == nil {
 				w.Header().Set("Content-Type", "text/html; charset=utf-8")
-				serveFile(filepath.Base(fspath), ifi.ModTime(), index)
+				serveFile("index.html", ifi.ModTime(), index)
 				return true
 			}
 		}
@@ -243,11 +261,13 @@ func HandleStatic(h *config.WebStatic, w http.ResponseWriter, r *http.Request) (
 				mb := float64(e.Size()) / (1024 * 1024)
 				var size string
 				var sizepad bool
-				if mb >= 10 {
+				if !e.IsDir() {
-					size = fmt.Sprintf("%d", int64(mb))
+					if mb >= 10 {
-					sizepad = true
+						size = fmt.Sprintf("%d", int64(mb))
-				} else {
+						sizepad = true
-					size = fmt.Sprintf("%.2f", mb)
+					} else {
+						size = fmt.Sprintf("%.2f", mb)
+					}
 				}
 				const dateTime = "2006-01-02 15:04:05" // time.DateTime, but only since go1.20.
 				modified := e.ModTime().UTC().Format(dateTime)
@@ -309,12 +329,12 @@ func HandleRedirect(h *config.WebRedirect, w http.ResponseWriter, r *http.Reques
 		u.ForceQuery = h.URL.ForceQuery
 		u.RawQuery = h.URL.RawQuery
 		u.Fragment = h.URL.Fragment
-	}
+		if r.URL.RawQuery != "" {
-	if r.URL.RawQuery != "" {
+			if u.RawQuery != "" {
-		if u.RawQuery != "" {
+				u.RawQuery += "&"
-			u.RawQuery += "&"
+			}
+			u.RawQuery += r.URL.RawQuery
 		}
-		u.RawQuery += r.URL.RawQuery
 	}
 	u.Path = dstpath
 	code := http.StatusPermanentRedirect
@@ -336,7 +356,7 @@ func HandleForward(h *config.WebForward, w http.ResponseWriter, r *http.Request,
 		if cid <= 0 {
 			return ""
 		}
-		return " (requestid " + mox.ReceivedID(cid) + ")"
+		return " (id " + mox.ReceivedID(cid) + ")"
 	}
 
 	xr := *r
@@ -351,8 +371,7 @@ func HandleForward(h *config.WebForward, w http.ResponseWriter, r *http.Request,
 	// Remove any forwarded headers passed in by client.
 	hdr := http.Header{}
 	for k, vl := range r.Header {
-		switch k {
+		if k == "Forwarded" || k == "X-Forwarded" || strings.HasPrefix(k, "X-Forwarded-") {
-		case "Forwarded", "X-Forwarded-For", "X-Forwarded-Host", "X-Forwarded-Proto":
 			continue
 		}
 		hdr[k] = vl
@@ -374,7 +393,11 @@ func HandleForward(h *config.WebForward, w http.ResponseWriter, r *http.Request,
 	proxy.ErrorLog = golog.New(mlog.ErrWriter(mlog.New("net/http/httputil").WithContext(r.Context()), mlog.LevelDebug, "reverseproxy error"), "", 0)
 	proxy.ErrorHandler = func(w http.ResponseWriter, r *http.Request, err error) {
 		log().Errorx("forwarding request to backend webserver", err, mlog.Field("url", r.URL))
-		http.Error(w, "502 - bad gateway"+recvid(), http.StatusBadGateway)
+		if os.IsTimeout(err) {
+			http.Error(w, "504 - gateway timeout"+recvid(), http.StatusGatewayTimeout)
+		} else {
+			http.Error(w, "502 - bad gateway"+recvid(), http.StatusBadGateway)
+		}
 	}
 	whdr := w.Header()
 	for k, v := range h.ResponseHeaders {

http/webserver_test.go

@@ -0,0 +1,117 @@
+package http
+
+import (
+	"bytes"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/mjl-/mox/mox-"
+)
+
+func TestWebserver(t *testing.T) {
+	os.RemoveAll("../testdata/webserver/data")
+	mox.ConfigStaticPath = "../testdata/webserver/mox.conf"
+	mox.ConfigDynamicPath = filepath.Join(filepath.Dir(mox.ConfigStaticPath), "domains.conf")
+	mox.MustLoadConfig()
+
+	srv := &serve{Webserver: true}
+
+	test := func(method, target string, reqhdrs map[string]string, expCode int, expContent string, expHeaders map[string]string) {
+		t.Helper()
+
+		req := httptest.NewRequest(method, target, nil)
+		for k, v := range reqhdrs {
+			req.Header.Add(k, v)
+		}
+		rw := httptest.NewRecorder()
+		rw.Body = &bytes.Buffer{}
+		srv.ServeHTTP(rw, req)
+		resp := rw.Result()
+		if resp.StatusCode != expCode {
+			t.Fatalf("got statuscode %d, expected %d", resp.StatusCode, expCode)
+		}
+		if expContent != "" {
+			s := rw.Body.String()
+			if s != expContent {
+				t.Fatalf("got response data %q, expected %q", s, expContent)
+			}
+		}
+		for k, v := range expHeaders {
+			if xv := resp.Header.Get(k); xv != v {
+				t.Fatalf("got %q for header %q, expected %q", xv, k, v)
+			}
+		}
+	}
+
+	test("GET", "http://redir.mox.example", nil, http.StatusPermanentRedirect, "", map[string]string{"Location": "https://mox.example/"})
+
+	test("GET", "http://mox.example/static/", nil, http.StatusOK, "", map[string]string{"X-Test": "mox"})                              // index.html
+	test("GET", "http://mox.example/static/dir/", nil, http.StatusOK, "", map[string]string{"X-Test": "mox"})                          // listing
+	test("GET", "http://mox.example/static/dir", nil, http.StatusTemporaryRedirect, "", map[string]string{"Location": "/static/dir/"}) // redirect to dir
+	test("GET", "http://mox.example/static/bogus", nil, http.StatusNotFound, "", nil)
+
+	test("GET", "http://mox.example/nolist/", nil, http.StatusOK, "", nil)            // index.html
+	test("GET", "http://mox.example/nolist/dir/", nil, http.StatusForbidden, "", nil) // no listing
+
+	test("GET", "http://mox.example/tls/", nil, http.StatusPermanentRedirect, "", map[string]string{"Location": "https://mox.example/tls/"}) // redirect to tls
+
+	test("GET", "http://mox.example/baseurl/x?y=2", nil, http.StatusPermanentRedirect, "", map[string]string{"Location": "https://tls.mox.example/baseurl/x?q=1&y=2#fragment"})
+	test("GET", "http://mox.example/pathonly/old/x?q=2", nil, http.StatusTemporaryRedirect, "", map[string]string{"Location": "http://mox.example/pathonly/new/x?q=2"})
+	test("GET", "http://mox.example/baseurlpath/old/x?y=2", nil, http.StatusPermanentRedirect, "", map[string]string{"Location": "//other.mox.example/baseurlpath/new/x?q=1&y=2#fragment"})
+
+	test("GET", "http://mox.example/strip/x", nil, http.StatusBadGateway, "", nil)   // no server yet
+	test("GET", "http://mox.example/nostrip/x", nil, http.StatusBadGateway, "", nil) // no server yet
+
+	badForwarded := map[string]string{
+		"Forwarded":         "bad",
+		"X-Forwarded-For":   "bad",
+		"X-Forwarded-Proto": "bad",
+		"X-Forwarded-Host":  "bad",
+		"X-Forwarded-Ext":   "bad",
+	}
+
+	// Server that echoes path, and forwarded request headers.
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		for k, v := range badForwarded {
+			if r.Header.Get(k) == v {
+				w.WriteHeader(http.StatusInternalServerError)
+				return
+			}
+		}
+
+		for k, vl := range r.Header {
+			if k == "Forwarded" || k == "X-Forwarded" || strings.HasPrefix(k, "X-Forwarded-") {
+				w.Header()[k] = vl
+			}
+		}
+		w.Write([]byte(r.URL.Path))
+	}))
+	defer server.Close()
+
+	serverURL, err := url.Parse(server.URL)
+	if err != nil {
+		t.Fatalf("parsing url: %v", err)
+	}
+	serverURL.Path = "/a"
+
+	// warning: it is not normally allowed to access the dynamic config without lock. don't propagate accesses like this!
+	mox.Conf.Dynamic.WebHandlers[len(mox.Conf.Dynamic.WebHandlers)-2].WebForward.TargetURL = serverURL
+	mox.Conf.Dynamic.WebHandlers[len(mox.Conf.Dynamic.WebHandlers)-1].WebForward.TargetURL = serverURL
+
+	test("GET", "http://mox.example/strip/x", badForwarded, http.StatusOK, "/a/x", map[string]string{
+		"X-Test":            "mox",
+		"X-Forwarded-For":   "192.0.2.1", // IP is hardcoded in Go's src/net/http/httptest/httptest.go
+		"X-Forwarded-Proto": "http",
+		"X-Forwarded-Host":  "mox.example",
+		"X-Forwarded-Ext":   "",
+	})
+	test("GET", "http://mox.example/nostrip/x", map[string]string{"X-OK": "ok"}, http.StatusOK, "/a/nostrip/x", map[string]string{"X-Test": "mox"})
+
+	test("GET", "http://mox.example/bogus", nil, http.StatusNotFound, "", nil)         // path not registered.
+	test("GET", "http://bogus.mox.example/static/", nil, http.StatusNotFound, "", nil) // domain not registered.
+}

main.go

@@ -766,14 +766,20 @@ var examples = []struct {
 	{
 		"webhandlers",
 		func() string {
-			const webhandlers = `# Snippet of domains.conf to configure WebHandlers.
+			const webhandlers = `# Snippet of domains.conf to configure WebDomainRedirects and WebHandlers.
+
+# Redirect all requests for mox.example to https://www.mox.example.
+WebDomainRedirects:
+	mox.example: www.mox.example
+
+# Each request is matched against these handlers until one matches and serves it.
 WebHandlers:
 	-
 		# The name of the handler, used in logging and metrics.
 		LogName: staticmjl
 		# With ACME configured, each configured domain will automatically get a TLS
 		# certificate on first request.
-		Domain: mox.example
+		Domain: www.mox.example
 		PathRegexp: ^/who/mjl/
 		WebStatic:
 			StripPrefix: /who/mjl
@@ -786,7 +792,7 @@ WebHandlers:
 				X-Mox: hi
 	-
 		LogName: redir
-		Domain: mox.example
+		Domain: www.mox.example
 		PathRegexp: ^/redir/a/b/c
 		# Don't redirect from plain HTTP to HTTPS.
 		DontRedirectPlainHTTP: true
@@ -799,7 +805,7 @@ WebHandlers:
 			StatusCode: 307
 	-
 		LogName: oldnew
-		Domain: mox.example
+		Domain: www.mox.example
 		PathRegexp: ^/old/
 		WebRedirect:
 			# Replace path, leaving rest of URL intact.
@@ -807,7 +813,7 @@ WebHandlers:
 			ReplacePath: /new/$1
 	-
 		LogName: app
-		Domain: mox.example
+		Domain: www.mox.example
 		PathRegexp: ^/app/
 		WebForward:
 			# Strip the path matched by PathRegexp before forwarding the request. So original
@@ -826,7 +832,8 @@ WebHandlers:
 			// Parse just so we know we have the syntax right.
 			// todo: ideally we would have a complete config file and parse it fully.
 			var conf struct {
-				WebHandlers []config.WebHandler
+				WebDomainRedirects map[string]string
+				WebHandlers        []config.WebHandler
 			}
 			err := sconf.Parse(strings.NewReader(webhandlers), &conf)
 			xcheckf(err, "parsing webhandlers example")

mox-/admin.go

@@ -401,6 +401,31 @@ func DomainRemove(ctx context.Context, domain dns.Domain) (rerr error) {
 	return nil
 }
 
+func WebserverConfigSet(ctx context.Context, domainRedirects map[string]string, webhandlers []config.WebHandler) (rerr error) {
+	log := xlog.WithContext(ctx)
+	defer func() {
+		if rerr != nil {
+			log.Errorx("saving webserver config", rerr)
+		}
+	}()
+
+	Conf.dynamicMutex.Lock()
+	defer Conf.dynamicMutex.Unlock()
+
+	// Compose new config without modifying existing data structures. If we fail, we
+	// leave no trace.
+	nc := Conf.Dynamic
+	nc.WebDomainRedirects = domainRedirects
+	nc.WebHandlers = webhandlers
+
+	if err := writeDynamic(ctx, log, nc); err != nil {
+		return fmt.Errorf("writing domains.conf: %v", err)
+	}
+
+	log.Info("webserver config saved")
+	return nil
+}
+
 // todo: find a way to automatically create the dns records as it would greatly simplify setting up email for a domain. we could also dynamically make changes, e.g. providing grace periods after disabling a dkim key, only automatically removing the dkim dns key after a few days. but this requires some kind of api and authentication to the dns server. there doesn't appear to be a single commonly used api for dns management. each of the numerous cloud providers have their own APIs and rather large SKDs to use them. we don't want to link all of them in.
 
 // DomainRecords returns text lines describing DNS records required for configuring

mox-/config.go

@@ -198,11 +198,12 @@ func (c *Config) AccountDestination(addr string) (accDests AccountDestination, o
 	return
 }
 
-func (c *Config) WebHandlers() (l []config.WebHandler) {
+func (c *Config) WebServer() (r map[dns.Domain]dns.Domain, l []config.WebHandler) {
 	c.withDynamicLock(func() {
+		r = c.Dynamic.WebDNSDomainRedirects
 		l = c.Dynamic.WebHandlers
 	})
-	return l
+	return r, l
 }
 
 func (c *Config) allowACMEHosts() {
@@ -245,6 +246,9 @@ func (c *Config) allowACMEHosts() {
 		}
 
 		if l.WebserverHTTPS.Enabled {
+			for from := range c.Dynamic.WebDNSDomainRedirects {
+				hostnames[from] = struct{}{}
+			}
 			for _, wh := range c.Dynamic.WebHandlers {
 				hostnames[wh.DNSDomain] = struct{}{}
 			}
@@ -678,11 +682,13 @@ func prepareDynamicConfig(ctx context.Context, dynamicPath string, static config
 	}
 	checkMailboxNormf(static.Postmaster.Mailbox, "postmaster mailbox")
 
-	var haveSTSListener bool
+	var haveSTSListener, haveWebserverListener bool
 	for _, l := range static.Listeners {
 		if l.MTASTSHTTPS.Enabled {
 			haveSTSListener = true
-			break
+		}
+		if l.WebserverHTTP.Enabled || l.WebserverHTTPS.Enabled {
+			haveWebserverListener = true
 		}
 	}
 
@@ -991,6 +997,29 @@ func prepareDynamicConfig(ctx context.Context, dynamicPath string, static config
 	}
 
 	// Check webserver configs.
+	if (len(c.WebDomainRedirects) > 0 || len(c.WebHandlers) > 0) && !haveWebserverListener {
+		addErrorf("WebDomainRedirects or WebHandlers configured but no listener with WebserverHTTP or WebserverHTTPS enabled")
+	}
+
+	c.WebDNSDomainRedirects = map[dns.Domain]dns.Domain{}
+	for from, to := range c.WebDomainRedirects {
+		fromdom, err := dns.ParseDomain(from)
+		if err != nil {
+			addErrorf("parsing domain for redirect %s: %v", from, err)
+		}
+		todom, err := dns.ParseDomain(to)
+		if err != nil {
+			addErrorf("parsing domain for redirect %s: %v", to, err)
+		} else if fromdom == todom {
+			addErrorf("will not redirect domain %s to itself", todom)
+		}
+		var zerodom dns.Domain
+		if _, ok := c.WebDNSDomainRedirects[fromdom]; ok && fromdom != zerodom {
+			addErrorf("duplicate redirect domain %s", from)
+		}
+		c.WebDNSDomainRedirects[fromdom] = todom
+	}
+
 	for i := range c.WebHandlers {
 		wh := &c.WebHandlers[i]
 

testdata/web/domains.conf

@@ -0,0 +1,23 @@
+Domains:
+	mox.example:
+		LocalpartCaseSensitive: false
+Accounts:
+	mjl:
+		Domain: mox.example
+		Destinations:
+			mjl: nil
+WebDomainRedirects:
+	redir.mox.example: mox.example
+WebHandlers:
+	-
+		LogName: static
+		Domain: mox.example
+		PathRegexp: ^/static/
+		DontRedirectPlainHTTP: true
+		WebStatic:
+			StripPrefix: /static/
+			# This is run from the http package.
+			Root: ../testdata/web/static
+			ListFiles: true
+			ResponseHeaders:
+				X-Test: mox

testdata/web/mox.conf

@@ -0,0 +1,13 @@
+DataDir: data
+User: 1000
+LogLevel: trace
+Hostname: mox.example
+Listeners:
+	local:
+		IPs:
+			- 0.0.0.0
+		WebserverHTTP:
+			Enabled: true
+Postmaster:
+	Account: mjl
+	Mailbox: postmaster

testdata/web/static/dir/hi.txt

@@ -0,0 +1 @@
+hi

testdata/web/static/index.html

@@ -0,0 +1 @@
+html

testdata/webserver/domains.conf

@@ -0,0 +1,85 @@
+Domains:
+	mox.example:
+		LocalpartCaseSensitive: false
+Accounts:
+	mjl:
+		Domain: mox.example
+		Destinations:
+			mjl: nil
+WebDomainRedirects:
+	redir.mox.example: mox.example
+WebHandlers:
+	-
+		LogName: static
+		Domain: mox.example
+		PathRegexp: ^/static/
+		DontRedirectPlainHTTP: true
+		WebStatic:
+			# This is run from the http package.
+			Root: ../testdata/web
+			ListFiles: true
+			ResponseHeaders:
+				X-Test: mox
+	-
+		LogName: nolist
+		Domain: mox.example
+		PathRegexp: ^/nolist/
+		DontRedirectPlainHTTP: true
+		WebStatic:
+			StripPrefix: /nolist/
+			# This is run from the http package.
+			Root: ../testdata/web/static
+	-
+		LogName: httpsredir
+		Domain: mox.example
+		PathRegexp: ^/tls/
+		WebStatic:
+			# This is run from the http package.
+			Root: ../testdata/web/static
+	-
+		LogName: baseurlonly
+		Domain: mox.example
+		PathRegexp: ^/baseurl/
+		DontRedirectPlainHTTP: true
+		WebRedirect:
+			BaseURL: https://tls.mox.example?q=1#fragment
+	-
+		LogName: pathonly
+		Domain: mox.example
+		PathRegexp: ^/pathonly/
+		DontRedirectPlainHTTP: true
+		WebRedirect:
+			OrigPathRegexp: ^/pathonly/old/(.*)$
+			ReplacePath: /pathonly/new/$1
+			StatusCode: 307
+	-
+		LogName: baseurlpath
+		Domain: mox.example
+		PathRegexp: ^/baseurlpath/
+		DontRedirectPlainHTTP: true
+		WebRedirect:
+			BaseURL: //other.mox.example?q=1#fragment
+			OrigPathRegexp: ^/baseurlpath/old/(.*)$
+			ReplacePath: /baseurlpath/new/$1
+	# test code depends on these last two webhandlers being here.
+	-
+		LogName: strippath
+		Domain: mox.example
+		PathRegexp: ^/strip/
+		DontRedirectPlainHTTP: true
+		WebForward:
+			StripPath: true
+			# replaced while testing
+			URL: http://127.0.0.1:1/a
+			ResponseHeaders:
+				X-Test: mox
+	-
+		LogName: nostrippath
+		Domain: mox.example
+		PathRegexp: ^/nostrip/
+		DontRedirectPlainHTTP: true
+		WebForward:
+			# replaced while testing
+			URL: http://127.0.0.1:1/a
+			ResponseHeaders:
+				X-Test: mox

testdata/webserver/mox.conf

@@ -0,0 +1,13 @@
+DataDir: data
+User: 1000
+LogLevel: trace
+Hostname: mox.example
+Listeners:
+	local:
+		IPs:
+			- 0.0.0.0
+		WebserverHTTP:
+			Enabled: true
+Postmaster:
+	Account: mjl
+	Mailbox: postmaster

testdata/webserver/static/dir/hi.txt

@@ -0,0 +1 @@
+hi

testdata/webserver/static/index.html

@@ -0,0 +1 @@
+html