We now store the parsed site/server block keys with the server block,
rather than parsing the addresses every time we read them.
Also detect conflicting schemes, i.e. TLS and non-TLS cannot be served
from the same server (natively -- modules could be built for it).
Also do not add site subroutes (subroutes generated specifically from
site blocks in the Caddyfile) that are empty.
Thus far the fuzzers have found a few crashers in the Caddyfile parser. However, the fuzzer have been stuck at import glob expansion after import glob expansion, which aren't reproducible.
Certificate selection used to be a module, but this seems unnecessary,
especially since the built-in CustomSelectionPolicy allows quite complex
selection logic on a number of fields in certs. If we need to extend
that logic, we can, but I don't think there are SO many possibilities
that we need modules.
This update also allows certificate selection to choose between multiple
matching certs based on client compatibility and makes a number of other
improvements in the default cert selection logic, both here and in the
latest CertMagic.
The hardest part of this was the conn policy consolidation logic
(Caddyfile only, of course). We have to merge connection policies that
we can easily combine, because if two certs are manually loaded in a
Caddyfile site block, that produces two connection policies, and each
cert is tagged with a different tag, meaning only the first would ever
be selected. So given the same matchers, we can merge the two, but this
required improving the Tag selection logic to support multiple tags to
choose from, hence "tags" changed to "any_tag" or "all_tags" (but we
use any_tag in our Caddyfile logic).
Combining conn policies with conflicting settings is impossible, so
that should return an error if two policies with the exact same matchers
have non-empty settings that are not the same (the one exception being
any_tag which we can merge because the logic for them is to OR them).
It was a bit complicated. It seems to work in numerous tests I've
conducted, but we'll see how it pans out in the release candidates.
If a site block has a key like "http://localhost:2016", then the log for
that site must be mapped to "localhost:2016" and not just "localhost"
because "localhost:2016" will be the value of the Host header of requests.
But a key like "localhost:80" does not include the port since the Host
header will not include ":80" because it is a standard port.
Fixes https://caddy.community/t/v2-common-log-format-not-working/7352?u=matt
See https://caddy.community/t/v2-match-any-path-but-files/7326/8?u=matt
If rewrites (or redirects, for that matter) match on file existence,
the file matcher would need to know the root of the site.
Making this change implies that root directives that depend on rewritten
URIs will not work as expected. However, I think this is very uncommon,
and am not sure I have ever seen that. Usually, dynamic roots are based
on host, not paths or query strings.
I suspect that rewrites based on file existence will be more common than
roots based on rewritten URIs, so I am moving root to be the first in
the list.
Users can always override this ordering with the 'order' global option.
* admin: Refactor /load endpoint out of caddy package
This eliminates the caddy package's dependency on the caddyconfig
package, which helps prevent import cycles.
* v2: adapter: register config adapters as Caddy modules
* v2: adapter: simplify adapter registration as adapters and modules
* v2: adapter: let RegisterAdapter be in charge of registering adapters as modules
* v2: adapter: remove underscrores placeholders
* v2: adapter: explicitly ignore the error of writing response of writing warnings back to client
* Implicitly wrap config adapters as modules
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* rewrite: strip_prefix, strip_suffix, uri_replace -> uri (closes#3140)
* Add period, to satisfy @whitestrake :) and my own OCD
* Restore implied / prefix
It's hard to say whether this was actually a bug, but the linked issue
shows why the old behavior was confusing. Basically, we infer that a
rewrite handler is supposed to act as an internal redirect, which likely
means it will no longer match the matcher(s) it did before the rewrite.
So if the rewrite directive shares a matcher with any adjacent route or
directive, it can be confusing/misleading if we consolidate the rewrite
into the same route as the next handler, which shouldn't (probably) match
after the rewrite is complete.
This is kiiiind of a hacky workaround to a quirky problem.
For edge cases like these, it is probably "cleaner" to just use handle
blocks instead, to group handlers under the same matcher, nginx-style.
* added sni tests
* set the default sni when there is no host to match
* removed invalid sni test. Disabled tests that rely on host headers.
* readded SNI tests. Added logging of config load times
* add integration tests
* removed SNI test
* remove integration test condition
* minor edit
* fix sni when using static certificates
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* pki: Initial commit of PKI app (WIP) (see #2502 and #3021)
* pki: Ability to use root/intermediates, and sign with root
* pki: Fix benign misnamings left over from copy+paste
* pki: Only install root if not already trusted
* Make HTTPS port the default; all names use auto-HTTPS; bug fixes
* Fix build - what happened to our CI tests??
* Fix go.mod
This is a breaking change primarily in two areas:
- Storage paths for certificates have changed
- Slight changes to JSON config parameters
Huge improvements in this commit, to be detailed more in
the release notes.
The upcoming PKI app will be powered by Smallstep libraries.
* remove the certificate tag tracking from global state
* refactored helper state, added log counter
* moved state initialisation close to where it is used.
* added helper state comment
Previously the formatter did not include support for
blocks inside other blocks. Hence the formatter could
not indent some files properly. This fixes it.
Fixes#3104
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
This takes the config file as input and formats it.
Prints the result to stdout. Can write changes to
file if `--write` flag is passed.
Fixes#3020
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
Small expansion to the work done in https://github.com/caddyserver/caddy/pull/2963 which simply calls `os.ExpandEnv` so env vars like `{$URL}` where `$URL=$SCHEME://$HOST:$PORT` (contrived) get the expanded $SCHEME, $HOST, and $PORT variables included
* httpcaddyfile: Begin implementing log directive, and debug mode
For now, debug mode just sets the log level for all logs to DEBUG
(unless a level is specified explicitly).
* httpcaddyfile: Finish 'log' directive
Also rename StringEncoder -> SingleFieldEncoder
* Fix minor bug in replacer (when vals are empty)
* caddytls: Add CipherSuiteName and ProtocolName functions
The cipher_suites.go file is derived from a commit to the Go master
branch that's slated for Go 1.14. Once Go 1.14 is released, this file
can be removed.
* caddyhttp: Use commonLogEmptyValue in common_log replacer
* caddyhttp: Add TLS placeholders
* caddytls: update unsupportedProtocols
Don't export unsupportedProtocols and update its godoc to mention that
it's used for logging only.
* caddyhttp: simplify getRegTLSReplacement signature
getRegTLSReplacement should receive a string instead of a pointer.
* caddyhttp: Remove http.request.tls.client.cert replacer
The previous behavior of printing the raw certificate bytes was ported
from Caddy 1, but the usefulness of that approach is suspect. Remove
the client cert replacer from v2 until a use case is presented.
* caddyhttp: Use tls.CipherSuiteName from Go 1.14
Remove ported version of CipherSuiteName in the process.
See end of issue #3004. Loading the same certificate file multiple times
with different tags will result in it being de-duplicated in the in-
memory cache, because of course they all have the same bytes. This
meant that any certs of the same filename loaded with different tags
would be overwritten by the next certificate of the same filename, and
any conn policies looking for the tags of the previous ones would never
find them, causing connections to fail.
So, now we remember cert filenames and their tags, instead of loading
them multiple times and overwriting previous ones.
A user crafting their own JSON might make this error too... maybe we
won't see it happen. But if it does, one possibility is, when loading
a duplicate cert, instead of discarding it completely, merge the tag
list into the one that's already stored in the cache, then discard.
Configuration via the Caddyfile requires use of env variables, but
an upstream issue is currently blocking that:
https://github.com/go-acme/lego/issues/1054
Providers will need to be retrofitted upstream in order to support env
var configuration.
If user provides their own certs or makes any hostname-specific TLS
connection policy, it means that no TLS connection would be served for
any other hostnames, even though you'd expect that TLS is enabled for
them, too. So now we append a catch-all conn policy if none exist, which
allows all ClientHellos to be matched and served.
We also fix the consolidation of automation policies, which previously
gobbled up automation policies without hosts in favor of automation
policies with hosts. Instead of a host-specific policy eating up an
identical catch-all policy, the catch-all policy eats up the identical
host-specific policy, ensuring that the policy is applied to all hosts
which need it.
See also:
https://caddy.community/t/v2-automatic-https-certificate-errors/6847/9?u=matt
This ensure that if there are multiple certs that match a particular
ServerName or other parameter, then specifically the one the user
provided in the Caddyfile will be used.
Before, listener ports could be wrong because ParseAddress doesn't know
about the user-configured HTTP/HTTPS ports, instead hard-coding port 80
or 443, which could be wrong if the user changed them to something else.
Now we defer port and scheme validation/inference to a later part of
building the output JSON.
Using rewrite is like saying, "I accept this request, but I just need
to act on it as if it came in differently."
Whereas redir implies more of, "I reject this request, send it to me
differently, then I will process it."
Makes sense for it to come before rewrites. This can always be changed
using the 'order' global option if needed.
The fix that was initially put forth in #2971 was good, but only for
up to one layer of nesting. The real problem was that we forgot to
increment nesting when already inside a block if we saw another open
curly brace that opens another block (dispenser.go L157-158).
The new 'handle' directive allows HTTP Caddyfiles to be designed more
like nginx location blocks if the user prefers. Inside a handle block,
directives are still ordered just like they are outside of them, but
handler blocks at a given level of nesting are mutually exclusive.
This work benefitted from some refactoring and cleanup.
This allows individual directives to be ordered relative to others,
where order matters (for example HTTP handlers). Will primarily be
useful when developing new directives, so you don't have to modify the
Caddy source code. Can also be useful if you prefer that redir comes
before rewrite, for example. Note that these are global options. The
route directive can be used to give a specific order to a specific group
of HTTP handler directives.
In the v1 Caddyfile, only the first matching site definition would be
used, so setting these `Terminal: true` ensures that only the first
matching one is used in v2, too.
We also have to sort by key specificity... Caddy 1 had a special data
structure for selecting the most specific site definition, but we don't
have that structure in v2, so we need to sort by length (of host and
path, separately). For blocks where more than one key is present, we
choose the longest host and path (independently, need not be from same
key) by which to sort.
When we append a token to the new dispenser, we need to consume it in the parent, too; otherwise it gets scanned twice, which in this case messed up the nesting count which got decremented once too many times.
* fix replacing variables on imported files
* refactored replaceEnvVars to ensure it is always called
* Use byte slices for easier use
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
This is because of our sequential handling logic which was recently
merged; if vars is the first handler in the chain, it will be run before
the next route's matchers are executed, so there's no need to nest the
handlers anymore.
* http: path matcher: exact match by default; substring matches (#2959)
This is a breaking change.
* caddyfile: Change "matcher" directive to "@matcher" syntax (#2959)
* cmd: Assume caddyfile adapter for config files named Caddyfile
* Sub-sort handlers by path matcher length (#2959)
Caddyfile-generated subroutes have handlers, which are sorted first by
directive order (this is unchanged), but within directives we now sort
by specificity of path matcher in descending order (longest path first,
assuming that longest path is most specific).
This only applies if there is only one matcher set, and the path
matcher in that set has only one path in it. Path matchers with two or
more paths are not sorted like this; and routes with more than one
matcher set are not sorted like this either, since specificity is
difficult or impossible to infer correctly.
This is a special case, but definitely a very common one, as a lot of
routing decisions are based on paths.
* caddyfile: New 'route' directive for appearance-order handling (#2959)
* caddyfile: Make rewrite directives mutually exclusive (#2959)
This applies only to rewrites in the top-level subroute created by the
HTTP caddyfile.
* transform a caddyfile with environment variables
* support adapt time and runtime variables in the caddyfile
* caddyfile: Pre-process environment variables before parsing
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
This commit goes a long way toward making automated documentation of
Caddy config and Caddy modules possible. It's a broad, sweeping change,
but mostly internal. It allows us to automatically generate docs for all
Caddy modules (including future third-party ones) and make them viewable
on a web page; it also doubles as godoc comments.
As such, this commit makes significant progress in migrating the docs
from our temporary wiki page toward our new website which is still under
construction.
With this change, all host modules will use ctx.LoadModule() and pass in
both the struct pointer and the field name as a string. This allows the
reflect package to read the struct tag from that field so that it can
get the necessary information like the module namespace and the inline
key.
This has the nice side-effect of unifying the code and documentation. It
also simplifies module loading, and handles several variations on field
types for raw module fields (i.e. variations on json.RawMessage, such as
arrays and maps).
I also renamed ModuleInfo.Name -> ModuleInfo.ID, to make it clear that
the ID is the "full name" which includes both the module namespace and
the name. This clarity is helpful when describing module hierarchy.
As of this change, Caddy modules are no longer an experimental design.
I think the architecture is good enough to go forward.
* Always cleanup admin endpoint first
* Error out if no config has been set (#2833)
* Ignore explicitly missing admin config (#2833)
* Separate config loading from admin initialization (#2833)
* Add admin option to specify admin listener address (#2833)
* Use zap for reporting admin endpoint status
* fuzz: add missing fuzzer by fixing .gitignore adding a negation for caddyfile/ directory
* ci: print fuzzing type for debuggability and traceability
* README: update the Fuzzit badge to point to the correct Caddy server Github organization
* fuzz: lay down the foundation for continuous fuzzing
* improve the fuzzers and add some
* fuzz: add Fuzzit badge to README & enable fuzzers submission in CI
* v2-fuzz: do away with the submodule approach for fuzzers
* fuzz: enable fuzzit
* file_server: Make tests work on Windows
* caddyfile: Fix escaping when character is not escapable
We only escape certain characters depending on inside or outside of
quotes (mainly newlines and quotes). We don't want everyone to have to
escape Windows file paths like C:\\Windows\\... but we can't drop the
\ either if it's just C:\Windows\...
* v2: split golangci-lint configuration into its own file to allow code editors to take advantage of it
* v2: simplify code
* v2: set the correct lint output formatting
* v2: invert the logic of linter's configuration of output formatting to allow the editor convenience over CI-specific customization. Customize the output format in CI by passing the flag.
* v2: remove irrelevant golangci-lint config
This implements HTTP basicauth into Caddy 2. The basic auth module will
not work with passwords that are not securely hashed, so a subcommand
hash-password was added to make it convenient to produce those hashes.
Also included is Caddyfile support.
Closes#2747.
This integrates a feature that was previously reserved for enterprise
users, according to https://github.com/caddyserver/caddy/issues/2786.
The /config and /id endpoints make granular config changes possible as
well as the exporting of the current configuration.
The /load endpoint has been modified to wrap the /config handler so that
the currently-running config can always be available for export. The
difference is that /load allows configs of varying formats and converts
them using config adapters. The adapted config is then processed with
/config as JSON. The /config and /id endpoints accept only JSON.
Newlines (\n) can be escaped outside of quoted areas and the newline
will be treated as whitespace but not as an actual line break. Escaping
newlines inside a quoted area is not necessary, and because quotes
trigger literal interpretation of the contents, the escaping backslash
will be parsed as a literal backslash, and the newline will not be
escaped.
Caveat: When a newline is escaped, tokens after it until an unescaped
newline will appear to the parser be on the same line as the initial
token after the last unescaped newline. This may technically lead to
some false line numbers if errors are given, but escaped newlines are
counted so that the next token after an unescaped newline is correct.
See #2766
Before this change, only response headers could be manipulated with the
Caddyfile's 'header' directive.
Also handle the request Host header specially, since the Go standard
library treats it separately from the other header fields...
Use piles from which to draw config values.
Module values can return their name, so now we can do two-way mapping
from value to name and name to value; whereas before we could only map
name to value. This was problematic with the Caddyfile adapter since
it receives values and needs to know the name to put in the config.
Along with several other changes, such as renaming caddyhttp.ServerRoute
to caddyhttp.Route, exporting some types that were not exported before,
and tweaking the caddytls TLS values to be more consistent.
Notably, we also now disable automatic cert management for names which
already have a cert (manually) loaded into the cache. These names no
longer need to be specified in the "skip_certificates" field of the
automatic HTTPS config, because they will be skipped automatically.
