* reverseproxy: Add more debug logs
This makes debug logging very noisy when reverse proxying, but I guess
that's the point.
This has shown to be useful in troubleshooting infrastructure issues.
* Update modules/caddyhttp/reverseproxy/streaming.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Update modules/caddyhttp/reverseproxy/streaming.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Add opt-in `trace_logs` option
* Rename to VerboseLogs
---------
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Enhancement: Allow X-Forwarded-For Header in httpInclude Virtual Requests
The goal of this enhancement is to modify the funcHTTPInclude function in the Caddy codebase to include the X-Forwarded-For header in the virtual request. This change will enable reverse proxies to set the X-Forwarded-For header, ensuring that the client's IP address is correctly provided to the target endpoint. This modification is essential for applications that depend on the X-Forwarded-For header for various functionalities, such as authentication, logging, or content customization.
* Updated tplcontext.go - set `virtReq.RemoteAddr = "127.0.0.1"`
i have made the suggested changes
* Apply suggestions from code review
* Update modules/caddyhttp/templates/tplcontext.go
---------
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
fix a nil pointer dereference in AUpstreams.GetUpstreams when AUpstreams.Versions is not set (fixes caddyserver#5809)
Signed-off-by: Pascal Vorwerk <info@fossores.de>
* use gofmput to format code
* use gci to format imports
* reconfigure gci
* linter autofixes
* rearrange imports a little
* export GOOS=windows golangci-lint run ./... --fix
* reverseproxy: do not parse upstream address too early if it contains replaceble parts
* remove unused method
* cleanup
* accommodate partially replaceable port
* caddyhttp: Make use of http.ResponseController
Also syncs the reverseproxy implementation with stdlib's which now uses ResponseController as well 2449bbb5e6
* Enable full-duplex for HTTP/1.1
* Appease linter
* Add warning for builds with Go 1.20, so it's less surprising to users
* Improved godoc for EnableFullDuplex, copied text from stdlib
* Only wrap in encode if not already wrapped
* update quic-go to v0.37.0
* Bump to Go 1.20
* Bump golangci-lint version, yml syntax consistency
* Use skip-pkg-cache workaround
* Workaround needed for both?
* Seeding weakrand is no longer necessary
---------
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* added weighted round robin algorithm to load balancer
* added an adapt integration test for wrr and fixed a typo
* changed args format to Caddyfile args convention
* added provisioner and validator for wrr
* simplified the code and improved doc
Allow registering a custom network mapping for HTTP/3. This is useful
if the original network for HTTP/1.1 and HTTP/2 is not a standard `unix`,
`tcp4`, or `tcp6` network. To keep backwards compatibility, we fall back
to `udp` if the original network is not registered in the mapping.
Fixes#5555
* Create an includeRaw template function to include a file without parsing it as a template.
Some formatting fixes
* Rename to readFile, various docs adjustments
---------
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Make grid entries take up full width on mobile and fix breadcrumb color issue in dark mode
Signed-off-by: Pistasj <odyssey346@disroot.org>
* Do mholt's suggestions
Signed-off-by: Pistasj <odyssey346@disroot.org>
---------
Signed-off-by: Pistasj <odyssey346@disroot.org>
When only a single request has the least amount of requests, there's no need to compute a random number, because the modulo of 1 will always be 0 anyways.
* Add isDir template function
* Update modules/caddyhttp/templates/tplcontext.go
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* Fix funcIsDir return value on error
* Fix funcIsDir return false when root file system not specified
* Add stat function, remove isDir function
* Remove isDir function (really)
* Rename stat to fileStat
---------
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* Serve http2 when listener wrapper doesn't return *tls.Conn
* close conn when h2server serveConn returns
* merge from upstream
* rebase from latest
* run New and Closed ConnState hook for h2 conns
* go fmt
* fix lint
* Add comments
* reorder import
* log: make `sink` encodable
* deduplicate logger fields
* extract common fields into `BaseLog` and embed it into `SinkLog`
* amend godoc on `BaseLog` and `SinkLog`
* minor style change
---------
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Delete all existing fields when fieldName is `*`
* Rearrange deletion before addition in headers
* Revert "Rearrange deletion before addition in headers"
This reverts commit 1b50eeeccc92ccd660c7896d8283c7d9e5d1fcb0.
* Treat deleting all headers as a special case
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
---------
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* caddyhttp: Determine real client IP if trusted proxies configured
* Support customizing client IP header
* Implement client_ip matcher, deprecate remote_ip's forwarded option
* cmd: Expand cobra support
* Convert commands to cobra, add short flags
* Fix version command typo
Co-authored-by: Emily Lange <git@indeednotjames.com>
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
---------
Co-authored-by: Emily Lange <git@indeednotjames.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* chore: Upgrade various dependencies
* Support CEL file matcher with no args
* Document `http.request.orig_uri.path.*`, reorder placeholders in docs
---------
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* added some tests for parseUpstreamDialAddress
Test 4 fails because it produces "[[::1]]:80" instead of "[::1]:80"
* support absolute windows path in unix reverse proxy address
* make IsUnixNetwork public, support +h2c and reuse it
* add new tests
* caddyauth: Add singleflight for basic auth
* Fixes#5338
* it occurred the thunder herd problem like this https://medium.com/@mhrlife/avoid-duplicate-requests-while-filling-cache-98c687879f59
* Update modules/caddyhttp/caddyauth/basicauth.go
Fix comment
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
---------
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* If upstreams are all using same host but with different ports
ie:
foobar:4001
foobar:4002
foobar:4003
...
Because fnv-1a has not a good enough avalanche effect
Then the hostByHashing result is not well balanced over
all upstreams
As last byte FNV input tend to affect few bits, the idea is to change
the concatenation order between the key and the upstream strings
So the upstream last byte have more impact on hash diffusion
* reverseproxy: Mask the WS close message when we're the client
* weakrand
* Bump golangci-lint version so path ignores work on Windows
* gofmt
* ugh, gofmt everything, I guess
* fileserver: Reject ADS and short name paths
* caddyhttp: Trim trailing space and dot on Windows
Windows ignores trailing dots and spaces in filenames.
* Fix test
* Adjust path filters
* Revert Windows test
* Actually revert the test
* Just check for colons
* httpcaddyfile: Skip some logic if auto_https off
* Try removing this check altogether...
* Refine test timeouts slightly, sigh
* caddyhttp: Assume udp for unrecognized network type
Seems like the reasonable thing to do if a plugin registers its own
network type.
* Add comment to document my lack of knowledge
* Clean up and prepare to merge
Add comments to try to explain what happened
PR #4066 added a dark color scheme to the file_server browse template.
PR #4356 later set the links for the `:visited` pseudo-class, but did
not set anything for the dark mode, resulting in poor contrast. I
selected some new colors by feel.
This commit also adds an `a:visited:hover` for both, to go along with
the normal blue hover colors.
It was not accurate. Placeholders could be used in outputs that are
defined in the same mapping as long as that placeholder does not do the
same.
A more general solution would be to detect it at run-time in the
replacer directly, but that's a bit tedious
and will require allocations I think.
A better implementation of this check could still be done, but I don't
know if it would always be accurate. Could be a "best-effort" thing?
But I've also never heard of an actual case where someone configured
infinite recursion...
* core: Refactor, improve listener logic
Deprecate:
- caddy.Listen
- caddy.ListenTimeout
- caddy.ListenPacket
Prefer caddy.NetworkAddress.Listen() instead.
Change:
- caddy.ListenQUIC (hopefully to remove later)
- caddy.ListenerFunc signature (add context and ListenConfig)
- Don't emit Alt-Svc header advertising h3 over HTTP/3
- Use quic.ListenEarly instead of quic.ListenEarlyAddr; this gives us
more flexibility (e.g. possibility of HTTP/3 over UDS) but also
introduces a new issue:
https://github.com/lucas-clemente/quic-go/issues/3560#issuecomment-1258959608
- Unlink unix socket before and after use
* Appease the linter
* Keep ListenAll
e338648fed introduced multiple upstream
addresses. A comment notes that mixing schemes isn't supported and
therefore the first valid scheme is supposed to be used.
Fixes setting the first scheme.
fixes#5087
* caddyhttp: Honor grace period in background
This avoids blocking during config reloads.
* Don't quit process until servers shut down
* Make tests more likely to pass on fast CI (#5045)
* caddyhttp: Even faster shutdowns
Simultaneously shut down all HTTP servers, rather than one at a time.
In practice there usually won't be more than 1 that lingers. But this
code ensures that they all Shutdown() in their own goroutine
and then we wait for them at the end (if exiting).
We also wait for them to start up so we can be fairly confident the
shutdowns have begun; i.e. old servers no longer
accepting new connections.
* Fix comment typo
* Pull functions out of loop, for readability
Ideally I'd just remove the parameter to caddy.Context.Logger(), but
this would break most Caddy plugins.
Instead, I'm making it variadic and marking it as partially deprecated.
In the future, I might completely remove the parameter once most
plugins have updated.
* fix encode handler header manipulation
also avoid implementing ReadFrom because it breaks when io.Copied to directly
* strconv.Itoa should be tried as a last resort
WriteHeader during Close
* feat: Multiple 'to' upstreams in reverse-proxy cmd
* Repeat --to for multiple upstreams, rather than comma-separating in a single flag
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* caddyhttp: Explicitly disallow multiple regexp matchers
Fix#5028
Since the matchers would overwrite eachother, we should error out to tell the user their config doesn't make sense.
* Update modules/caddyhttp/matchers.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Doing so allows for splice/sendfile optimizations when available.
Fixes#4731
Co-authored-by: flga <flga@users.noreply.github.com>
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* fileserver: Support glob expansion in file matcher
* Fix tests
* Fix bugs and tests
* Attempt Windows fix, sigh
* debug Windows, WIP
* Continue debugging Windows
* Another attempt at Windows
* Plz Windows
* Cmon...
* Clean up, hope I didn't break anything
* caddyhttp: Support sending HTTP 103 Early Hints
This adds support for early hints in the static_response handler.
* caddyhttp: Don't record 1xx responses
* reverseproxy: Close hijacked conns on reload/quit
We also send a Close control message to both ends of
WebSocket connections. I have tested this many times in
my dev environment with consistent success, although
the variety of scenarios was limited.
* Oops... actually call Close() this time
* CloseMessage --> closeMessage
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Use httpguts, duh
* Use map instead of sync.Map
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* core: Refactor listeners; use SO_REUSEPORT on Unix
Just an experiment for now
* Fix lint by logging error
* TCP Keepalive configuration (#4865)
* initial attempt at TCP Keepalive configuration
* core: implement tcp-keepalive for linux
* move canSetKeepAlive interface
* Godoc for keepalive server parameter
* handle return values
* log keepalive errors
* Clean up after bad merge
* Merge in pluggable network types
From 1edc1a45e3
* Slight refactor, fix from recent merge conflict
Co-authored-by: Karmanyaah Malhotra <karmanyaah.gh@malhotra.cc>
* break up code and use lazy reading and pool bufio.Writer
* close underlying connection when operation failed
* allocate bufWriter and streamWriter only once
* refactor record writing
* rebase from master
* handle err
* Fix type assertion
Also reduce some duplication
* Refactor client and clientCloser for logging
Should reduce allocations
* Minor cosmetic adjustments; apply Apache license
* Appease the linter
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* added the httpError function into the document
* Update templates.go
* Update templates.go
* Fix gofmt
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Until now, the vars matcher has unintentionally lacked parity with the
map directive: the destination placeholders of the map directive would
be expressed as placeholders, i.e. {foo}. But the vars matcher would
not use { }: vars foo value
This looked weird, and was confusing, since it implied that the key
could be dynamic, which doesn't seem helpful here.
I think this is a proper bug fix, since we're not used to accessing
placeholders literally without { } in the Caddyfile.
* core: Plugins can register listener networks
This can be useful for custom listeners.
This feature/API is experimental and may change!
* caddyhttp: Expose server listeners
This allows users to, for example, get upstreams from multiple SRV
endpoints in order (such as primary and secondary clusters).
Also, gofmt went to town on the comments, sigh
* fileserver: Support virtual file systems (close#3720)
This change replaces the hard-coded use of os.Open() and os.Stat() with
the use of the new (Go 1.16) io/fs APIs, enabling virtual file systems.
It introduces a new module namespace, caddy.fs, for such file systems.
Also improve documentation for the file server. I realized it was one of
the first modules written for Caddy 2, and the docs hadn't really been
updated since!
* Virtualize FS for file matcher; minor tweaks
* Fix tests and rename dirFS -> osFS
(Since we do not use a root directory, it is dynamic.)
Hahaha this is the ultimate "I have no idea what I'm doing" commit but it
compiles and the tests pass and I declare victory!
... probably broke something, should be tested more.
It is nice that the protobuf dependency becomes indirect now.
* reverseproxy: Implement retry count, alternative to try_duration
* Add Caddyfile support for `retry_match`
* Refactor to deduplicate matcher parsing logic
* Fix lint
See https://caddy.community/t/using-forward-auth-and-writing-my-own-authenticator-in-php/16410, apparently it didn't work when `copy_headers` wasn't used. This is because we were skipping adding a handler to the routes in the "good response handler", but this causes the logic in `reverseproxy.go` to ignore the response handler since it's empty. Instead, we can just always put in the `header` handler, even with an empty `Set` operation, it's just a no-op, but it fixes that condition in the proxy code.
* Make reverse proxy TLS server name replaceable for SNI upstreams.
* Reverted previous TLS server name replacement, and implemented thread safe version.
* Move TLS servername replacement into it's own function
* Moved SNI servername replacement into httptransport.
* Solve issue when dynamic upstreams use wrong protocol upstream.
* Revert previous commit.
Old commit was: Solve issue when dynamic upstreams use wrong protocol upstream.
Id: 3c9806ccb6
* Added SkipTLSPorts option to http transport.
* Fix typo in test config file.
* Rename config option as suggested by Matt
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Update code to match renamed config option.
* Fix typo in config option name.
* Fix another typo that I missed.
* Tests not completing because of apparent wrong ordering of options.
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Make reverse proxy TLS server name replaceable for SNI upstreams.
* Reverted previous TLS server name replacement, and implemented thread safe version.
* Move TLS servername replacement into it's own function
* Moved SNI servername replacement into httptransport.
* Solve issue when dynamic upstreams use wrong protocol upstream.
* Revert previous commit.
Old commit was: Solve issue when dynamic upstreams use wrong protocol upstream.
Id: 3c9806ccb6
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
In v2.5.0, upstream health was fixed such that whether an upstream is
considered healthy or not is mostly up to each individual handler's
config. Since "healthy" is an opinion, it is not a global value.
I unintentionally left in the "healthy" field in the API endpoint for
checking upstreams, and it is now misleading (see #4792).
However, num_requests and fails remains, so health can be determined by
the API client, rather than having it be opaquely (and unhelpfully)
determined for the client.
If we do restore this value later on, it'd need to be replicated once
per reverse_proxy handler according to their individual configs.
