Add --header CLI flag for custom HTTP headers on verification requests

[infraguard-web]

Description

Closes #3138.

Adds a repeatable --header "Name: value" CLI flag that injects custom HTTP headers into every detector verification request. Useful for proxy/WAF authentication, scanner identification at WAFs/SOCs, correlation/telemetry tagging, and other cross-cutting needs that require headers on outbound verification traffic.

Usage

trufflehog [source] [target] \
  --header "X-Audit-Id: 1234" \
  --header "Authorization-Proxy: Bearer ${PROXY_TOKEN}"

The flag is repeatable. Format must be Name: value (curl-style). Names and values are validated at startup with httpguts.ValidHeaderFieldName / ValidHeaderFieldValue; malformed entries cause the binary to exit with a clear error rather than silently dropping. Empty values are permitted per RFC 7230. Empty names are rejected.

Architecture

Detectors use one of three HTTP patterns. All three are covered:

1. Shared HTTP client (most detectors). The existing CustomTransport and detectorTransport already wrap every RoundTrip; they now call a shared common.ApplyCustomHeaders(req) helper that injects user-configured headers. To avoid stacking on User-Agent and other transport-set defaults, ApplyCustomHeaders uses Set on the first value of each name and Add on subsequent values, giving single-value user headers true override semantics. AtomicHeader.Load returns a defensive clone of the configured headers so callers cannot mutate the shared map and race with other readers.

2. Stdlib HTTP defaults and x/oauth2 fallthrough. Detectors that use http.DefaultClient, transport-less &http.Client{} constructions, or oauth2.Transport with a nil Base read http.DefaultTransport at request time. main.go wraps DefaultTransport with CustomTransport at startup, but only when --header is set, so users not using the flag see no default-behavior change. This single change replaces what would otherwise be per-detector edits.

3. aws-sdk-go-v2. The AWS SDK builds its own complete transport stack and ignores http.DefaultTransport. Coverage is provided by applyCustomHeadersMiddleware, a Smithy Build-phase middleware registered on the STS (verifyMatch) and SNS (verifyCanary) clients in pkg/detectors/aws/access_keys. It mirrors the placement and pattern of the existing replaceUserAgentMiddleware.

Supporting changes

• pkg/detectors/readme/readme.go switches from http.DefaultClient to detectors.DetectorHttpClientWithNoLocalAddresses. Routes through the shared transport (so it honors --header) and closes an SSRF avenue on the verification path.
• pkg/detectors/html2pdf/html2pdf.go replaces http.Post(...) with http.NewRequestWithContext + common.SaneHttpClient().Do. Routes through the shared transport and gives the verifier proper context propagation it had been missing.

Regression prevention

pkg/detectors/transport_audit_test.go (TestNoStdlibHTTPBypassInDetectors) walks pkg/detectors/**/*.go and fails the build if any non-test source uses http.DefaultClient, http.Get, http.Post, http.Head, or http.PostForm. The allowlistedFiles map is a documented extension point for future SDK-managed exceptions (currently empty — the AWS middleware path doesn't match the regex patterns).

Tests

• 16-case table-driven parser test covering valid input, empty values, CRLF rejection, missing separators, and dropped = separator
• 4-subtest ApplyCustomHeaders test covering no-op, single-value override, multi-value Set-then-Add, and Load-clone safety
• End-to-end httptest.NewServer test confirming custom headers reach the wire through CustomTransport
• AWS SDK middleware test using a fake HTTPClient that captures the outbound request — verifies headers reach the wire through the middleware stack without requiring real AWS credentials
• CI guard test (passes on the current detector set)

Manually verified through Burp Suite against three representative verifiers covering all three architectures: Jotform (shared client), Onelogin US + EU (DefaultTransport wrap path), and AWS STS (SDK middleware). All four captures showed both custom headers reaching the wire; User-Agent stayed single-valued.

Trade-offs called out

• http.DefaultTransport mutation at startup. Mutating a Go global is generally a smell; here it is gated behind --header being set, executed once after flag parse in main.go, and clearly commented. The alternative — routing every http.Client construction site through a shared helper — would be much more invasive. Open to changing the approach if maintainers prefer.
• Fatal-at-startup on malformed --header values. Silent skip would let pipeline misconfigurations propagate to production verifications; failing loud at startup matches what users would expect from a security tool. If maintainers want a --header-best-effort style escape hatch, easy to add.

The flag's help entry (and regenerated man page) explicitly warn that custom headers travel to every verification endpoint, including external SaaS, so users should not configure credentials scoped to a single target via --header.

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

---

Note

Medium Risk
Adds global injection of user-specified HTTP headers into verification traffic (including wrapping http.DefaultTransport) and extends AWS SDK middleware, which could affect outbound request behavior across many detectors if misconfigured.

Overview
Adds a repeatable --header "Name: value" CLI flag and man-page docs to attach custom HTTP headers to all detector verification requests, with startup-time validation and a hard fail on malformed entries.

Implements global header storage (feature.CustomHeaders) and applies it in shared HTTP transports (common.CustomTransport and detectors transport) with override semantics (first value Set, subsequent Add), plus conditionally wraps http.DefaultTransport at startup when --header is provided to catch stdlib-default clients.

Extends coverage to non-stdlib paths by injecting headers into AWS aws-sdk-go-v2 verification calls via Smithy middleware, updates a couple of detectors to use the shared HTTP clients (e.g., readme, html2pdf), and adds tests (header parsing, transport/header propagation, AWS middleware) plus a detectors audit test to prevent future direct use of stdlib HTTP helpers that would bypass the shared transport chain.

^{Reviewed by Cursor Bugbot for commit 5e61bfd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ infraguard-web
❌ forgedhallpass
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 5e61bfd. Configure here.}

[cursor]

Double-wrapped CustomTransport duplicates User-Agent header

Medium Severity

When --header is set, http.DefaultTransport is replaced with a CustomTransport. Subsequently, any call to NewCustomTransport(nil) (which falls through to http.DefaultTransport) creates a nested CustomTransport wrapping another CustomTransport. Each layer calls req.Header.Add("User-Agent", ...), producing duplicate User-Agent values on the wire. This affects the new huggingface/client.go code and existing runtime callers like RetryableHTTPClientTimeout used by GitHub and CircleCI sources.

Additional Locations (2)

• pkg/sources/huggingface/client.go#L129-L130
• pkg/common/http.go#L124-L130

 

^{Reviewed by Cursor Bugbot for commit 5e61bfd. Configure here.}