Video overview (click to play):

Listen to this article:

A medium-severity GuardDuty finding flagged a single GET /version against the Kubernetes API of one of our production EKS clusters, made from a Tor exit node. The request succeeded, which on its own is unremarkable: Kubernetes serves that path to unauthenticated callers by default. It turned out to be the only visible trace of a scan of roughly 1,300 requests that tried to reach internal-only services through our internet-facing Application Load Balancer (ALB). The scanner paired a legitimate public TLS name with internal-looking Host headers and let a shared ingress controller do the rest.

In this post, we walk through the incident end to end: how we detected it, how we scoped it from load balancer access logs, how we reproduced the flaw safely in a sandbox, the single listener change that closed it, and how we monitored the result.

Background: how a public load balancer ended up in front of private hosts

A common EKS setup puts two Application Load Balancers in front of one ingress controller: an internet-facing ALB for public applications and an internal ALB for everything else. Both forward to the same NGINX ingress controller, which routes by Host header to hundreds of ingresses. At the time of the incident, public applications and internal tools were separated by hostname and by which load balancer people happened to use, not by anything the ingress itself enforced.

Before the change. The public ALB’s default rule forwarded every request to the same ingress that serves private hosts.

Our public listener had certificates for public domains and a single default rule: forward everything to the ingress. There was no host condition. That is a common starting point because it works. Public hostnames route correctly, and nobody notices what else routes.

SNI and Host: two names, two layers

A single HTTPS request carries two names, and different components read them.

The same connection carries two names. The load balancer uses SNI to pick a certificate. The ingress uses Host to pick a backend.

• SNI (Server Name Indication) is sent in the TLS handshake. The ALB uses it to choose which certificate to present.
• The Host header is sent inside the encrypted request. The ingress controller uses it to choose which backend serves the request.

Nothing requires the two to match. Presenting one name at the TLS layer and another at the HTTP layer is the core of domain fronting, a technique better known from CDN abuse and censorship circumvention. Against a single origin, the same trick works whenever the component that terminates TLS forwards every Host value to a backend that serves more than it should.

# The TLS handshake and certificate validation use the public name.
# The Host header names an internal-only application.
curl -H "Host: internal-tool.example.internal" https://app.example.com/

The handshake completes against a valid public certificate, so there is nothing to bypass at the TLS layer. If the load balancer forwards the request and the ingress has a rule for that Host value, the internal application answers.

Detection: one low-severity alert

The first signal was a GuardDuty EKS audit-log finding of type Discovery:Kubernetes/TorIPCaller, default severity Medium. It said that a Kubernetes API commonly used for discovery had been called from a Tor exit node. The details were thin:

• one GET /version that returned 200
• an unauthenticated identity, relayed by a component that talks to the Kubernetes API on a caller’s behalf
• a Tor exit node as the client address, preserved through forwarding headers
• a browser-style user agent

The 200 was not a breach. The cluster’s API endpoint is private, and Kubernetes serves /version to unauthenticated callers by default. What mattered was the path. A private API server had seen a Tor address, which meant a request had crossed from the internet, through a load balancer and the ingress, to something that talks to the API. Only one request in the sequence reached the API server, so only one was visible to GuardDuty.

That reasoning set the next step. The interesting traffic would not be in Kubernetes audit logs, so we pivoted to the internet-facing ALB’s access logs and searched around the finding’s timestamp for the same source and user agent.

Takeaway: a Medium finding for a request that returned 200 and exposed nothing is easy to close. Treat it as a pivot point instead, and ask what else the same source did.

Scoping the scan from load balancer logs

Making the logs queryable

ALB access logs land in S3 as compressed files. We defined an Athena table over them with partition projection by date, so a query over one day touches one prefix. The columns that carry the story are documented in the ALB access log format:

The most useful first query asks which requests carried an SNI that differs from their Host:

-- Requests whose Host header differs from the TLS SNI name.
-- Column names follow the ALB log format; adjust them to the table definition.
WITH parsed AS (
  SELECT
    domain_name AS sni,
    lower(regexp_extract(request, '^\S+ https?://([^:/ ]+)', 1)) AS host,
    split_part(client, ':', 1) AS source_ip,
    target_status_code
  FROM alb_access_logs
  WHERE year = '<yyyy>' AND month = '<mm>' AND day = '<dd>'
)
SELECT
  sni,
  host,
  count(*) AS requests,
  count(DISTINCT source_ip) AS sources,
  count_if(CAST(target_status_code AS varchar) LIKE '2%') AS ok_responses
FROM parsed
WHERE sni <> host
GROUP BY sni, host
ORDER BY requests DESC
LIMIT 100;

On ordinary days this returns a small amount of benign noise. On the scan day it returned hundreds of distinct host values paired with a handful of sni values.

What the scan looked like

• Source. Traffic came from about 70 Tor exit nodes that rotated frequently. Source IP was a poor grouping key. Behavior was a good one.
• Two phases. In the first phase, the scanner pinned its SNI to a legitimate public hostname and rotated the Host header. In the second, it sent no usable SNI at all (about 44% of requests) and kept rotating Host. The same weakness applies to both, which is why the fix matches on Host alone.
• Wordlist. The scanner tried a few hundred guessed hostnames, mostly service-style names, and mixed in made-up names to learn what a “not found” response looks like. It then concentrated on names that answered differently.

About 95% of the guesses matched no ingress and returned the ingress controller’s default 404. About 20 matched real ingresses. That difference in response was all the scanner needed. It never had to know what existed, only to notice which names answered differently.

Of about 340 guessed hostnames, about 20 mapped to real ingresses.

Joining the probes to the ingress inventory

Status codes alone mislead. An application can return 200 for every path, and a single-page app with a catch-all route is the classic case. To separate real hits from noise, we joined the probed hostnames against the cluster’s ingress inventory:

# Every host the ingress controller is configured to serve
kubectl get ingress -A -o json \
  | jq -r '.items[].spec.rules[]?.host' | sort -u

For each of the roughly 20 matching hosts, we classified the outcome:

• Answered. A small number of internal HTTP interfaces answered unauthenticated requests. Some returned diagnostic data, and a few accepted requests that changed state.
• Rejected. Services with enforced authentication returned 401 or 403 to every request.
• Backend unavailable. Some requests reached the ingress but found no healthy backend (502). Those services were protected by being down, not by a control.
• Application 404. The route existed, but the application did not know the path.

Credentials did not appear in the responses we could inspect. ALB logs record neither request nor response bodies, so we handed the owning teams the timestamps and paths, because the remaining questions can only be answered from application logs.

One service looked alarming at first: dozens of 200 responses for sensitive-looking management paths. A request to a nonsense path returned the same small HTML page, so those responses were a single-page app’s catch-all route, not exposed endpoints. Our first summary changed in four places after direct, read-only probing of each service.

How the analysis was done

After the initial detection, all analysis was carried out by Claude Opus 5.5 Max. It had access to the logs, the query engine, and the cluster. The loop was the same throughout: form a hypothesis, write the query, cross-check it against a second source, and record what is and is not known. Live probing stayed read-only. The first summary was treated as a draft and re-verified against the running systems, which is how the false alarm above was caught.

Reproducing the flaw in a sandbox

Logs showed what happened. They did not prove why. Before changing a production load balancer, we wanted to reproduce the behavior in a sandbox cluster that mirrors production topology: the same ingress controller behind a public and an internal ALB.

The test workload was deliberately boring: one pod returning a fixed marker string, a ClusterIP service, and an ingress for a private-style hostname with no counterpart on the public side. Three requests were enough:

1. Baseline. Request the private hostname through the internal ALB. The marker comes back. That is the intended path.
2. Simulation. Send a request to the public ALB with a legitimate public name in SNI and the private hostname in Host. The handshake succeeds against the real public certificate, with no TLS verification bypass, and the marker comes back. Same pod, wrong door.
3. Negative control. Same public ALB, same valid SNI, but a Host that no ingress serves. NGINX returns 404. That shows NGINX actively matches on Host rather than acting as a catch-all, so the gap is specifically a missing host constraint upstream.

The root cause was confirmed one layer up. The public listener’s default rule had no host condition, so it forwarded everything to the ingress, the same ingress the internal ALB uses, and the load balancer’s security group allowed the internet. SNI only picks a certificate. It never gated which Host could pass. Inside NGINX, nothing records which load balancer a connection arrived through, so the ingress had no way to tell the two doors apart.

When the safeguards said no

This is the stage where the workflow changed models. The simulation, the implementation of the fix, and its testing were done by Claude Sonnet 5 Max. Three other models, Claude Opus 5.5, Claude Opus 5, and Claude Sonnet 5.5, each declined to help because the attack simulation tripped their cybersecurity safeguards.

That is a reasonable default: a request to craft traffic that reaches private services through a public endpoint looks like offense from the outside. It is still worth planning for. Teams that want AI help with defensive validation should expect that a preferred model may decline, keep the scope visibly narrow (an owned sandbox, placeholder data, a fixed marker string), and have a fallback ready so the response does not block on one model’s judgment.

The fix: fail closed at the edge

The goal was a change that is fast to deploy, reversible in seconds, and independent of application teams: stop the public load balancer from forwarding hosts it was never meant to serve.

The rule

On the public HTTPS listener, we added a priority-1 rule that forwards only when the Host matches the public domain families, and changed the listener’s default action from forward to a fixed 403:

# 1. Allow rule first, so public traffic keeps flowing
aws elbv2 create-rule --listener-arn <public-listener-arn> --priority 1 \
  --conditions '[{"Field":"host-header","HostHeaderConfig":{"Values":["*.example.com","*.example.org"]}}]' \
  --actions '[{"Type":"forward","TargetGroupArn":"<ingress-target-group-arn>"}]'

# 2. Then flip the default action to fail closed
aws elbv2 modify-listener --listener-arn <public-listener-arn> \
  --default-actions '[{"Type":"fixed-response","FixedResponseConfig":{"StatusCode":"403","ContentType":"text/plain","MessageBody":"Forbidden"}}]'

The order matters. Creating the allow rule first means there is never a moment when a public hostname hits the new default. We built the list from the ingress inventory. Every non-internal hostname the ingress served fell under a handful of root domains, so a handful of wildcard values covered them.

Three behaviors worth verifying

We tested each of these in the sandbox instead of relying on documentation alone:

• ALB wildcards are not certificate wildcards. In a host-header condition, * matches zero or more characters, dots included, so *.example.com also matches a.b.example.com. A certificate wildcard covers a single label. One rule therefore covers every depth of subdomain, which is convenient and worth knowing before assuming otherwise.
• ALB listener rules have hard limits. A rule accepts five condition values. We hit the API error on purpose to confirm it. Longer lists need more rules.
• A wildcard never matches the bare apex. *.example.com does not match example.com. If the apex should be served, it needs its own literal value.

Production cutover

Production went in as an emergency change. The two AWS CLI calls above ran. We then checked live traffic, not only synthetic tests. The highest-volume public hostname and a consumer-facing application still returned 200. Replaying the sandbox simulation against production, with a public SNI and an internal-only Host, now returned the 403 from the load balancer instead of reaching the internal application:

# Expect 403 from the load balancer itself
curl -H "Host: internal-tool.example.internal" https://app.example.com/

After the change. The public path is gated by the allowlist. The internal path is untouched.

An allowlist has a cost: a new public domain is blocked until it is added, so onboarding needs a step for it. An edge allowlist is also just one layer. We’ve since hardened security even more by rejecting requests with mismatched SNI & host header & deploying a separate ingress controller for public hosts.

Monitoring the block

A fail-closed rule is only safe if its blocks are visible. Two layers cover it.

A fast signal. The ALB emits HTTP_Fixed_Response_Count per load balancer. Because the default 403 is the only fixed-response action on the listener, the metric is a one-to-one count of allowlist blocks at one-minute granularity.

Forensic detail. The same Athena table answers who, what, and how often:

-- Requests the default rule blocked, by Host header
SELECT
  lower(regexp_extract(request, '^\S+ https?://([^:/ ]+)', 1)) AS host,
  count(*) AS blocked,
  count(DISTINCT split_part(client, ':', 1)) AS sources
FROM alb_access_logs
WHERE year = '<yyyy>' AND month = '<mm>' AND day >= '<first day after the change>'
  AND TRY_CAST(matched_rule_priority AS integer) = 0
  AND actions_executed LIKE '%fixed-response%'
GROUP BY 1
ORDER BY blocked DESC
LIMIT 50;

By Host, the blocked traffic fell into three groups:

• About two thirds targeted a retired consumer domain that still had a certificate but no backing ingress.
• About a quarter used the load balancer’s own IP addresses as the Host.
• The remainder was a mix of the load balancer’s DNS names and unrelated domains.

By behavior, just over half was scanning and probing: path guessing, probes for dotfiles and configuration files, exploit attempts against popular web frameworks, and known internet-wide scanners. About a fifth was an uptime monitor that had already been failing with 404 before the change. About a sixth was ordinary-looking browser traffic to old pages. Crawlers, including search and AI crawlers, accounted for most of the rest.

The question that matters for any fail-closed rule is what it blocked that it should not have. Three checks answered it:

1. In the week before the change, every blocked host but one had already been failing at the backend with a 404 or 401. The exception was an internal-only tool that had been answering through the public ALB. Blocking it was the point.
2. None of the blocked requests was for a hostname in the allowed domain families.
3. We joined the blocked Host values against the current ingress inventory. Of roughly 430 distinct blocked values, two matched an ingress, and both are internal-only by design.

We have not seen the scanner’s behavior return in the logs since. Internet-wide noise, on the other hand, is constant, and the default rule now doubles as a sensor for it. An alarm on the metric is the obvious next step, and the baseline above is what its threshold should be tuned against.

Timeline

Response timeline in days from the scan. The dashed tags show what produced each stage.

• Day 0: a roughly two-hour scan from about 70 Tor exit nodes.
• Day 0: one request reaches the Kubernetes API. GuardDuty raises a finding within a minute.
• Days 0 to 1: triage and scoping from ALB access logs. By the time the investigation opened, the scan had stopped.
• Day 2: sandbox reproduction confirms the flaw.
• Days 2 to 3: the fix is validated in non-production environments, then approved as an emergency change.
• Day 3: production enforcement.
• Days 3 to 6: monitoring.

Key takeaways

• SNI selects a certificate; it never authorizes a Host. A public load balancer that forwards every Host value to a shared ingress turns each private virtual host into a public one.
• GuardDuty saw one request. The other ~1,300 were visible only in ALB access logs, so access logs belong in the investigation from the first hour.
• Reproducing the flaw in a sandbox first turned a hypothesis into evidence and gave the fix a regression test.
• The fix is a host-header allowlist on the public listener with a default 403. Production cutover took under a minute and was validated against live traffic.
• In the first four days after enforcement, the default rule blocked more than 20,000 requests from roughly 3,000 sources. None of them targeted a legitimate public hostname.
• All analysis after the initial detection was done by Claude Opus 5.5 Max. The attack simulation, the fix, and its testing were done by Claude Sonnet 5 Max, after Opus 5.5, Sonnet 5.5 & Opus 5 all refused to help because the sandbox simulation tripped their cybersecurity guardrails.

Mitigation and detection guidance

For teams running a shared ingress controller behind an internet-facing load balancer:

• Constrain what the public listener forwards. Add host-header conditions for the domains that should be public and make the default action a fixed 403. Create the allow rule before changing the default.
• Build the list from inventory, per environment. Query the cluster’s ingresses, and re-check on every environment the rule is promoted to.
• Verify matching semantics. Test wildcard behavior and quotas in a sandbox. ALB host wildcards and certificate wildcards do not behave the same way.
• Do not treat hostnames as authentication. Anything reachable through a shared ingress should authenticate its callers, whether or not its name looks internal.
• Log SNI and the matched rule. domain_name, matched_rule_priority, and actions_executed turn the load balancer into a sensor. Alert on SNI and Host mismatches and on default-rule hits.
• Watch the metric that counts blocks. HTTP_Fixed_Response_Count is a cheap early signal for both attacks and missing allowlist entries.
• Treat low-severity findings as pivots. Ask what else the same source did, and look in the logs that record the other 99.9% of the traffic.
• Layer the controls. An edge allowlist is one layer. Keep applying defense in depth to the ingress tier and to the services behind it.

What we learned

The scan was over before the investigation began. The block does not stop that scan. It closes the path for the next one.

Reproduce before changing production. The sandbox turned a log-derived hypothesis into evidence, and it gave the fix a test to flip from 200 to 403. It also settled the question of root cause, which logs alone could not.

Analysis is where AI paid off. Query writing, inventory joins, and impact write-ups are tedious for people and quick for a model, and Claude Opus 5.5 Max handled all of it after the initial detection. Verification stayed deliberately skeptical, and that is what caught the single-page-app false alarm.

Plan for refusals. The simulation tripped the safeguards of three models, and Claude Sonnet 5 Max finished the simulation, the fix, and its testing. Defensive teams should keep a fallback model and a narrow, well-labeled scope ready before they need one.

Conclusion

A single GET /version was the visible tip of a scan that never needed to know what it was looking for. It only needed a load balancer that forwarded every Host to a shared ingress, and a response difference to follow. Closing that path took one listener rule, a default 403, and a careful rollout. Knowing the scan existed at all took a medium-severity alert that was worth following to the logs.

Athena EKS GuardDuty Ingress NGINX