Case FilesCase 02 / 9 min read

Incident Case 02

CF-RAY: -

curl from the pod worked. The app from the same pod did not. Cloudflare returned a 400 for a request it pretended never existed.

01 / The Alert

The integration had been stable for a long time. Then a network migration placed the upstream endpoint behind Cloudflare, and one Java service started receiving 400 Bad Request responses.

The confusing part was not the 400. The confusing part was the missing Ray ID. Cloudflare normally stamps every edge response with CF-RAY. Here, the header came back as a dash: CF-RAY: -.

The app was inside Kubernetes. The endpoint was reachable. DNS resolved. TLS worked. curl from the same pod succeeded and returned a real Ray ID. Only the application failed.

When curl works from the pod and the application does not, stop blaming the network. Start reading what the application actually sends.

02 / The Hunt

The first test was to remove infrastructure from the suspect list.

Control test from the same podbash
kubectl exec -it <pod> -n <namespace> -- curl -vk https://<cloudflare-endpoint>

curl succeeded. The response contained a real Ray ID. That proved DNS, routing, egress, TLS, firewall path, and Cloudflare reachability were all good enough for a clean request.

The next move was to inspect the Java client without introducing a proxy or certificate tricks. JSSE can dump plaintext HTTP before TLS encryption:

Temporary JVM diagnostic flagbash
-Djavax.net.debug=ssl:record:plaintext

This is dangerous in production because it logs credentials and tokens. But for a tightly controlled capture, it answers the question that packet captures cannot: what exact HTTP headers did the application send before TLS encrypted them?

03 / The Discovery

The wire log showed two separate problems.

First, the outbound RestTemplate was carrying an inbound Bearer token into the outbound request. The app's own integration credential was also present. That created duplicate/conflicting Authorization semantics.

Second, the Java client was sending a huge Accept-Charset header. Older firewalls tolerated it. Cloudflare did not.

The application was not blocked by the network. It was manufacturing an HTTP request that Cloudflare rejected before assigning a normal Ray ID.

A Cloudflare migration is not just a network change. It is an HTTP compliance audit.

04 / The Fix

Two fixes were required from the application team, corresponding to the two root causes.

Fix 1 - isolate the outbound RestTemplate from Spring Securityjava
@Configuration
public class WebConfig {

  @Bean
  public RestTemplate cnsRestTemplate() {
    HttpComponentsClientHttpRequestFactory factory =
      new HttpComponentsClientHttpRequestFactory();

    // Create a RestTemplate that does NOT inherit Spring Security interceptors
    RestTemplate restTemplate = new RestTemplate(factory);
    restTemplate.setInterceptors(Collections.emptyList());

    return restTemplate;
  }
}
Fix 2 - strip Accept-Charset from outbound requestsjava
restTemplate.getInterceptors().add((request, body, execution) -> {
    request.getHeaders().remove(HttpHeaders.ACCEPT_CHARSET);
    return execution.execute(request, body);
});

After both fixes were applied, requests to the Cloudflare-fronted endpoint started returning real Ray IDs. The origin was reached. Integrations recovered.

The diagnostic flag was removed immediately after the log capture.

Removing the diagnostic flagbash
kubectl patch deployment <app-deployment> -n <namespace> \
  --type=json -p='[{
    "op": "remove",
    "path": "/spec/template/spec/containers/0/args/0"
  }]'
05 / The Lesson

Cloudflare enforces HTTP spec strictly - your old firewall probably did not

The previous firewall silently tolerated duplicate headers and oversized headers. Cloudflare rejected them immediately.

Spring Security interceptors can leak into RestTemplate instances

If your outbound client inherits security interceptors, inbound authentication can be propagated to outbound calls where it does not belong.

JSSE plaintext logging is powerful and dangerous

-Djavax.net.debug=ssl:record:plaintext can show every header before TLS encryption. Use it carefully, capture only what you need, and remove it immediately.

When CF-RAY is a dash, replay via curl

curl from the same pod is the fastest way to isolate whether the problem is network or application.

The CF-RAY: - diagnostic playbooktext
1. curl from the same pod
   -> gets real Ray ID? Problem is in the app, not the network.

2. Add -Djavax.net.debug=ssl:record:plaintext
   -> grep logs for "Plaintext before ENCRYPTION"
   -> read every header the app is actually sending

3. Look for:
   -> Duplicate headers
   -> Oversized headers
   -> Content-Length / Transfer-Encoding conflicts

4. Replay the exact headers via curl to confirm

5. Remove the diagnostic flag immediately after capture