← sz3yan.com

#szejo ·

The one missing YAML line that made half our chat infra disappear

Symptom. Two unrelated-looking features broke at once, on two different apps, days apart — a "Create Workspace" page in one internal tool, and a chat sidebar's live connection in another. Both errors…

2 min read

Symptom. Two unrelated-looking features broke at once, on two different apps, days apart — a "Create Workspace" page in one internal tool, and a chat sidebar's live connection in another. Both errors identical: the WebSocket connects, then dies immediately, no useful message.

Investigation. The instinct is always to blame the newest thing you touched, or the flakiest-looking layer — here, that meant suspecting Cloudflare first, since its WebSocket support has a literal on/off toggle. Checked the API, confirmed it was on. Reproduced the exact same request twice — once as an ordinary GET, once as a WebSocket-upgrade request — against the reverse proxy directly, bypassing Cloudflare entirely. The plain GET succeeded. The WebSocket version failed with a bare, bodyless 403: no error page, no log line, nothing to grep for.

That absence was the actual clue. The proxy's auth-check layer keeps its own counters for every request it allows or denies, and after the failed WebSocket attempt, none of those counters had moved. The rejection was happening before the request ever reached the part of the system that could explain why.

Root cause and fix. A modern reverse proxy is often described as allowing WebSocket upgrades "by default" when you don't configure anything special. In practice, on this build, that default didn't hold — the connection manager needed one explicit line saying WebSocket upgrades were allowed at all, or it killed them at the protocol level, before any of the proxy's actual logic (auth, routing) ever ran. Adding that one line fixed both unrelated-looking bugs at once, because they had always been the same bug.

Lesson. When a failure produces zero evidence anywhere in the stack you would normally check, that is not bad luck — it is a clue that the failure is happening earlier than every place you are looking. Move the investigation one layer further upstream before adding more logging to the layers you have already ruled out.