#szejo ·
What actually happens to your data when your dev machine is the server
Symptom. Not a bug report — a question. Running a real personal infrastructure stack entirely on one Mac's local Kubernetes cluster raises an obvious but easy-to-defer question: if this machine died…
2 min read
Symptom. Not a bug report — a question. Running a real personal infrastructure stack entirely on one Mac's local Kubernetes cluster raises an obvious but easy-to-defer question: if this machine died today, what would actually survive?
Investigation. Traced one service's storage down to a PersistentVolumeClaim — Kubernetes' abstraction for "storage that outlives a pod restart." It is reasonable to assume that abstraction implies real durability. It does not, by itself: this cluster's storage class resolves to a folder inside the virtual machine the local Kubernetes runs in, not the host machine's real filesystem. Confirmed by trying to list the path directly on the host, where it does not exist at all.
Then went looking for the nightly backup job that was supposed to already cover this, expecting to confirm it worked. It did not exist. Its config file had been deleted during an unrelated repo reorganization weeks earlier and never rebuilt at the new location. The only reason this was not already a disaster was that nothing had gone wrong yet.
Root cause and fix. Rebuilt the backup job, and extended it to cover the one service it had never included in the first place. Verified it by actually running it on demand and checking the output existed — a config applying cleanly and a job doing its job are two different claims. Also pulled a manual copy of the data straight to the host as an immediate safety net while the recurring job was being rebuilt.
And honestly: even after all that, everything above still lives on the same machine. A backup that never leaves the computer it is backing up does not protect against the one failure mode that prompted the question. That part is still open, on purpose — it needs a real decision about where the off-site copy goes, not a default.
Lesson. "It is backed up" and "it is backed up somewhere that survives the thing you are actually worried about" are different claims. Periodically ask a system the blunt version of the question — not "does this have a backup job" but "if this exact machine vanished right now, what would I have left" — because the answer is rarely what the architecture implies.