Kubernetes Sidecar Readiness: When One Container Takes Down the Pod

When I opened weather.example.net, the page appeared. Only the data panel showed “불러오기 실패” (“Loading failed”) throughout.

/ always returned 200, while only /v1/* intermittently returned 503. The API container remained ready, and its restart count stayed at 0.

Another container in the same pod was dying. Because of that, the healthy API was removed from the Service endpoints.

The SPA Returned 200, but the API Endpoint Had Disappeared

It looked as though the entire site was down, but the result differed by path.

Request	Result
/	Always 200
/v1/*	Intermittent 503, "no available server"
SPA data panels	All showed "Failed to load"

/ served static files, while /v1/* passed through the API Service. The fact that the first screen opened and the fact that the API was alive were separate things.

The point where the paths diverged was a single Ingress. When I read it again today, I found six rules, five of which routed to the API.

# kubectl -n posy-weather get ingress posy-weather (2026-08-15 읽기)
weather.example.net/v1        → posy-weather        # API
weather.example.net/healthz   → posy-weather
weather.example.net/readyz    → posy-weather
weather.example.net/livez     → posy-weather
weather.example.net/metrics   → posy-weather
weather.example.net/          → posy-weather-web    # SPA 정적 파일

There were two Services behind one hostname. Since the browser address bar showed only one address, users experienced it as a single site. The screen appearing meant that the last rule was alive, while data arriving meant that the five rules above it were alive. They could fail independently.

The pod had three containers: api, nowcast-loop, and daily-loop. The request-handling server and the periodically running batch workers were in the same pod.

nowcast-loop was dying with exit 1. The CrashLoopBackOff backoff was five minutes, and it had restarted more than 56 times over eight hours.

One not-running container made the pod not-Ready

nowcast-loop had no readiness probe. Without a probe, it seemed that the pod would be unaffected because nothing asked whether that container was ready.

That was not the case. Even without a probe, the pod was not Ready if a container was not running. A probe asked whether a running container was ready to receive requests; the fact that execution itself had stopped was a condition that came before that.

Order	What happened	What was visible externally
1	nowcast-loop exit 1	The worker stopped
2	CrashLoopBackOff, 5-minute backoff	A not-running container appeared in the pod
3	Pod Ready=False	This happened even without a probe
4	0 Service endpoints	Traefik had nowhere to send traffic
5	/v1 503	The data panel cleared

I counted the endpoints directly during the crash window to confirm this. kubectl -n posy-weather get endpoints posy-weather showed 0 endpoints, and the pod was Ready=False.

api was also ready during that time and had 0 restarts. It was removed from traffic for a reason unrelated to its own failure.

nowcast-loop started and died 26 seconds later. Its startedAt was 20:20:15, and its finishedAt was 20:20:41. Because the backoff was 5 minutes, the pod became not-Ready for 5 minutes every 5 minutes. That was why the 503s appeared “intermittent.”

The reason I could not obtain a traceback was buffering

The command run by nowcast-loop is python -m posy_platform.cli nowcast-loop --store-dir /workspace/var/store --interval 300.

This is all that remained in the log.

STEPS 초기화
Rain fraction 0.0046
Extrapolation complete and precipitation fields aligned
(exit 1)

There was one warning immediately beforehand: x contains non-finite values. This meant that the radar input contained NaN or inf values.

There was no exact traceback. Python was buffering stdout, and PYTHONUNBUFFERED was not set. When the process died, the output still in the buffer disappeared.

So the furthest I could go was to say that it died "during the pysteps STEPS ensemble calculation." I did not know which exception occurred on which line. I also did not know what happened between the warning and exit 1.

Buffering is the default for performance. In containers, that default erases diagnostic information. When a process exits normally, the buffer is flushed, but a crash is not a normal exit.

I wrote down two options, and both went in later

The structure itself was the problem. A batch worker that died frequently and the server handling requests were in the same pod, with no isolation between them.

At the time, I only investigated. I did not change the cluster; instead, I wrote down two directions.

Option	What it changes	What remains
A	Move the worker out of the API pod into a separate Deployment	Manage worker deployment separately
B	Set PYTHONUNBUFFERED=1 to capture the traceback and handle non-finite input	Requires changing the deployment source

A cuts off the symptom. Even if the worker dies, api remains Ready. B goes after the cause. It makes the logs visible first and then fixes input handling.

That structure was already documented in the repository. It was item P2-4 in the operational readiness review document.

# clab_weather/docs/architecture/live-readiness-review-v1.md:67-70
P2-4. 단일 레플리카 / RWO PVC  (📋 = 미적용)
근거: replicas:1, strategy Recreate, RWO PVC를
      api + nowcast-loop + daily-loop 3컨테이너가 공유
영향: 노드 장애 = 서비스 전면 중단. 수평 확장 불가

The review recorded this as a scalability problem. 📋 indicated that it had not been implemented. I already knew that the three containers shared one volume, but it was not documented that this also coupled their readiness.

The same single line produced two consequences. The inability to scale hurt when traffic increased. The disappearance of the endpoint hurt when the worker died. The second one came first.

The namespace looks like this now

I looked again today. Both had gone in.

The Deployments had been split into three. posy-weather has only one api container. The workers had moved to posy-weather-loops, which contains four: nowcast-loop, daily-loop, weather-history-loop, and air-quality-history-loop. Even if a worker dies, the API endpoints remain available.

B had gone into the worker containers' environment variables.

PYTHONUNBUFFERED=1        # 크래시 때 출력을 잃지 않는다
PYTHONFAULTHANDLER=1      # 치명적 시그널에도 스택을 찍는다
OMP_NUM_THREADS=1         # 수치 라이브러리의 스레드를 하나로
OPENBLAS_NUM_THREADS=1
MKL_NUM_THREADS=1
NUMEXPR_NUM_THREADS=1

PYTHONFAULTHANDLER was an additional change I had not expected. It leaves a stack trace even when the process dies from a signal rather than an exception. If it dies inside a numerical computation library, it can end with a segmentation fault rather than a Python exception.

The thread count had also been pinned to 1. On a single-node cluster, if a numerical library uses all the cores, other pods on the same node are starved.

KMA_DAILY_CALL_CAP also differed by worker. It was 2,000 for nowcast and 12,000 for daily. These values divided the external weather API's daily call limit by worker.

When I placed the four workers side by side, their intervals and limits were all different.

Worker	Interval	Daily call limit	Thread limit
nowcast-loop	300 seconds	2,000	1 for all four
daily-loop	43,200 seconds	12,000	1 for all four
weather-history-loop	86,400 seconds	None	None
air-quality-history-loop	86,400 seconds	None	None

Only nowcast-loop ran every five minutes. The other three ran every 12 or 24 hours. The worker that ran every five minutes was the one that had initially been dying. The frequently running worker died frequently, and it had been in the same pod as the request-handling server.

POSY_DAILY_MIN_REFRESH_SECONDS=21600 was attached to daily-loop as well. Its interval was 12 hours, but its minimum interval was 6 hours. Because the loop started from the beginning every time the pod came back up, repeated restarts could cause it to fetch the external API again while ignoring the normal interval. That lower bound had been set to 6 hours.

The thread limit applied to only two of the four. It was attached to the numerical workers, nowcast-loop and daily-loop, but not to the two workers that only fetched and recorded history.

However, the same variables were also in the image. I found the deployment source and opened the Dockerfile; they appeared from the first line.

# clab_weather/Dockerfile:3-9
ENV PYTHONUNBUFFERED=1 \
    PYTHONFAULTHANDLER=1 \
    OMP_NUM_THREADS=1 \
    OPENBLAS_NUM_THREADS=1 \
    MKL_NUM_THREADS=1 \
    NUMEXPR_NUM_THREADS=1 \
    VECLIB_MAXIMUM_THREADS=1

There were seven. That was one more than the six in the Deployment. VECLIB_MAXIMUM_THREADS limited the macOS Accelerate-family backend; it was in the image but not in the Deployment.

Because the same values were written in two places, changing only one would make them diverge. If the thread limits were later removed from the Deployment, the values in the image would remain and the behavior would not change. The person reading the manifest would then think there was no limit, while the container would still run with one. What I confirmed this time was only that the values in both places pointed in the same direction.

Splitting them also settled two more things

When I read the manifest, I saw that the separation was not only for readiness. The values below were the ones recorded in the repository. The current live values are in section 6.

	posy-weather	posy-weather-loops
Replicas	2	1
Rollout strategy	RollingUpdate, maxUnavailable: 0	Recreate
Store mount	Read-only	Write
probe	/readyz, /livez	None

Because the API only read from the store, it could run with two replicas and no downtime. With maxUnavailable: 0, the old pod went down only after the new pod was ready.

The worker was the opposite. It had to have one replica. The store was RWO, and there had to be only one writer. That was why the strategy was Recreate.

The reason for the separation remained in three lines of comments in the manifest.

# deploy/k8s/posy-weather.yaml:83-96
# Single-writer loops: exactly one pod owns store writes (RWO local-path).
# Kept separate from the api Deployment so api rollouts/replicas never
# duplicate the KMA-fetching loops (quota) or race store writes.
spec:
  replicas: 1
  strategy:
    type: Recreate            # never two writers, even during a rollout

The comments gave two reasons: quota and competing writes. Readiness was not mentioned. Readiness was the symptom I encountered first, while the two reasons that became concrete when the split was made were the other two.

Keeping them in the same pod made it impossible to separate those two concerns. The moment I increased the API replicas to two, there would also be two copies of the loop fetching from the external weather API. With an API that had a daily call limit, that would quietly consume twice the quota.

The limits divided by worker were also correct only when there was one replica. The reason nowcast-loop had 2,000 and daily-loop had 12,000 was that their fetch ranges differed. The CLI help provided the basis for that. daily-loop fetched at 5 km grid-cell granularity and used a little over 10,000 calls per day. nowcast-loop used radar at the attempt level. The grid-scanning side received six times as much.

Readiness isolation had prompted the separation, while quota and a single writer were what came with it.

Same Namespace, but Only Half Is GitOps

The manifest is in the repository: deploy/k8s/posy-weather.yaml. That made me think the root from episode 20 would also read this file.

Today, I asked Argo CD directly. There is one Application watching the posy-weather namespace, and it manages three resources.

# kubectl -n argocd get application posy-weather-web (2026-08-15 읽기)
repo: <org>/clab-weather-web      # 웹 저장소
path: deploy/k8s
resources:
  Service     posy-weather-web
  Deployment  posy-weather-web       # SPA만
  Ingress     posy-weather           # 경로 여섯 줄이 여기 있다

posy-weather and posy-weather-loops were not in the list. When I read them again in the live cluster, they had no argocd.argoproj.io/instance label and had kubectl.kubernetes.io/last-applied-configuration attached. That indicated that someone had applied them with kubectl apply.

	posy-weather-web	posy-weather · posy-weather-loops
Applied by	Argo CD	A person
Repository	clab-weather-web	Another repository
Manually changed values	Reverted	Remain unchanged
If the file is deleted	Also disappears from the cluster	Nothing happens

The most concerning cell in this table was which side owned the Ingress. The Ingress with the six path entries was in the web repository, while the API Service targeted by those five entries was applied by a person. The routing rules and the targets they pointed to were under different management systems.

When I deleted the /v1 line from the Ingress in the web repository, Argo CD removed that line from the cluster within three minutes. The API remained alive, but no one could call it. Conversely, when I changed the port on the API Deployment, Argo CD did not know about it. A resource that is not in its list is not drift.

Having a declaration in Git and having the cluster follow that Git are different things. If this is moved to the structure from episode 20, these two Deployments will also be included in the list read by the root. For now, someone has to remember to apply them.

It was 0/0, but there was no red light anywhere

Today, all three Deployments were 0/0. I had not left the service running.

But there was nothing red on the screen.

What I asked	Answer
Deployment posy-weather	0/0, Available=True
Reason for the Deployment condition	MinimumReplicasAvailable
Argo CD Application	Synced, Healthy
Service posy-weather	Present
Endpoints posy-weather	<none>

When I scaled the replicas down to 0, the minimum replica condition was satisfied. Zero replicas had to be ready, and zero replicas were ready. Argo CD read the same value and displayed it as Healthy.

Zero endpoints had the same value as during a crash. The causes were opposite. At that time, the pods existed but were not Ready, so there were zero endpoints; now, there were zero endpoints because the pods themselves did not exist. Since the symptoms seen from outside were the same, I could not distinguish between the two by looking only at the /v1 503.

That was why I treated counting endpoints as the second line of diagnosis, not the first. I first counted how many pods there were, and then counted how many of them were Ready.