ArgoCD App-of-Apps: What sync-wave and selfHeal Cannot See

When I changed a value by hand in the cluster, the declaration in the repository diverged from the actual state. No one knew they had diverged.

So I treated the repository as the source of truth. But even when the repository is the source of truth, nothing happens if one line is missing from the list. That was what happened when I attached a new node and received no metrics at all. The node was Ready, and the exporter pods were 13/13.

I manually applied one root Application

Argo CD reads a Git repository and applies Kubernetes resources. This approach, in which the declarations in the repository become the state of the cluster, is called GitOps.

The first step has to be applied by hand. The component that reads the repository does not exist yet.

I created one Application named root. It is not a service. It reads the argocd/applications directory and creates the Applications in it as children.

# clab-cluster/argocd/root-app.yaml
spec:
  source:
    path: argocd/applications   # 이 디렉터리를 목록으로 읽는다
    directory:
      recurse: false            # 바로 아래 파일만 본다
  syncPolicy:
    automated:
      prune: true               # Git에서 사라진 것은 클러스터에서도 지운다
      selfHeal: true            # 손으로 바꾼 값은 Git 쪽으로 되돌린다

recurse: false can be confusing. It means that subdirectories are not scanned. root sees only the files in that directory, and each file specifies its own path separately.

prune deletes Applications removed from the repository from the cluster as well, and selfHeal restores manually changed values to the Git declaration.

This structure is called app-of-apps. Adding one file adds one service. The parent does not need to be changed.

I counted five things left outside GitOps

Manual work not managed by GitOps is documented in the README. I counted them again one by one this time and also checked them in the live cluster.

What	Why it is outside	Live today
First apply of root-app.yaml	The component that reads the repository does not exist yet	Application root exists
server.insecure in argocd-cmd-params-cm	Traefik terminates TLS	"true" confirmed
Traefik HelmChartConfig	The default k3s Traefik rejects ExternalName	allowExternalNameServices: true confirmed for both providers
kube-prometheus-stack CRD	It hits the 256KiB annotation limit	Not checked
OAuth client secret	It is not committed to Git	Not checked

The third line was interesting. The contents of that patch exist as a file in the repository. It is bootstrap/traefik-helmchartconfig.yaml, and the README says, “For reference; already applied to k3s.”

Having a declaration in the repository and having root apply it are different things. bootstrap/ is not the argocd/applications that root reads. The file exists, but the component that applies it is a person. Reading the repository and concluding, “This is managed by GitOps,” would be wrong.

The fourth line is not a one-time task. I excluded the CRDs from the chart with helm.skipCrds: true and added them separately with kubectl apply --server-side. The same command has to be run again each time the chart version is upgraded. This manual task returns with every upgrade; it is not just part of bootstrap.

If I rebuild the cluster from scratch, I have to remember these five things. Other than writing them in the README, I have done nothing to automate them.

I divided eleven child Applications into three sync-wave layers

There are eleven YAML files in argocd/applications/. There is no ApplicationSet. The list is committed by a person rather than generated.

sync-wave is a number that determines the application order. The next layer starts after the lower layer has finished.

wave	Application	Source	Version	Namespace
0	cert-manager	Helm charts.jetstack.io	v1.16.2	cert-manager
0	kube-prometheus-stack	Helm prometheus-community	66.3.1	monitoring
0	oauth2-proxy	Helm oauth2-proxy	7.9.2	auth
1	cluster-issuer	Git infra/cluster-issuer	main	cert-manager
1	oauth-gateway	Git infra/oauth-gateway	main	monitoring
1	observability	Git infra/observability	main	monitoring
2	baguette	Git workloads/baguette	main	baguette
2	fp-router	Git workloads/fp-router	main	fp-router
2	hello-clab	Git workloads/hello-clab	main	hello-clab
2	postal	Git workloads/postal	main	postal
2	stalwart	Git workloads/stalwart	main	mail

The reason there are three layers is the direction of the references. Wave 0 installs the charts. Wave 1 places resources on top of the CRDs created by those charts. The reason is written on the first line of observability.yaml: it is placed on the Operator CRDs installed by kube-prometheus-stack. Applying those resources first to a cluster without the CRDs fails.

Wave 2 contains the workloads. The certificates and authentication gateway have to be in place before they can be attached.

All eleven use prune: true, selfHeal: true, and CreateNamespace=true. kube-prometheus-stack and observability also have ServerSideApply=true. The reason is written in a comment on the observability side. The ConfigMap containing the dashboard JSON is large, so client-side application can hit the 256KiB limit of the last-applied-configuration annotation.

A single push reaches the cluster through three stages

The deployment flow documented in the README is short.

# clab-cluster/README.md:80-82
1. workloads/ 또는 argocd/applications/ 변경 → main 브랜치 push
2. Argo CD가 저장소를 polling(기본 3분) 또는 webhook으로 감지, 자동 sync
3. git revert로 되돌리면 클러스터 상태도 자동으로 되돌아온다

The third line is why I use this structure. git revert does not delete a file; it makes the previous declaration into a new commit. When Argo CD syncs that commit, the cluster returns to the same state.

The way to revert is the same as the way to deploy. I do not memorize a separate rollback procedure.

Reconciliation runs every 3 minutes, while sync runs only when something changes

When I opened the status of root today, the two timestamps were different.

Value	Time
reconciledAt	2026-08-15 04:48
Time the last sync finished	2026-08-07 14:33

The first value is updated every 3 minutes. Argo CD reads the repository again and compares it with the cluster. If they are the same, nothing happens. Sync occurs only when they differ, and the last such event was 8 days ago.

There is no timeout.reconciliation key in argocd-cm. Therefore, 3 minutes is Argo CD's default value. The README saying “default 3 minutes” means that the setting was not added, not that it was configured.

The same ConfigMap contains resource.exclusions, whose list includes Endpoints and EndpointSlice. Argo CD does not look at these at all. A pod being removed from an endpoint is not Argo CD's concern; a person checks that. The reason the Application remained Healthy when the API was removed from traffic in Part 23 is in this one line.

The exporter was healthy, and one line was missing from the list

I attached a new worker node, aks-node-02. Its metrics did not appear in Grafana. aks-node-01 appeared normally.

This incident occurred in another cluster, not the clab cluster. That cluster uses an ApplicationSet. The structure differs, but the problem of how the list is managed is the same.

I only read the configuration. I changed nothing. This was the order in which I checked it.

What I checked	aks-node-01	aks-node-02
Node status	Ready	Ready (joined 26 hours ago)
exporter DaemonSet	13/13 Ready	Pod ovh-node-exporter-sm8dp 2/2 Running
PodMonitor	vg-aks-node-01-node-exporter exists	None
ArgoCD Application	vg-aks-node-01 exists	None
Generator input file	values/nodes/aks-node-01.yaml exists	None
Prometheus series	Exists	None

The two lines at the top were normal. Both the node and the pod were alive. Starting at the third line, nothing existed.

A pod being ready and Prometheus scraping that pod are different facts. To scrape it, a scrape target has to be declared, and that declaration is the PodMonitor.

The generator used a manually written list, not a glob

I traced upward to find out why there was no PodMonitor. There was no Application to create it, and the ApplicationSet that creates Applications did not have that node in its inputs.

The ApplicationSet git-files generator was not a pattern. It was a list containing thirteen paths written out as-is.

# aks-cluster/k8s/argocd/apps/10-observability.yaml:351-377
values/nodes/aks-node-01.yaml
values/nodes/ovh-node-1.yaml … ovh-node-9.yaml
values/nodes/ws-node-1.yaml
values/nodes/la-node-1.yaml
values/nodes/cherry-node-1.yaml

aks-node-02.yaml was missing. Even if a new file is created in the repository, no Application is created unless its path is added to this list.

The file itself was missing too. values/nodes/ contained only aks-node-01.yaml, and there was no new node entry in catalog.yaml. Fixing only the list would not have been enough.

I finished by querying the central Prometheus directly. In up{node=~"aks-node-01|aks-node-02"}, only the series for 01 appeared. This matched what I saw in Grafana.

The cause was neither the exporter nor the network. Two files had been omitted from the onboarding procedure. At the time, approval from the user was still pending, so I investigated but did not fix it.

selfHeal restores drift but cannot find omissions

The two structures catch different things.

	State different from the declaration	State where the declaration itself is missing
What catches it	selfHeal	Nothing
How it appears	Argo CD restores it	Silently
How long it takes	Within 3 minutes of polling	Until someone notices

selfHeal protects only what it knows about. An undeclared resource is not something it is responsible for protecting. Absence is not drift.

In the aks-node-02 incident, every layer was green. The Application was Synced and Healthy. Only one node, absent from the list, was missing.

When automation holds its list by hand, that list becomes a person's memory. Memory omits things.

The root in the clab cluster reads a directory. Adding a file updates the list automatically. The other cluster, where I attached the new node, has paths written by hand. It is the same GitOps, but in one case creating a file is enough, while in the other two places have to be changed.

There were two roots, and one list had no ordering

When I queried the cluster for Applications today, there were 23. That was more than the eleven I counted earlier.

There was another parent besides root: clab-app-root. The two parents had the same shape. Only the repositories they read were different.

	root	clab-app-root
Repository	clab-cluster	clab-app
Path read	argocd/applications	argocd/applications
recurse	false	false
Child count	11	10
prune·selfHeal	Both true	Both true
sync-wave	0, 1, or 2 for each child	None

11 plus 10 is 21, and adding the two parents makes 23. The two parents themselves are Applications, so they also appear in the list. All 23 were Synced and Healthy today.

The last line is the difference between the two lists. The eleven children of clab-cluster have sync-wave values, while none of the ten children of clab-app do.

There is a reason this is not necessary. The clab-cluster list contains certificates, the authentication gateway, and observability tools. They depend on one another. The clab-app list contains product services, and they do not depend on one another. The backend does not need to come up before the console.

clab-app has another characteristic instead. Its ten entries point to seven repositories.

Application	Repository	Path
discourse, plane	clab-app	workloads/…
justsend-backend, justsend-console, justsend-ops	justsend-platform	backend, console, ops
justsend-render	justsend-render	.
justsend-share-web	justsend-share-web	.
justsend-web	justsend-web	.
posy-weather-web	clab-weather-web	deploy/k8s
test-agents	test-agents	.

There are four path: . entries. This is a layout in which manifests are placed at the repository root and one repository corresponds to one service. When adding a service, this is where the choice arises between creating a new repository and creating a directory inside an existing repository; here, both patterns are mixed.

Either way, adding one file to the list is the same. What differs is what comes next. With seven repositories, there are also seven places to revert with git revert. I have to remember which service is in which repository.

The copy on my disk was not the source of truth

While verifying this manuscript, I got two different lists twice.

There were two copies of clab-app on disk. One had justsend-ocr.yaml, justsend-ocr-normalizer.yaml, and justsend-web-app.yaml, while the other had discourse.yaml, justsend-ops.yaml, and justsend-render.yaml. They differed by three files each.

# 같은 저장소의 두 사본
clab/clab-app                 HEAD e810581 (2026-08-04)
justsend-platform/clab-app    HEAD 1fbd5d2 (2026-08-14)
origin/main                        1fbd5d2

The older copy was ten days behind. When I queried the cluster, discourse, justsend-ops, and justsend-render existed, while justsend-ocr did not. The newer copy was correct.

If I had written this manuscript using only the old copy, I would have recorded three services as absent and three absent services as present. Since the file list is the service list, a wrong copy makes the entire list wrong.

The source of truth protected by selfHeal is main in the remote repository. It is not the checkout on my laptop. Argo CD looks at what it fetched, while a person looks at the directory they opened. There is no signal anywhere when the two diverge.

So when I count a list, I check three places together: the remote main, the Applications in the cluster, and the directory I am reading. I wrote the numbers in this manuscript only after all three matched.