oauth2-proxy Shared Gate: Adding Auth via Traefik forwardAuth

I added one more line to the file. It was an email address.

That one line opened the four sites: baguette, Grafana, Alertmanager, and Prometheus. With this setup, there was no way to open only one and close the rest.

There Is One Gate, So There Is One Allowlist

There are two ways to add authentication in front of each service: add authentication to every service, or check once at the front and let requests through.

I used the latter. When Traefik receives a request, it asks oauth2-proxy before sending it to the original service. This method is called forwardAuth. If the authentication server returns 200, the request is allowed through; if it returns 401, the user is sent to the login page.

oauth2-proxy uses Google as its provider. I left the domain restriction open (email_domains = [ "*" ]) and used an email list for the actual allowlist.

# clab-cluster/argocd/applications/oauth2-proxy.yaml
authenticatedEmailsFile:
  enabled: true
  restricted_access: |-      # 이 목록이 전부다
    owner@example.com
    member@example.com

There are two lines. This change added the second line, and the commit was 4ab8d80.

The same file also contains several values that determine how the decision-maker behaves.

# 같은 Application의 인라인 Helm values
upstreams        = [ "static://202" ]                 # 뒤에 보낼 서비스가 없다
cookie_domains   = [ ".example.net", ".example.com" ]
whitelist_domains= [ ".example.net", ".example.com" ] # 이 밖으로는 리다이렉트 금지
cookie_expire    = "168h0m0s"                         # 7일
set_xauthrequest = true                               # 헤더에 사용자 정보를 실는다
skip-provider-button = "true"                         # Google 선택 화면을 건너뛴다

The first line defines this pod's role. static://202 means that there is no backend service to proxy to. This pod does not carry traffic; it only makes the decision. Traefik sends the request body directly to the original service.

cookie_expire is seven days. Even after an email is removed from the list, a cookie that was already issued remains valid for up to seven days. To revoke access, removing the email from the list is not enough; the cookie secret must be rotated.

There Are Three Sets of Middleware, but One Gate

Traefik Middleware is a namespace-scoped resource. A Middleware from another namespace cannot be attached to an Ingress. For that reason, I had to create one set for each namespace containing a protected service.

When I queried the cluster today, there were three sets: baguette, monitoring, and justsend-platform. When I first wrote this draft, there were two; in the meantime, the back-office console had been placed behind this gate.

All three sets call the same address.

forwardAuth:
  address: <in-cluster oauth2-proxy>/oauth2/auth

There are three Middleware resources, but only one decision-maker behind them. It is a single pod in the auth namespace, which had been Running for 24 days today with zero restarts.

That is why there is also one allowlist. To open baguette while keeping Grafana closed, I would have to create two decision-makers. The current structure does not allow this, regardless of how the files are divided.

I also read and compared the contents of all three sets one by one today. The four authResponseHeaders and the errors block were identical down to the characters in all three namespaces. They had been copied and pasted, so if another header needs to be added later, all three locations must be updated.

Protected host	Middleware location	Decision-maker
baguette.example.net	baguette namespace	oauth2-proxy.auth
grafana.example.net	monitoring	same pod
alertmanager.example.net	monitoring	same pod
prometheus.example.net	monitoring	same pod
admin.example.com	justsend-platform	same pod

Across the entire cluster, only a minority of services are behind the gate. There were 24 Ingresses today.

Type	Count	What
Behind the gate	5	baguette, Grafana, Alertmanager, Prometheus, back-office console
Login callbacks	5	The /oauth2 paths for the five above
Own login	1	Argo CD
Public	13	Product web, API, community, mail console, demo, and others

There are thirteen public ones. This gate does not guard the cluster's door; it protects five screens. Those five are important because they are observability tools and a back-office console, while the other thirteen are supposed to remain open.

argocd.example.net does not pass through this Middleware. Argo CD connects to Google OIDC directly and sets permissions with argocd-rbac-cm. Even if its Ingress is in the same directory, it uses a different gate. Having files together in a directory is not the same as passing through the same door.

When I read that ConfigMap today, policy.default was an empty string, and only one account had role:admin in policy.csv. Anyone could log in, but only one person had permissions.

There was one more line: scopes: "[groups, email]". By default, Argo CD checks only the JWT's groups and does not check email. Without this line, login succeeds but the Application list appears completely empty. The issue is not a lack of permissions; Argo CD is not reading the claim it needs to match.

The table growing from four to five reflects the nature of this structure. Sharing a gate keeps the cost of adding a service low. Adding two annotations to an Ingress is enough. In return, one email line has an equally broad scope. When another service is added next time, the same two lines will open six sites.

The cookie domains are .example.net and .example.com. After logging in once, users are not asked again under those domains. The scope is broad in proportion to the convenience.

The reason I split the middleware in two was 401

I had two middleware components attached to one Ingress. Their names were oauth2-errors and oauth2-auth.

oauth2-auth asked the authorization service for a decision. When the request passed, it forwarded four response headers to the original service. These values let the service know who had logged in.

# infra/oauth-gateway/monitoring-middleware.yaml
forwardAuth:
  address: <in-cluster oauth2-proxy>/oauth2/auth
  trustForwardHeader: true
  authResponseHeaders:            # 서비스가 받을 값 넷
    - X-Auth-Request-User
    - X-Auth-Request-Email
    - X-Auth-Request-Access-Token
    - Authorization

One of the four headers was actually used somewhere: Grafana.

# monitoring/kube-prometheus-stack-grafana ConfigMap의 grafana.ini
[auth.proxy]
enabled = true
header_name = X-Auth-Request-Email   # 이 헤더의 값을 사용자로 믿는다
header_property = username
auto_sign_up = true                  # 처음 온 사람은 계정을 만든다
whitelist = 10.42.0.0/16             # 클러스터 내부에서 온 요청만
sync_ttl = 60

Grafana did not display its own login screen. It trusted that the decision had already been made upstream and used the email in the header directly as the username.

whitelist was the safety mechanism in this configuration. 10.42.0.0/16 was the pod network for this cluster. Grafana trusted the header only for requests coming from that range. Without it, anyone could manually attach an X-Auth-Request-Email header and log in as anyone.

Both the service that trusted the header and the gateway that created it were necessary. If even one path reached Grafana directly without passing through the gateway, it was effectively unauthenticated. That was why the Grafana Service was a ClusterIP and the only door out was one Ingress.

Alertmanager and Prometheus did not have this layer. They had no built-in authentication at all, so the gateway was their only defense. I attached the same middleware, but the systems behind it had different characteristics.

oauth2-errors handled failures. It sent statuses from 401 through 403 to the login screen.

errors:
  status: ["401-403"]
  service:
    name: oauth2-proxy-local        # ExternalName 서비스
  query: /oauth2/sign_in?rd={url}   # 원래 가려던 주소를 들고 간다
  statusRewrites:
    "401": 302                      # 브라우저가 따라가도록 바꾼다

The reason this middleware existed was the last line. A 401 was a response that did not make the browser change screens. Changing it to 302 made the browser navigate to the login address.

rd={url} served the same purpose. The original destination had to be carried along so that the user would return there after logging in. Without it, the user logged in and then landed on the initial screen.

One manual patch remained in Traefik because of the ExternalName Service

oauth2-proxy-local, which oauth2-errors referenced, was a Service with no actual pods. It was an ExternalName pointing to a name in the auth namespace.

I created this workaround because the middleware could reference only Services in its own namespace. Traefik also rejected ExternalName Services by default. I therefore manually patched allowExternalNameServices: true once.

That patch was still outside GitOps. If I rebuilt the cluster, this gateway returned 502. The cause was not in the middleware configuration but in the Traefik provider option.

I kept the login path separate, outside the gateway

Each protected host had two Ingress resources. When I counted them that day, five protected hosts had five callback Ingress resources.

# monitoring/grafana-oauth2-callback (라이브 읽기)
rules:
  - host: grafana.example.net
    http:
      paths:
        - path: /oauth2                 # 이 접두사만
          backend: <in-cluster oauth2-proxy>
# 미들웨어 어노테이션이 없다

The key point of this file was that it had no middleware. The login screen and the Google callback were under /oauth2. If I applied authentication to that path, logging in would first require logging in.

Path	Middleware	Destination
grafana.example.net/	oauth2-errors, oauth2-auth	Grafana
grafana.example.net/oauth2/…	None	Decision-making pod

The same host name went to different destinations depending on the path. Because cookies were attached at the host level, the login screen and the protected screen had to use the same host. That was why I gave each host one path for the decision-maker instead of placing it on a separate domain.

When I added a host, I added two annotations and one Ingress resource. If I omitted the callback Ingress, the screen went to login, but the login address returned 404.

I Received Certificates Separately for Each Host

I added one annotation line to the Ingress, and cert-manager obtained the certificate.

annotations:
  cert-manager.io/cluster-issuer: letsencrypt-prod
tls:
  - hosts: [grafana.example.net]
    secretName: grafana-example-net-tls   # 호스트 하나에 secret 하나

The ClusterIssuer uses the HTTP-01 solver. Let's Encrypt checks http://호스트/.well-known/acme-challenge/... (host/...), and the request must reach our Traefik for issuance to succeed.

This meant that the host had to point to us through public DNS, with port 80 open. As the number of hosts increased, the number of certificates issued increased accordingly.

I counted all the Certificates in the cluster today. There were eighteen, all using letsencrypt-prod, and all Ready. Each one handled a single host, with the host included directly in its name.

Wildcards Follow a Different Path

There was one Ingress in the same cluster that used a wildcard certificate: *.rpc.example.org.

	Host	secret	Created by
18 cert-manager Certificates	grafana.example.net, etc.	One per host	cert-manager, HTTP-01
RPC gateway	*.rpc.example.org	One rpc-wildcard-tls	A person with kubectl apply

The RPC side created a subdomain for each app. Because the gateway authenticated with a bearer key rather than checking the host, any subdomain went to the same service. In a structure where hosts continued to increase, using host-specific issuance meant obtaining a certificate for each new app.

Mixing the two paths caused confusion. When the hosts were fixed, I obtained certificates for each host; when the number of hosts increased, I used a wildcard certificate.

There Was No Renewal Owner for the Wildcard

While checking the wildcard, I found that it was not present in the Certificate list. Only the secret existed; there was no Certificate known to cert-manager.

I extracted and read the certificate from the secret.

# kubectl -n fp-router get secret rpc-wildcard-tls → openssl x509 (2026-08-15)
subject   CN=*.rpc.example.org
issuer    Let's Encrypt YE2
notBefore Jul 15 08:12:12 2026 GMT
notAfter  Oct 13 08:12:11 2026 GMT

Because it was from Let's Encrypt, its validity period was 90 days. This secret had also been added manually with kubectl apply on 2026-07-15. It had a last-applied-configuration annotation and no cert-manager labels.

Wildcard certificates could not be obtained with HTTP-01. Wildcards required DNS-01, and DNS-01 required adding the DNS provider's API credentials to the cluster. Since those credentials had not been added, the certificate had been obtained manually and inserted manually.

Eighteen certificates renewed automatically, while one depended on someone remembering to renew it. It expired on October 13, and today was August 15.

It took five steps for one line to reach the pod

I fixed the file and pushed it. That was not the end.

Step	What happened
1	The push reached main
2	The root Application synced to 4ab8d80
3	The oauth2-proxy child Application re-rendered the inline Helm values
4	The ConfigMap oauth2-proxy-accesslist in the auth namespace was updated
5	The Helm chart's config-checksum changed, and the pods rolled

The reason for the five steps was that the pod read the file when it started. When only the ConfigMap was updated, an already-running pod still held the old list. The chart enforced this by putting the configuration hash in the pod template.

The new pod started at 07:48:01. The logs showed using authenticated emails file /etc/oauth2-proxy/oauth2-proxy-accesslist and the watcher registration.

The key name changed once along the way

I read that ConfigMap again today. The list still had two lines. However, the key name differed from the one written in the values.

# Helm values에 적은 것
authenticatedEmailsFile.restricted_access: |-
  owner@example.com
  member@example.com

# 클러스터의 ConfigMap oauth2-proxy-accesslist
data.restricted_user_access: "owner@example.com\nmember@example.com"

I had written restricted_access, but it appeared as restricted_user_access. The chart renamed it while rendering.

This was where verification could get stuck. When I queried the ConfigMap using the name written in the values, nothing appeared. It was the second trap that made it look as though the change had not been applied, and its cause differed from the earlier 180 seconds.

There was an interval in the middle that looked false

When I checked step 4 before step 2, I saw the old list. The root Application's polling interval was 180 seconds.

I had pushed, the file was correct, and the ConfigMap was old. If I concluded at that point that “it had not been applied,” I would start looking for a problem that did not exist.

When I checked again after the root sync finished, it had been applied immediately. So I changed the order of verification. I checked the root revision before checking the ConfigMap.

I Read Outside the Container with distroless

I tried to check the file directly inside the pod. exec cat did not work.

The oauth2-proxy image was distroless. It contained neither a shell nor cat. The image was built that way to reduce the attack surface.

So I checked three things from outside. I reread the ConfigMap, read the Helm values of the live Application, and looked through the pod logs for the line indicating that the process had opened the file.

The three were each different layers. The ConfigMap was the declaration, the Helm values were the rendered result, and the logs showed what the process had actually done. When all three reported the same value, it provided the same level of confidence as running cat inside.

The execution environment determined the verification method. When I chose an image without a shell, I also decided how I would verify it.