Saurabh Singh
Journal

15 min readSaurabh Singh

Whose Identity Does the Tool See? Authenticating MCP in Production

Four ways to decide which identity sits behind an agent's tool call on Kubernetes, what each one costs, and how to layer them.

  • #mcp
  • #security
  • #kubernetes
  • #oauth
  • #agents

It is 6:42 on a Friday evening. An engineer pushes the checkout release, then asks the release copilot to keep an eye on it.

The copilot is an internal agent. It can do three things: read CI build logs, query the metrics API, and open tickets. So it watches. At 6:51 the error rate on the payments tier doubles. The copilot reads the failing build log, pulls the last ten minutes of latency, decides a human should know, and opens an incident ticket. The ticketing system pages whoever is on call.

The engineer rolls back. By 7:30 the graphs are flat and everyone goes home. The system did exactly what it was built to do.

Now jump to Monday morning. Someone opens that ticket to write the postmortem and looks at the reporter field. It says release-copilot.

That is all it says. Which engineer asked for the watch? Was that engineer even allowed to page the payments tier? If a new intern had typed the same sentence, the ticket would look identical. The credential was valid, the ticketing API accepted it, and the audit log is complete and almost useless.

The rest of this post is about fixing that one line. The copilot and the scenario are invented, but every tradeoff in it is real. It all comes down to a question worth writing down:

When an agent calls a tool, whose identity should the system behind that tool see?

Why this is harder than it looks

Your first reaction might be that this is ordinary service-to-service auth. You write down that service A may call endpoint B, and you are done.

The copilot breaks that in two ways.

First, it picks its own tools. On Friday it went logs, then metrics, then ticket. On Saturday it might go metrics, then nothing. You cannot predict the sequence from the prompt, so a fixed list of allowed endpoints either blocks real work or allows far too much. The decision has to be made on each call, close to the tool.

Second, every call has two parties behind it. A human asked, and an agent acted. The ticketing team may want to enforce what the human is allowed to do. The platform team may want to limit and audit the agent as its own thing. Security wants both names in the same row.

Every approach below is a different answer to the same two questions: who does the upstream system see, and who vouches for that? The strips below are a map of all four. Read each lane left to right and watch where the credential changes. A lane where it never changes deserves suspicion.

Hop strips: where the credential changes Four lanes, one per approach, showing user, agent runtime, MCP server and upstream API with the credential carried on each hop. WHERE THE CREDENTIAL CHANGESUserAgent runtimeMCP serverUpstream API1 Forward tokenhumanagentvalidatesprotecteduser tokensame token, forwardedsame token, forwardedno swap: replay risk2 Workload identityhumanagentvalidatesprotecteduser id (audit only)SA token, aud = MCPMCP's own SA token, aud = upstreamswap at MCP3 Token exchangehumanagentvalidatesprotecteduser tokensub: user, act: agentsub: user, act: MCPswap at both hops4 SPIFFE mTLShumanagentvalidatesprotecteduser id claimmTLS, SVIDmTLS, SVIDnew identity per hop
One lane per approach: the credential on each hop, and where it is swapped.

Before trying anything, it helps to know the rules of the road.

First, what the protocol already says

MCP, the protocol the copilot uses to reach its tools, has an authorization spec and a security best practices page. I read both as of the 2026-07-28 revision, the current "latest" at the time of writing. The spec moves, so check the revision you actually target.

Four things matter for us:

  • Authorization is optional and applies to HTTP transports. For stdio the spec says to take credentials from the environment instead.
  • An MCP server acts as an OAuth resource server. It must publish RFC 9728 protected resource metadata, which is how a client finds out where to get a token.
  • Clients must send an RFC 8707 resource parameter naming the MCP server. The server must reject any token that was not issued for it. In plain words: a token has an audience, the one service it is addressed to, and a service must refuse tokens addressed to someone else.
  • When the MCP server calls an upstream API, it needs a separate token from the upstream's own authorization server. It must not pass through the token it received. The best-practices page calls passthrough an anti-pattern: it bypasses rate limits and validation, muddies audit trails, and breaks trust boundaries.

Keep that last rule in mind. The most obvious fix runs straight into it.

Attempt one: forward the user's token

In plain words: the agent borrows the engineer's login.

The engineer is already signed in. The agent runtime holds their access token. So forward that token on every hop. The upstream sees the human, your existing roles, quotas and audit rows keep working, and nothing upstream changes.

It feels like a win for about a week. Then the problems show up, one at a time:

  • It expires mid-task. Access tokens live minutes to hours. A long rollout watch outlives one, so the agent runtime now needs refresh logic.
  • It needs too much power. One token has to cover logs, metrics and ticketing, so the engineer's token carries scopes for all three. That is the opposite of least privilege.
  • It can be replayed. If the MCP server is ever compromised, the attacker holds a credential that works anywhere the engineer's token works.
  • It has the wrong audience. A token addressed to the ticketing API is not addressed to the MCP server. Accepting it skips the check the spec requires, and forwarding it onward is exactly the passthrough the spec forbids.

That last one ends the experiment, but it leaves something useful behind. Accepting a user's token at the MCP server is correct and required, as long as it was issued for that server. Relaying the same token upstream is the part that is common and discouraged. What you may do is accept the audience-bound token, then get a different credential for the upstream hop. That idea comes back in attempts two and three.

Here is the inbound half done properly, with jose. It is illustrative, not a drop-in.

import { createRemoteJWKSet, jwtVerify } from "jose";

const jwks = createRemoteJWKSet(
  new URL("https://idp.example.org/.well-known/jwks.json"),
);

export async function verifyUserToken(bearer: string) {
  const { payload } = await jwtVerify(bearer, jwks, {
    issuer: "https://idp.example.org",
    audience: "https://release-mcp.example.org/mcp", // this server, not upstream
    algorithms: ["RS256", "ES256"],
  });
  return payload; // payload.sub is the human
}

If you already sign users in at the edge, an auth proxy such as oauth2-proxy behind NGINX auth_request can hand the access token to your backend. With Istio, RequestAuthentication validates JWTs at the mesh edge, but by itself it does not reject requests that carry no token. Pair it with an AuthorizationPolicy that requires a request principal.

Whatever else you change, keep three things from this attempt: short lifetimes, mTLS between hops, and an audience on every token.

So if the user's token cannot travel, what can?

Attempt two: let Kubernetes vouch for the workload

In plain words: stop asking who the human is. Prove which program is calling.

Everything here runs in clusters you own, so why involve the identity provider at all? Every pod already has a ServiceAccount, and Kubernetes will mint it a signed, expiring token. There are two ways to use that:

  • Server identity. The MCP server calls upstream as itself. Every agent using that server gets the same upstream permissions. Simple.
  • Agent identity. The agent presents its own token, the MCP server checks it, and the upstream can tell the copilot apart from, say, a docs agent.

The second variant has the same trap as attempt one. A token aimed at the MCP server is not a token for upstream, and relaying it unchanged is passthrough by another name. There are two clean options. The MCP server can verify the agent and then tell the upstream who it was over mTLS. Or the agent can request a second token aimed at the upstream, which the server forwards. Either way, each token has exactly one recipient.

Now for the details that decide whether this works.

Ask for a token aimed at the right service. The kubelet can project a token with a specific audience. The projected volume docs put the minimum expirationSeconds at 600, and the kubelet refreshes the file before it expires.

apiVersion: v1
kind: Pod
metadata:
  name: release-copilot
  namespace: copilot-agents
spec:
  serviceAccountName: release-copilot
  containers:
    - name: agent
      image: registry.example.org/release-copilot:1.4.0
      volumeMounts:
        - name: mcp-token
          mountPath: /var/run/secrets/mcp
          readOnly: true
  volumes:
    - name: mcp-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              audience: https://release-mcp.example.org/mcp
              expirationSeconds: 900

Read the file on every call. Because the token rotates, one cached at startup goes stale. Reading a small file per request costs almost nothing.

import { readFile } from "node:fs/promises";

export async function callMetrics(path: string): Promise<Response> {
  const token = (await readFile("/var/run/secrets/upstream/token", "utf8")).trim();
  return fetch(`https://metrics.internal.example.org${path}`, {
    headers: { authorization: `Bearer ${token}` },
  });
}

Check the token with TokenReview. The receiving side asks the API server whether the token is genuine. Pass audiences so the check is bound to you, and filter by namespace as a coarse first gate. This uses the current @kubernetes/client-node, where calls take a body parameter.

import {
  AuthenticationV1Api,
  AuthorizationV1Api,
  KubeConfig,
} from "@kubernetes/client-node";

const kc = new KubeConfig();
kc.loadFromCluster();
const authn = kc.makeApiClient(AuthenticationV1Api);
const authz = kc.makeApiClient(AuthorizationV1Api);
const ALLOWED_NS = new Set(["copilot-agents"]);

export async function authenticateAgent(token: string) {
  const review = await authn.createTokenReview({
    body: { spec: { token, audiences: ["https://release-mcp.example.org/mcp"] } },
  });
  const status = review.status;
  if (!status?.authenticated || !status.user?.username) {
    throw new Error("unauthenticated");
  }
  const [, kind, ns, name] = status.user.username.split(":");
  if (kind !== "serviceaccount" || !ALLOWED_NS.has(ns)) {
    throw new Error("namespace not allowed");
  }
  return { username: status.user.username, ns, name };
}

Then decide what it may do with SubjectAccessReview. Knowing who is calling is not the same as knowing what it may do. Kubernetes RBAC can answer that for your own application-level verbs too. Invent an API group for your domain and write rules for resources that never need a CRD, because they exist only to be asked about.

One trap: RBAC on the Kubernetes Service object protects the object, not the traffic to it. Model the real permissions as your own resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: incident-opener
  namespace: release-tools
rules:
  - apiGroups: ["copilot.authz.example.org"]
    resources: ["incident-tickets"]
    verbs: ["create"]
  - apiGroups: ["copilot.authz.example.org"]
    resources: ["build-logs", "metrics"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: copilot-incident-opener
  namespace: release-tools
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: incident-opener
subjects:
  - kind: ServiceAccount
    name: release-copilot
    namespace: copilot-agents

The check itself, reusing the authz client from the previous snippet:

export async function mayDo(username: string, verb: string, resource: string) {
  const review = await authz.createSubjectAccessReview({
    body: {
      spec: {
        user: username,
        resourceAttributes: {
          group: "copilot.authz.example.org",
          resource,
          verb,
          namespace: "release-tools",
        },
      },
    },
  });
  return review.status?.allowed === true;
}

This suits the copilot well. The decision happens on every tool call, and the policy is plain data your platform team already knows how to review.

Outside the cluster. Suppose ticketing runs in a different environment. The API server publishes issuer discovery at /.well-known/openid-configuration and keys at /openid/v1/jwks. A service elsewhere can then verify these tokens as ordinary JWTs, if you expose that endpoint. The audience you projected is what makes that safe.

const clusterJwks = createRemoteJWKSet(
  new URL("https://oidc.cluster.example.org/openid/v1/jwks"),
);

export const verifyClusterToken = (token: string) =>
  jwtVerify(token, clusterJwks, {
    issuer: "https://oidc.cluster.example.org",
    audience: "https://ticketing.example.org",
  });

It works, and it is pleasantly boring. But go back to Monday's ticket. It would now say that release-copilot opened it, more reliably than before, and it still would not say who asked. This is workload identity, not user identity. You can pass a user ID in a header, and you should, for audit. But nothing has verified it, so treat it as a log field. Never authorize on it.

What you want is one credential that names both.

Attempt three: a token that carries both names

In plain words: a signed note that says "this user, acting through this agent".

That is what RFC 8693 token exchange does. The agent runtime hands the identity provider the user's token (subject_token), its own token (actor_token), and the audience it wants. The provider returns a new token where sub is still the user and act names who is acting.

Delegation chain with token exchange Three boxes showing how the sub claim stays the user while the act claim changes from the agent to the MCP server across two token exchanges. ONE EXCHANGE PER HOP, THE USER STAYS IN subAgent runtimereceivesuser tokenaud: runtimeMCP serversub: useract: release-copilotaud: release-mcpUpstream APIsub: useract: release-mcpaud: incidents-apiexchange #1exchange #2Policy at the upstream API:allow if sub may open incidents AND act is an approved caller for this route
With token exchange the user stays in sub while act changes at every hop.

Now a rule like this becomes possible: this user may open incidents, and this caller is an approved route for doing it. A compromised agent cannot exceed what the user may do, and a stolen user token cannot be replayed through an unapproved caller.

It also fits the spec's "separate token upstream" rule. The exchange simply runs twice: agent to MCP server, then MCP server to upstream. The sub stays the human. Only act changes.

export async function exchange(
  subjectToken: string,
  actorToken: string,
  audience: string,
): Promise<string> {
  const res = await fetch("https://idp.example.org/oauth2/token", {
    method: "POST",
    body: new URLSearchParams({
      grant_type: "urn:ietf:params:oauth:grant-type:token-exchange",
      subject_token: subjectToken,
      subject_token_type: "urn:ietf:params:oauth:token-type:access_token",
      actor_token: actorToken,
      actor_token_type: "urn:ietf:params:oauth:token-type:access_token",
      audience,
    }),
  });
  if (!res.ok) throw new Error(`exchange failed: ${res.status}`);
  return ((await res.json()) as { access_token: string }).access_token;
}

actor_token_type is required whenever actor_token is present. It is an easy field to forget.

There are two catches. The first is support. Check your identity provider before designing around this. Keycloak documents standard token exchange (RFC 8693) as supported, but its delegation support is marked experimental and, per its docs, uses a may_act mechanism that differs from plain act semantics. I could not confirm actor-token delegation for other identity providers, so verify yours end to end with a real exchange.

The second is speed. Exchanging on every tool call puts a network round trip in front of each one, so you cache. Key the cache on (subject, actor, audience). Read exp from the new token and set the TTL to exp − now − skew, with something like 30 to 60 seconds of skew. Never cache beyond the token's own life.

When several requests miss the cache at once, don't build a distributed lock. Dedupe in flight inside each process and tolerate a few duplicate exchanges across replicas. Duplicates cost a request. A lock service costs an outage.

import { decodeJwt } from "jose";

type Entry = { token: string; expiresAt: number };
const cache = new Map<string, Entry>();
const inflight = new Map<string, Promise<string>>();
const SKEW_MS = 45_000;

export async function cachedExchange(
  subject: string,
  actor: string,
  audience: string,
  subjectToken: string,
  actorToken: string,
) {
  const key = JSON.stringify([subject, actor, audience]);
  const hit = cache.get(key);
  if (hit && hit.expiresAt > Date.now()) return hit.token;

  const pending =
    inflight.get(key) ??
    exchange(subjectToken, actorToken, audience)
      .then((token) => {
        const exp = decodeJwt(token).exp;
        if (exp) cache.set(key, { token, expiresAt: exp * 1000 - SKEW_MS });
        return token;
      })
      .finally(() => inflight.delete(key));
  inflight.set(key, pending);
  return pending;
}

The price is real. The token service is now on the hot path of every agent action: one more critical dependency, extra latency on each call, and failure modes you now own.

And all three attempts still share one soft spot.

Attempt four: make the credential impossible to steal

In plain words: stop handing out secrets that can be copied.

Tokens are bearer credentials. Whoever holds the string is the caller. My own custody work at CoinDCX was HSM-backed, where the point was that key material never leaves the device. SPIFFE is the closest thing workload auth has to that rule.

Here is the vocabulary in one breath:

  • A SPIFFE ID is a URI naming a workload, such as spiffe://example.org/ns/copilot-agents/sa/release-copilot.
  • An X.509 SVID is a certificate carrying that ID. Two workloads run mutual TLS with their SVIDs and check each other's IDs.
  • The SPIRE server is the certificate authority for a trust domain. A SPIRE agent runs on each node and serves the Workload API on a local Unix socket.
  • Attestation is how SPIRE decides which identity a process gets. On Kubernetes, the agent inspects pod properties and matches the selectors k8s:ns and k8s:sa against registration entries.
  • Rotation is automatic. SPIRE's server default SVID lifetime is one hour, and the default strategy rotates at half of lifetime, per its configuration reference.

Two practical upgrades make the rollout sane. Instead of a hostPath mount for the socket, the SPIFFE CSI driver bind-mounts it read-only into pods. Instead of hand-made registration entries, the SPIRE Controller Manager reconciles them from a ClusterSPIFFEID resource.

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: copilot-agents
spec:
  spiffeIDTemplate: >-
    spiffe://{{.TrustDomain}}/ns/{{.PodMeta.Namespace}}/sa/{{.PodSpec.ServiceAccountName}}
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: copilot-agents
  podSelector:
    matchLabels:
      app: release-copilot

The server side is Go, because go-spiffe is the SPIFFE project's own library. The X509Source keeps the certificate fresh in the background, and one option pins the exact peer you accept.

package main

import (
	"context"
	"log"
	"net/http"

	"github.com/spiffe/go-spiffe/v2/spiffeid"
	"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
	"github.com/spiffe/go-spiffe/v2/workloadapi"
)

func main() {
	src, err := workloadapi.NewX509Source(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	defer src.Close()

	agent := spiffeid.RequireFromString(
		"spiffe://example.org/ns/copilot-agents/sa/release-copilot")
	srv := &http.Server{
		Addr:      ":8443",
		TLSConfig: tlsconfig.MTLSServerConfig(src, src, tlsconfig.AuthorizeID(agent)),
	}
	log.Fatal(srv.ListenAndServeTLS("", ""))
}

The limits are worth stating plainly. SPIFFE identifies workloads, not people, so user context still has to travel some other way. And the SPIRE server is a root of trust. It deserves its own namespace, tight NetworkPolicies, a small admin group, backups, and alerts on unexpected attestation failures. The learning curve is real. You are buying a credential with no bearer secret to steal, and you pay in operations.

Four attempts, four partial answers. So which one wins?

None of them. You stack them.

Each approach covers a gap in another, so the real answer is layers:

  • SPIFFE mTLS on every workload hop, so the channel is authenticated both ways and nothing replayable crosses it.
  • An exchanged token in the request, so the human and the agent are both named, and the upstream gets a token minted for it.
  • A policy engine, such as OPA or Cedar, judging both identities together: allow only if the caller is this workload and this user holds this permission.

Each layer should fail closed on its own, so that a leaked token, a misrouted pod and a bad policy rule each need a separate mistake to hurt you. If token exchange is out of reach today, SPIFFE plus a user ID header is a legitimate stepping stone. Label that header as audit data in your design doc, so nobody builds authorization on it later.

Now replay the evening. Another Friday, another release. The copilot watches, the error rate wobbles, a ticket appears. This time the reporter field reads: opened by release-copilot, on behalf of the engineer who asked, who is permitted to page the payments tier.

That is the line we set out to fix.

What happens as you grow

Two things tend to show up later.

The first is server sprawl. Repeating auth logic in every MCP server gets tedious, so teams reach for a gateway that terminates authentication, enforces policy, rate-limits and logs in one place. A few projects I checked are real and public:

  • Microsoft's MCP Gateway, a reverse proxy and management layer for MCP servers on Kubernetes, with session-aware routing and Entra ID bearer-token auth and RBAC.
  • IBM's ContextForge, a gateway and registry that federates MCP servers and REST or gRPC services behind one endpoint, with JWT and OAuth support.
  • agentgateway, an open source proxy for MCP and A2A traffic with JWT and OAuth auth and CEL-based authorization policy.

The category is young and moving, so read each project's current docs before committing. The tradeoffs are structural: the gateway becomes a single point of failure that needs replicas and its own capacity planning, every request pays some latency, and a central policy plane becomes something many teams depend on. With a handful of servers I would keep the logic in the servers. With dozens, or multiple tenants, the gateway earns its place.

The second is an outsider. A partner's agent asks to call your ticketing API. Everything above assumed one organization, so now you need federation. Token exchange needs your identity provider to trust tokens from theirs. SPIFFE needs trust bundles exchanged between SPIRE deployments. Both work, and both add operational surface that internal-only setups never pay for. Decide who owns each side of that trust before the first integration call, not after.

The cheat sheet

Identity at each hop, by approach A matrix of four authentication approaches showing what the MCP server sees, what the upstream API sees, and the blast radius if the credential leaks. WHO THE OTHER SIDE THINKS IT IS TALKING TOAPPROACHMCP SERVER SEESUPSTREAM API SEESIF THE TOKEN LEAKSForward user tokenUser's access tokenThe same tokenWhole user session(and the spec forbids it)Workload identityAgent's ServiceAccountIts own SA, or the agent'sOnly that workload,until token expiryToken exchangesub = user, act = agentsub = user, act = MCPOne audience,short lifetimeSPIFFE mTLSAgent's SPIFFE IDMCP server's SPIFFE IDNo bearer token to replay;needs the private keyOnly token exchange puts both the human and the agent in one signed credential.
What each side of the MCP server sees under the four approaches, and what a leaked credential is worth.
ApproachUser attributionStolen-credential riskOperational costBest fit
Forward user tokenYes, nativelyHigh: replayable user sessionLowNothing; spec forbids relaying it upstream
Workload identityNo, audit header onlyMedium: one workload until expiryLowEverything in one cluster you control
Token exchangeYes, sub and actLow: narrow audience, short lifeHigh: IdP on the hot pathUser-level rules across systems
SPIFFE mTLSNo, workload onlyLowest: no bearer secret to replayHigh: SPIRE is a root of trustZero-trust hop-to-hop transport
A rough comparison. Your compliance regime decides which column matters most.

The tree below is how I would walk a team through the choice.

Decision tree for MCP authentication A decision tree that asks whether audits must name the human and whether the identity provider supports token exchange with an actor token, ending in a recommended layered setup. PICKING A STARTING POINTMust an audit row name the humanand not only the agent?noyesWorkload identityServiceAccount, TokenReview, SARDoes your IdP do exchangewith an actor token?yesnoToken exchangeRFC 8693, sub + actWorkload identityuser ID as audit onlyThen harden: where most production setups landSPIFFE mTLS between workloads + exchanged user token in the request+ OPA or Cedar deciding on both identities at the MCP server or gateway
Two questions pick the base pattern; layering is what you add on top.

If you only remember a few things:

  • Decide who the upstream should see before you write code. The rest follows from that one question.
  • Audience-bind everything. A token for the MCP server is not a token for upstream, and the spec now says so.
  • A header with a user ID is a log line, not an identity.
  • Do the authorization on each tool call. Agents choose tools at runtime, so the check belongs next to the tool.
  • Start with workload identity, add mTLS when tokens worry you, add exchange when audit demands the human. Complexity should follow a requirement, not a diagram.