Your Own Services Are External

Your payment service tells your ledger service that a $4,200 transfer just settled. The ledger books it, revenue goes up, and the daily reconciliation report looks clean.

Two days later, someone in operations pulls the bank's settlement file. The transfer is still pending. The bank never confirmed it. Your payment service reported a status it read from its own cache, not from the bank, and the ledger trusted the claim because it came from inside the building.

The obvious fix is better monitoring, tighter alerts, maybe a circuit breaker on the payment service. But the payment service worked correctly by its own logic. It returned the status it had.

The problem is that the ledger accepted an internal claim with the same blind faith it would never extend to an outside caller. This is a story about what happens when your own services trust each other the way a monolith trusts itself, even though they stopped being a monolith years ago.

The perimeter delusion

Most fintech teams draw a hard line at the API gateway. Everything outside gets validated, rate-limited, schema-checked, and logged, but everything inside gets a pass because "we control both sides."

Google's security engineering team identified this pattern. The assumption that internal services deserve implicit trust because they share a codebase or a Kubernetes cluster is, in practice, a perception problem. Their Building Secure and Reliable Systems handbook starts from the position that no service inherently trusts another.

The reasoning is straightforward. Your services communicate over a network, run on different machines, deploy at different times, and cache state independently. They're as external to each other as any third-party API, except they have broader permissions and less monitoring.

A monolith hides this. When your payment logic and your ledger logic live in the same process, a function call is a function call. The data is consistent because it's reading from the same memory.

Split those into services, and the function call becomes a network request. The data consistency guarantee disappears, but the trust assumption doesn't.

Four ways internal trust fails

These scenarios share a pattern: one service passes data that looks correct, the next service accepts it without checking, and the error compounds silently.

Unverified settlement status. The payment service marks a transaction as settled based on an optimistic state update. The ledger books revenue. But the bank's confirmation hasn't arrived yet, and when it does, the status is still pending.

The ledger now has a $4,200 entry backed by nothing. If the settlement eventually fails, the reversal is a manual process that touches the user's balance, the revenue report, and the reconciliation log.

Stale cached balance. The account service caches balances with a 30-second TTL. A customer's balance just dropped to $12 from a debit that cleared. The payment service asks the account service for the current balance, gets $1,200 from before the debit, and approves a $500 payment.

The payment service didn't make an error by its own logic. It asked for a balance, got one, and acted on it.

The account service didn't make an error either. It returned what it had. The gap between what it had and what was true is where the overdraft lives.

Inherited bad assertions. The KYC service calls an identity provider and gets a verification response. It doesn't validate the response schema or check that the fields it needs are actually populated, something I covered in my piece on the three axioms as the "no trust" rule for external data. The KYC service stamps the user as verified and passes that assertion downstream.

Now the onboarding service trusts the KYC service's stamp, the account service trusts the onboarding service, and a lending service trusts the account service. Four services deep, nobody has checked the original claim. If the identity provider returned a malformed response that the KYC service deserialized into defaults, every downstream decision is built on air.

Silent deployment breakage. Service A updates its response schema. Service B, which consumes that response, hasn't deployed the matching change yet. The new field arrives as null or gets deserialized into a default. Service B doesn't fail. It proceeds with a zero where there should be a fee amount, or a default status where there should be a specific state. The data looks structurally valid but isn't true.

Why internal failures are worse

An external API going down is noisy. Timeouts fire, alerts trigger, someone pages the on-call. But your own service returning bad data is silent.

Internal services typically have broader permissions than external integrations. Your payment service can write directly to the ledger. An external webhook goes through validation middleware first. The attack surface of a compromised or buggy internal service is larger, and the monitoring is thinner.

SOC 2 auditors understand this distinction. The Trust Services Criteria evaluate processing integrity regardless of where the data originated, internal or external. If your ledger accepts unvalidated claims from an internal service, an auditor sees the same control gap as if it accepted unvalidated claims from a webhook.

NIST SP 800-207 formalizes it further. Zero trust means no implicit trust granted based on network location or asset ownership. Google's BeyondProd framework translates this into production practice: mutual TLS between services, per-service identity, and policy enforcement at every boundary, not just the edge.

What boundary validation looks like

Verification at every service boundary is the fix, and it has four layers.

Schema contracts. Every service validates inbound payloads against a schema, even from internal callers. Protobuf, JSON Schema, Zod, whatever your stack uses. If a field that should be present is missing, the service rejects the request rather than falling back to defaults or proceeding with partial data.

Domain invariant checks. Structural validation catches missing fields, but domain validation catches wrong values.

When the payment service says a $4,200 transfer settled, the ledger doesn't just check that the amount field is a number. It checks that $4,200 is consistent with the transfer it has on record, that the transfer exists, that its current status allows a settlement transition, and that the settlement amount matches the original authorization.

Fail closed on the unexpected. An internal service returning an unexpected field, an unknown enum value, or a payload that doesn't match the contract should halt processing, not log a warning and continue or substitute a default. I covered in an earlier piece why proceeding with invented data is how balances drift.

Independent verification for high-value operations. For anything above a threshold, the consuming service confirms the claim through a second channel. The ledger pulls the transaction status from the bank's API before booking it.

Two sources, one answer. If they disagree, the operation blocks until a human or a reconciliation process resolves it.

This costs latency. An extra API call to the bank adds 200 to 400 milliseconds. For a $4,200 settlement that will sit on the ledger for years, that trade is obvious.

The restaurant kitchen, again

I used a restaurant kitchen in my three axioms piece to explain why you check the ticket, not the voice. That analogy was about strangers walking in off the street.

This is about the line cook who's worked there for ten years telling the head chef that table 12 ordered the lobster. The chef still checks the ticket.

Not because the line cook is untrustworthy, but because the line cook might be reading yesterday's prep list, the order might have changed after the cook last looked, or the cook might be relaying what the server said, who misheard the customer.

The cook's tenure doesn't change the verification step. The ticket is the source of truth. Everything else is a claim.