API Gateway System Design Interview: A Spoken Walkthrough

The API gateway system design interview is a spoken performance, not a diagram dump. You pass it by narrating a thin edge layer: pin down requirements first, sketch what belongs at the edge and what must stay out, then defend each choice with its alternative and the condition that would change your mind. This walkthrough gives you a 35–40 minute script for exactly that, plus a running trade-off ledger you keep on the whiteboard so follow-ups extend your design instead of restarting it.

What is the interviewer actually testing in an API gateway system design interview?

They are checking whether you can separate edge concerns from service concerns, and whether you can justify each edge decision. In the design you should defend, the gateway terminates connections, authenticates, routes, enforces quotas and emits telemetry. It does not own business rules. Say that boundary out loud early; it usually signals seniority faster than a component list.

Three signals are usually scored:

• Requirement scoping. Do you ask before drawing?
• Component reasoning. Can you say why each piece exists, not just name it?
• Trade-off articulation. When two options are defensible, can you pick one and say what would flip the choice?

Set expectations in your first sentence: "I'll treat the gateway as a thin, stateful-at-the-edges layer, not a second business logic tier, and I'll spend the first few minutes on requirements." That framing buys you time and tells the interviewer you know the shape of the answer.

Here is an example of that opening, about sixty seconds, that you can adapt:

"I'll design a thin edge layer, not a second business logic tier. The gateway terminates TLS, validates identity, routes by host and path, enforces quotas and emits telemetry; business rules stay in the services. Before I draw, I want to pin down four things: who calls it — public clients, partners or internal services — the protocols, the peak request rate and payload size, and the p99 latency budget for the gateway hop. I'll assume single-region and token auth unless you'd rather I plan for multi-region from the start."

The minute-by-minute script for a 35–40 minute design

Keep this timeline in your head and say the time markers out loud now and then; it shows you are managing the conversation, not just the diagram.

1. 0–5 minutes: requirements. Ask the checklist below, state your defaults, and repeat the non-functional targets back.
2. 5–12 minutes: sketch. Draw the layers top to bottom and the not-here list, then explain the control plane and data plane split.
3. 12–20 minutes: routing and auth. Cover path, host and header matching, static versus dynamic discovery, and where token validation stops and service authorization begins.
4. 20–28 minutes: rate limiting and resilience. Name the algorithm, say where the counter lives, then walk the failure path: timeouts, retries, circuit breaking, degradation.
5. 28–35 minutes: ledger and follow-ups. Fill in the ledger rows and answer pushback by extending a row.
6. 35–40 minutes: wrap-up. Name the weakest part of your design and what you would do next.

The first five minutes: which system design requirements do you pin down before drawing anything?

Ask a short, ordered checklist and write the answers where you can both see them:

1. Callers. Public clients, partner integrations, internal services, or all three?
2. Protocols. HTTP/REST, gRPC, WebSocket, or a mix?
3. Load. Rough peak requests per second and typical payload size.
4. Latency budget. What p99 can the gateway hop add?
5. Regions. Single-region or multi-region from day one?
6. Auth model. Tokens, API keys, mTLS, or a combination?
7. Change cadence. Must route and quota changes go live without a deploy?

Here is the difference between a weak exchange and a stronger one.

Weak: "How many users will there be?"

Stronger: "What's the peak request rate per client, and is the gateway on the hot path for every internal call or only external traffic? I'm asking because internal east-west traffic changes whether I optimise for auth offload or for raw throughput."

The stronger version works because it links the question to a design fork. Interviewers score the reasoning, not the question count.

State the non-functional requirements back before sketching: an availability target, a p99 latency budget for the gateway hop, and whether configuration changes must be live. If the interviewer waves a detail away, pick a reasonable default, say it, and move on. For a deeper treatment of this step, see Non-Functional Requirements in a System Design Interview.

The architecture sketch: API gateway components in and out

Draw top to bottom and narrate each layer in one sentence:

• TLS termination and connection handling — where encryption ends and keep-alive is managed.
• Authentication and authorization — validate identity cheaply at the edge.
• Routing and path matching — map host, path and headers to a backend.
• Request and response transformation — header injection, protocol translation, light reshaping.
• Rate limiting and quotas — protect backends per client and per route.
• Observability — access logs, metrics, traces, correlation IDs.
• Resilience — timeouts, bounded retries, circuit breaking, bulkheads.

Then draw the not-here list, because it shows judgement: business logic, per-service data access, long-running orchestration, and heavy payload transformation. "If it needs to know what an order means, it doesn't live here."

Explain the control plane versus data plane split. A control plane holds route tables, quota policies and key sets, and pushes them to stateless data-plane nodes behind a load balancer. The data plane serves requests and, in this design, holds no durable state. That split is why you can scale request handling horizontally and why a config change is a push, not a redeploy. This is the core of any api gateway architecture you would defend, so say it once, clearly, and move on.

Routing and service discovery: how does a request find the right backend?

Start with the matching rules and a concrete example: /v1/orders/* goes to the orders service, but /v1/orders/*/shipments goes to shipping, because the more specific path wins. Host-based routing separates partner traffic, and header-based routing supports canaries.

Then compare static route tables with dynamic service discovery:

Say the trade-off sentence: "Dynamic discovery is more flexible, but it adds a dependency the gateway now fails with; static tables are simpler but slower to change, so I'd start static and revisit if the fleet churns." The phrase "start static and revisit" is what makes it a decision rather than a shrug.

Authentication and authorization at the edge: where do you draw the line?

Validate tokens at the gateway: check the JWT signature and expiry, reject bad traffic before it reaches a service, and cache public keys so validation does not call the identity provider on every request. Mention key rotation, because a cached key set needs a refresh path.

Then draw the line clearly. The gateway can reject unauthenticated traffic cheaply, but it should not encode business permissions that change often. Fine-grained rules like "only the account owner can cancel this order" belong in the service that owns the data.

"The gateway answers 'is this a valid caller?' The service answers 'is this caller allowed to do this to this resource?' Splitting it that way keeps permission changes out of the edge config."

That sentence is worth rehearsing, because it is a follow-up this prompt often invites.

Rate limiting and quotas: which algorithm, and what does it cost?

Name three algorithms and one sentence each on burst behaviour and memory:

• Token bucket — allows bursts up to the bucket size, cheap to store per client.
• Fixed window — simplest to reason about, but bursts can straddle the window boundary.
• Sliding window log — most accurate, but memory grows with request volume.

Then say where the counter lives. Per-node local counters are fast but inconsistent across nodes; a shared store is consistent but adds a network hop and a dependency on the hot path. One workable answer is local counters with periodic reconciliation, and you should say why: it keeps the common case fast and accepts a small overshoot.

Give a concrete example: 100 requests per second per API key with a burst of 200. When exceeded, the client gets a 429 with a Retry-After header, so well-behaved clients back off instead of retrying immediately.

Resilience and failure: what happens when a backend is slow or down?

Walk through the failure path in order. Timeouts bound how long the gateway waits. Retries must be bounded, jittered and idempotency-aware — never retry a non-idempotent write blindly. Circuit breaking stops sending traffic to a backend that is clearly failing. Bulkheads isolate one slow route from the rest.

Then describe graceful degradation: fail fast with a clear error, or serve a cached or default response where the product tolerates it. Say which you would choose and why.

Name the trade-off: "Aggressive timeouts protect the gateway but can cut off legitimately slow endpoints, so I'd set per-route budgets rather than one global timeout." Per-route budgets show you have thought about how different endpoints fail differently. For the flow-control side of this, see Backpressure System Design Interview: How to Reason About Flow Control.

The trade-off ledger: how do you keep your decisions defensible under follow-ups?

Keep a running ledger on the whiteboard with four columns: decision, alternative considered, why chosen, what would change it. Every time you commit to something, add a row. When a follow-up arrives, extend the row instead of restarting the design. This is where most system design trade offs get won or lost.

Three worked entries to use as templates:

When the interviewer pushes, answer by pointing at the ledger: "That's the second row — if the latency budget tightens, I'd switch to local counters and accept overshoot." That is the move that turns a follow-up into a demonstration. Our guide to Respond to Follow-Up Questions in an Interview: Extend, Don't Repeat covers the same extend-don't-repeat habit.

Here is how that works when the change is bigger than a pushback. Suppose the interviewer says, halfway through, "now make it multi-region." Don't redraw. Add a row: decision — region-local data planes with a global control plane; alternative — a single global data plane; why chosen — requests stay close to users and one region's outage doesn't take the edge down; what would change it — a requirement that every region see config changes at the same instant. Then name the new trade-off out loud: config propagation delay against regional independence. Finally, say what you would change first: quota state and key caches, because those are the pieces that go stale across regions. That one move shows you can absorb a pivot without losing the design you already defended.

The three most common mistakes, and the fix for each

• Skipping requirements. Fix: spend the first five minutes on the checklist above and say the defaults you chose.
• Listing components without reasons. Fix: after each box, say one sentence on why it exists and what breaks without it.
• Going silent during trade-off questions. Fix: answer with the ledger pattern — decision, alternative, reason, what would change it.

If you tend to over-explain, the plan in Stop Rambling Interview: A Live Answering Plan applies directly to design narration.

How do you practise this out loud before the interview?

Run a 30-minute solo drill, separate from the interview script above. Five minutes on requirements, speaking the checklist and the defaults. Fifteen minutes narrating the design from blank page to ledger, out loud, at interview pace. Ten minutes reviewing your recording against the three mistakes above.

Record yourself and listen for two things: whether you said the trade-off sentence for each decision, and whether you filled silence with reasoning rather than filler. The API gateway system design interview rewards the candidate who keeps talking in decisions, so practise the sentences, not just the boxes. For the requirements drill, Consistency Models System Design Interview: Pick and Defend shows how to state a target and defend it under pressure.

Before you walk in

The gateway prompt is won in the first five minutes and defended in the ledger. One takeaway: every component you draw should come with a reason and a condition that would change it. One thing to do next: run the 30-minute drill once, record it, and fix the single weakest moment before you run it again. Treat the API gateway system design interview as a rehearsal you can repeat, not a test you get one shot at.

Frequently asked questions

Should the API gateway be a single point of failure, and how do I answer that concern?

Answer from availability, not from the diagram. Say the entry point is redundant: several nodes serve traffic at once, so losing one costs capacity, not service. Then name the blast radius of a control-plane outage — running nodes keep serving the last good configuration, but new routes and quota changes wait until it recovers.

Do I need to design the gateway's own database or config store in detail?

Only at the level of the contract. Describe what the control plane stores — route tables, quota policies, key sets — and how it pushes changes to data-plane nodes. Deep storage design is usually out of scope unless the interviewer asks; say so and offer to go deeper.

How is an API gateway different from a load balancer or a service mesh in this interview?

A load balancer distributes connections. A gateway adds auth, routing by path or header, quotas and transformation for external callers. A service mesh handles service-to-service traffic with sidecars. Name the boundary, then say which one you are designing and why.

What if the interviewer asks me to add multi-region support halfway through?

Say the pivot in one sentence before you touch the diagram: "I'll keep the design and add a region dimension to the ledger." Then add the row and name the new trade-off. The phrasing matters because it tells the interviewer you are extending a decision, not abandoning the design you already defended.

What if I don't know the peak request rate the interviewer is assuming?

Say the number you are assuming and why it is plausible, then offer to revise it. For example: "I'll assume a few thousand requests per second at peak; if it's an order of magnitude higher, the rate-limit store and connection handling change first." Naming the assumption and its consequence keeps the design moving without pretending to know a figure you were never given.

Practise the narration out loud and review how it landed.