Service Discovery System Design Interview: Trade-Offs

Service discovery is how a caller finds a healthy instance of a service it needs, without hardcoding addresses. In a service discovery system design interview, the useful move is not to define it and stop — it is to explain why it exists, how registration and resolution work, and which trade-off you are choosing. This walkthrough gives you the order to say things in, the sentences that carry the reasoning, and the follow-ups to expect.

Start With the Problem, Not the Tool

Open by framing the problem. A single server talking to a single database is easy: the address never changes, so you put it in a config file and forget about it. Nothing in that picture scales, and nothing in it breaks quietly.

Now imagine an order service calling a payment service that runs on ten instances behind a load balancer. Those instances scale up during a sale, scale down overnight, get rescheduled onto different hosts, and occasionally fail. A static list of IP addresses goes stale within minutes. Someone has to answer the question: which address do I call right now?

That question is service discovery. Say it that way — as a problem you have diagnosed — before you name any solution. Interviewers tend to hear the difference between reciting a tool and explaining the pressure that produced it early in the answer.

A short opening you can borrow:

"The problem is that instances are ephemeral. They come and go, so any fixed address list rots. I need a way for a caller to ask, at request time, for a healthy instance of the service it wants."

That is roughly twenty seconds, and it sets up everything else.

The Two Halves: Registration and Resolution

Service discovery has two halves, and it helps to walk through the lifecycle in order rather than describing the halves in isolation.

Registration

When an instance starts, it announces itself and its address. That announcement goes into a registry — a shared record of who is currently available. Registration is the supply side of discovery.

Health checking

The registry cannot simply trust that an instance is alive because it registered once. Health checks probe the instance, and when it stops responding, the registry stops handing out its address. This is the part many candidates skip, and it is usually the part worth slowing down for.

Resolution

At request time, the caller looks up a healthy instance and sends traffic to it. Resolution is the demand side: the lookup that turns a service name into a concrete address that is currently in rotation.

The lifecycle, in one breath

Instance starts, registers, passes health checks, receives traffic, fails or is shut down, and is removed from rotation. If you can narrate that loop without notes, you have the core of the answer.

One thing worth adding out loud: the registry is itself a distributed system. It has its own availability concerns, its own consistency questions, and its own failure modes. Naming that shows you are not treating it as magic.

Client-Side vs Server-Side Discovery

This is the fork most interviews turn on, so have both sides ready.

Client-side discovery. The caller queries the registry directly and picks an instance itself, usually with a local cache so it is not hitting the registry on every request. The caller keeps more control over load balancing — it can prefer the closest instance, or one in the same rack.

Server-side discovery. The caller hits a stable endpoint, and a load balancer or proxy resolves the backend on its behalf. The caller does not know or care how many instances exist.

The trade-offs to say out loud:

Client-side pushes discovery logic into every client, which is fine when you have one or two languages and painful when you have six.
Server-side centralizes that logic but adds a hop and another component to operate and monitor.
Both raise the same question: how stale can a client's view be before it causes errors?

Then tie it back to the interviewer's constraints. A small team with a single language might reasonably prefer client-side and a shared library. A larger organization with many services and mixed stacks often leans toward server-side, because the logic lives in one place instead of being reimplemented badly in each client — that is a common pattern, not a rule — it depends on the operational overhead the team can absorb. If the interviewer has not stated constraints, say what you would need to know: team size, language diversity, latency budget. That is the same habit as pinning down requirements before choosing components — see our database sharding page for the same reason-out-loud structure applied to another part of the interview.

Where DNS Fits and Where It Doesn't

DNS is a form of discovery, and it is a common follow-up when you explain service discovery in an interview. Handle it as a trade-off, not a verdict.

DNS works well when services are stable, connections are long-lived, and churn is low. You get discovery essentially for free, with tooling everyone already understands.

DNS struggles when you need fast reactions. TTLs and caching mean a failed instance can keep receiving traffic for a while after it dies. That is fine for a service that changes monthly and painful for one that scales rapidly, deploys frequently, or needs fast failover. The middle ground is DNS-based discovery with short TTLs, which reacts faster but puts more load on the DNS path — so the question becomes how short a TTL your system can tolerate. SRV records are a separate lever: they let DNS carry port and weight metadata, which is useful for discovery, but they do not by themselves make resolution react faster to failure.

A two-sentence version you can practise:

"DNS gives you discovery with caching in the resolution path, which is great for stability and slow to react to failure. If I need to react to a dead instance in seconds, I want something with health checks and a shorter feedback loop."

Notice that this does not call DNS wrong. It names what you are buying and what you are paying.

Consistency, Availability, and the Registry Itself

The registry is a distributed system, so the usual questions apply — and the interviewer may well ask them.

If the registry is briefly unavailable, can callers still work? In client-side discovery with a local cache, usually yes, because clients keep serving from cached results. That is a deliberate choice: you are trading freshness for the ability to keep serving traffic when the registry is down. The caveat is worth saying: a client with no cached entry, or one starting up during the outage, still cannot resolve anything. Server-side discovery behaves differently — if the proxy or load balancer is the thing that is down, the cache is not in the caller's hands, so the failure mode is different.

If the registry is stale, callers may hit dead instances. So the client needs a safety net around imperfect discovery:

Timeouts, so a call to a dead instance fails fast instead of hanging.
Retries, ideally to a different instance, so one bad address does not become an outage.
Circuit breaking, so a struggling dependency does not consume all your capacity.

Frame the choice plainly: for most consumer-facing systems you prefer availability over strict consistency in the registry. Then name the failure mode you are accepting — a brief window where some requests go to instances that have already died — rather than only the one you are avoiding. Naming the accepted risk is what separates a rehearsed answer from a reasoned one.

A Spoken Walkthrough for the Service Discovery System Design Interview

Here is a structure that fits roughly ninety seconds before you invite questions:

Problem. Instances are ephemeral, so fixed addresses rot.
Registration. Instances announce themselves on start.
Health checks. The registry removes instances that stop responding.
Resolution. Callers look up a healthy instance at request time.
Trade-off. Client-side or server-side, depending on team size, languages and latency budget.
Failure handling. Caching, timeouts, retries and circuit breaking cover the gaps.

Here is what the whole thing sounds like delivered in one go, as an example:

"Instances are ephemeral, so any fixed address list goes stale. I would have each instance register its address with a registry on startup, and have the registry health-check it so dead instances drop out of rotation. At request time the caller resolves a healthy instance and sends traffic there. For resolution I would weigh client-side against server-side: client-side gives the caller more control over load balancing but pushes discovery logic into every client, while server-side centralizes it behind a proxy at the cost of an extra hop. I would cache the registry locally so callers survive a brief registry outage, and wrap calls in timeouts, retries to a different instance and circuit breaking, so a stale entry fails fast instead of taking the service down."

Leave hooks for the interviewer rather than closing every door. Something like: "I would cache the registry locally — happy to go deeper on staleness if that is interesting." That signals depth without monologuing.

Practise the trade-off sentence out loud until it sounds like something you thought of rather than something you memorized. The structure works whether the interviewer wants a quick sketch or a deep dive; you simply stop earlier or keep going.

Common Follow-Ups and How to Handle Them

When the topic comes up in a service discovery system design interview, the follow-ups are where the reasoning shows, so for each of these give a short answer plus the trade-off, then stop.

What happens during a rolling deploy when old and new versions coexist? Both versions register and receive traffic, so the contract between them has to tolerate that. You may need versioned endpoints or backward-compatible changes.

How do you avoid a thundering herd when a service restarts? Staggered restarts, jittered retry backoff, and clients that spread load rather than all retrying at once. The registry-side version is just as interesting: after a restart, every client may re-query the registry at the same moment, so a local cache plus jittered refresh keeps that lookup stampede from landing all at once.

Should the registry be global or per-region? Global is simpler to reason about; per-region keeps failure domains small and latency low. Many large systems lean regional, though it depends on failure-domain and latency requirements.

How do you secure discovery so only authorized services can register? Authentication on registration, plus network controls, so a rogue instance cannot insert itself into the pool.

How much detail should I give? Enough to show you know the lifecycle and one trade-off. If the interviewer wants more, they will ask.

Practising the Explanation Before the Interview

The skill here is reasoning out loud, not reciting. Four habits get you there.

Rehearse aloud, ideally recorded. Listening back is where the work happens: you will hear filler words, a step you keep skipping — for many candidates it is health checks — and the moment you drifted from explaining into listing. Note the timestamp of each stumble and run the walkthrough again from that point rather than from the top.

Practise with a timer and with a follow-up question thrown in. Ninety seconds for the walkthrough, then answer one curveball from the list above. If you can do that without losing the thread, you are ready.

Review your own explanation for the trade-off sentence. If it is missing, add it. An answer with no trade-off sounds like a definition; an answer with one sounds like engineering.

Keep the practice honest. The goal is to reason clearly about a problem you understand, not to memorize a script and deliver it back. If you cannot explain why a choice is better than its alternative, you have found the gap to study — and that is far more useful than polishing the wording.

The same discipline shows up across system design fundamentals: state the requirement, name the option, and say what you give up. For related practice, see our cloud architecture page, which uses the same reason-out-loud structure. You can find more on our blog.

Before You Walk In

The answer that lands in a service discovery system design interview is the one that names the trade-off, not the one that lists the most tools. Pick one sentence — client-side control versus server-side centralization — and make it yours. Then record a single ninety-second run of the walkthrough above, listen back once, and fix only the step you skipped. That one pass is worth more than another hour of reading.

Frequently asked questions

What is service discovery in a system design interview?

Treat it as the answer to a moving-target problem: instances change address constantly, so callers need a live lookup rather than a fixed list. In the interview, spend your words on the mechanism and the choice it forces, not on a tidy definition.

Should I choose client-side or server-side service discovery?

There is no default winner; the constraints decide. Ask how many languages the clients are written in and how much extra infrastructure the team can run. If the answer is one language and a small team, a shared client library is cheap. If it is many stacks and a platform team, a proxy you operate once is easier to keep consistent.

Is DNS enough for service discovery?

It depends on how quickly you need to react to a dead instance. Ask what your failover target is: if a request can wait out a TTL, DNS keeps the system simple; if you need to route around a failure in seconds, you need health checks in the path. Say which of those your design assumes.

What happens if the service registry goes down?

Ask first whether the registry is on the request path at all. With a local cache it usually is not, so existing callers keep going and only new ones struggle. Then say what you would monitor: cache age and failed lookups tell you how close you are to the edge.

How much detail should I give about service discovery in an interview?

Match the depth to the role and the time you have. A backend role may expect you to discuss health-check design and cache staleness; a broader role may only need the lifecycle and one trade-off. Offer the short version, then let the interviewer pull the thread they care about.

Do I need to know a specific service discovery tool to answer this question?

No. Explaining the lifecycle and the consistency-versus-availability choice in plain language is more useful than listing products.

Practise explaining service discovery out loud before your next interview.