Web Crawler System Design Interview: Reason It Out Step by Step

The web crawler system design interview rewards reasoning, not recall. You are not expected to reproduce a known architecture from memory; you are expected to scope an open-ended problem, make decisions, and explain why. This walkthrough covers the moves interviewers actually score: clarifying requirements first, building the pipeline a component at a time, handling the frontier, politeness, deduplication, storage, re-crawling and failure, and narrating trade-offs out loud as you go.

Treat it as a rehearsal script for your thinking, not a set of lines to recite. The same moves transfer to almost every other system design case.

Why the Web Crawler System Design Interview Keeps Coming Up

What the interviewer is watching for is whether you scope before you design. A crawler for a small research archive and a crawler for a search index are different systems, and the interviewer wants to see you notice that before you draw boxes. One problem also touches a lot of ground: breadth-first traversal, queues, deduplication, storage and distributed coordination, so they can watch you move between data structures and infrastructure without ever leaving the same scenario.

Step 1: Clarify Requirements Before You Draw Anything

Spend the first few minutes asking questions. Not because you are stalling, but because the answers change the design.

Questions worth asking

• What is the crawler for? Search indexing, monitoring, archiving and research each pull the design in a different direction. An indexer cares about coverage and freshness; a monitor cares about latency on a known set of pages; an archive cares about completeness and fidelity.
• What scale are we targeting? Pages per month, average page size, and how fresh the results need to be. These numbers decide whether you are designing for one machine or many.
• What counts as done? A one-off crawl of a fixed list, or a continuous crawl that never stops?
• Are there constraints on which sites we may crawl, and do we have to honour site-level rules?

Separate functional from non-functional

Functional requirements describe what the system does: fetch pages, extract links, store content, avoid re-fetching the same URL, respect site crawling rules. Non-functional requirements describe how well it does it: throughput, politeness, fault tolerance, cost, and how quickly a newly discovered page becomes available. Writing both lists down is the first move in any system design requirements discussion, and it is worth rehearsing until it is automatic. If either list feels shaky, the Rate Limiter System Design Interview: Reasoning Out Loud walkthrough shows the same two lists applied to a smaller case, and the Database Sharding System Design Interview: Lead With the Why shows how they steer a bigger one.

Here is an illustrative example of how that sounds out loud, not a real interview:

"Before I draw anything, let me pin down the purpose. I'll assume this crawler feeds a search index rather than an archive, because that changes how much I care about freshness. For scale, I'll assume a large number of pages per month and an average page size of a few hundred kilobytes, and I'll write those assumptions down so we can revisit them. Functionally, it fetches pages, extracts links, stores content and respects site crawling rules. Non-functionally, I care about throughput, politeness, fault tolerance and cost. One assumption I'm treating as fixed: we crawl only sites that permit it."

State your assumptions out loud, and say which ones you are treating as fixed. Interviewers usually grade the reasoning, and an assumption on the whiteboard is a decision you can revisit later. Requirement gathering is a skill you can rehearse, and structured practice helps you do it under time pressure.

Step 2: Sketch the High-Level Web Crawler Architecture

Now draw the pipeline, and keep it simple. A first pass might be:

1. Seed URLs go into a frontier queue.
2. Fetcher workers pull URLs from the frontier and download pages.
3. A parser extracts text and outgoing links.
4. A link extractor normalises those links and hands new ones back to the frontier.
5. A content store keeps the page body; a metadata store keeps URL, fetch time, status and hashes.
6. A deduplication layer decides whether a URL or page has been seen before.

Then explain the frontier as a choice. A plain FIFO queue is simpler and gives fair breadth-first coverage. A priority queue lets you encode policy such as freshness, page importance, or the politeness delay a host is currently owed, but it needs starvation handling so low-priority hosts are not ignored forever. Say which you would pick and why.

Describe the fetcher as a pool of workers with a DNS cache and connection reuse, because DNS lookups and TLS handshakes add overhead on top of the page fetch itself. Then stop. Add components only when a requirement forces you to, and say so: "I am adding a separate metadata store here because we need to query fetch history without scanning page bodies."

Step 3: The Frontier, Politeness and Crawl Rate

The frontier is where most of the interesting design lives, and politeness is the constraint candidates most often forget.

Per-host queues

Instead of one global queue, keep a queue per host and a scheduler that decides which host to serve next. One slow or very large site then cannot starve the rest of the crawl. The scheduler can round-robin across hosts, or weight by importance or freshness.

Robots rules and crawl delay

A crawler should read robots.txt before fetching and honour the allow and disallow rules it sets; treat them as a hard constraint in your design, as the safe default. The site's own terms, and any rules that apply to you, decide what you may actually crawl. Crawl-delay is a widely used convention rather than a universal rule, so treat it as a politeness default rather than a hard requirement. The design consequence is the same either way: the frontier needs per-host state, and your effective throughput is bounded by the slowest hosts, not by your worker count.

Rate limiting per domain

Rate limiting per domain is separate from overall throughput. You might target a large number of pages per second in aggregate while never exceeding a modest request rate to any single host. Say this explicitly, because it shows you understand that the two numbers are not the same.

The trade-off to narrate

Aggressive crawling finishes faster but risks blocks and overloads servers. Polite crawling is slower but sustainable. A reasonable position: default to polite, and only raise the rate for hosts where you have a reason and a permission to do so.

Step 4: Deduplication, URL Normalisation and Content Hashing

The same page appears under many URLs. Trailing slashes, session identifiers, query parameter order, uppercase hostnames and redirects all produce variants. If you do not normalise, you crawl the same content repeatedly.

Normalisation first

Lowercase the host, strip known tracking parameters, resolve relative links against the page URL, and pick a canonical form for trailing slashes. Do this before the URL reaches the deduplication store, not after.

Exact URL deduplication versus near-duplicate content

Exact URL deduplication answers "have I seen this address?" Content-level near-duplicate detection answers "have I seen this page, under a different address?" They solve different problems, and mature crawlers do both: a URL store to avoid refetching, and a content hash or similarity check to avoid storing the same article five times.

Where to store seen URLs

This is a memory-versus-accuracy trade-off worth stating plainly. A Bloom filter is compact and fast, but it can report a URL as seen when it was not, and a standard Bloom filter cannot remove entries, so it cannot track re-crawl bookkeeping on its own (counting and scalable variants exist, at extra cost). So do not treat the Bloom filter alone as a safe record of what you have crawled. Pair it with an exact store that you consult on every Bloom hit: the false positive then costs one extra lookup rather than a missed page. An exact store alone is precise but heavier as the crawl grows. Say which failure mode you prefer and why.

Step 5: Storage, Freshness and Re-crawling

Separate raw page storage from extracted text and metadata. Raw pages are large, immutable and rarely queried; extracted text and metadata are smaller and queried constantly. Putting them in the same place makes both expensive.

Re-crawling is a scheduling problem. Pages change at different rates, so a sensible design re-crawls based on observed change frequency and page importance rather than on a fixed interval for everything. A page that never changes should drift down the schedule; a news homepage should stay near the top.

Storage choices feed straight back into cost and query speed, so name the access pattern before naming the technology. The trade-off to narrate: re-crawl often for freshness, or re-crawl less and save bandwidth and budget. There is no universally right answer; there is a right answer for the stated requirement.

Step 6: Scaling Out and Handling Failure

Scale by partitioning the frontier by host, so workers can run in parallel without stepping on each other. A worker owns a set of hosts and is the only one allowed to fetch from them, which keeps politeness state coherent.

Coordination and worker death

Decide who owns which host, and what happens when a worker dies mid-crawl. The host assignments need to be recoverable, and in-flight URLs need to return to the frontier rather than vanish. A lease with a timeout is a simple way to express this: if the worker does not renew, the host is reassigned.

Retries, backoff and idempotent writes

Transient failures deserve retries with exponential backoff; permanent ones (a 404, a blocked host) should go straight to a dead-letter queue for inspection rather than being retried forever. Writes should be idempotent, so a retried fetch does not create a duplicate record.

The trade-off to narrate: strong coordination is simpler to reason about but adds a bottleneck and a single point of failure. Loose coordination scales further but risks duplicate work. Pick based on how much duplicate work you can tolerate.

Step 7: Narrating Trade-Offs in the Web Crawler System Design Interview

This is the part candidates skip, and it is the part being scored. Use a repeatable sentence shape: option, benefit, cost, and the condition under which you would pick it.

For example: "I would use a Bloom filter in front of the URL store. It saves memory and keeps lookups fast, at the cost of occasional false positives that cost an extra lookup. I would pick it if cost matters more than the extra check; if I wanted the simplest possible design, I would go exact."

Then invite the interviewer in: "I would pick X here unless you care more about Y." Keep a running list of decisions and revisit them when a new requirement appears. And practise saying the trade-off out loud rather than only thinking it, because spoken reasoning is usually what gets scored. A timer and a recording of yourself will show you exactly where you skipped the reasoning. Rehearsing out loud, then reviewing the recording against the moves above, is a structured way to practise rather than just re-reading notes.

Budgeting your time and depth

In a 45-minute web crawler system design interview, go deep on the frontier, politeness and deduplication, because those are the parts specific to a crawler, and cover storage, re-crawling and failure handling at a design level rather than in fine detail. Say your budget out loud at the start so the interviewer can steer you. If you get stuck halfway through, name what you are stuck on and what information would unblock you, then propose the simplest thing that could work and refine it. Interviewers generally prefer a candidate who names the uncertainty and keeps reasoning over one who goes quiet.

Common Mistakes and How to Avoid Them

These are the habits that most often cost marks in a web crawler system design interview.

• Jumping into components before clarifying scope. Fix: ask at least three questions first.
• Ignoring politeness, site crawling rules and duplicate URLs entirely. Fix: make them a named section of your design.
• Designing for one machine when the question implies web scale. Fix: state the scale assumption, then design for it.
• Never stating a trade-off, so the interviewer cannot tell whether you considered alternatives. Fix: one trade-off sentence per major decision.
• Running out of time because you spent ten minutes on the frontier and never reached storage or failure handling. Fix: budget your time out loud at the start.

A practical way to rehearse is to run the same case three times with different constraints: a small archive, a large index, and a monitoring crawler. Each run forces different decisions, and by the third you will notice you are reasoning rather than remembering.

Before Your Next Mock Interview

The takeaway: reason out loud, and attach one trade-off to every major decision, because that is what the interviewer is scoring. Next, run this case once with a timer, then watch or listen back and mark the moments where you chose a component without saying why. Those gaps are your practice list for the next run. For more walkthroughs, see the Blog.

Frequently asked questions

How deep should I go into web crawler architecture in a 45-minute system design interview?

Depth should follow the interviewer's interest. Lead with the crawler-specific parts, then check in: "Would you like me to go deeper on storage or on failure handling?" That keeps the remaining time pointed at what is being assessed.

Do I need to know the exact algorithm for URL deduplication, or is the trade-off enough?

The trade-off is usually enough. If you do name an algorithm, be ready to say what it costs and when you would not use it; a named algorithm with no trade-off attached is weaker than a clear trade-off with no name.

What are the most important non-functional requirements for a web crawler design?

Throughput, politeness, fault tolerance and cost are the ones that shape the design most. Freshness matters if the crawler feeds an index, and completeness matters if it feeds an archive. State which ones you are optimising for before you draw components, because they change the frontier, the scheduling and the storage choices.

How do I practise system design interviews without memorising answers?

Change one constraint between runs, such as scale or freshness, and notice which decisions move. If your answer stays identical when the requirements change, you are reciting rather than reasoning.

Should I mention robots.txt and politeness in a system design interview?

Yes, and phrase it as a constraint rather than a courtesy. If the interviewer pushes for maximum throughput, say that per-host limits cap the rate any single site will tolerate, so you would scale by adding hosts rather than by hammering one. That answers the throughput question without abandoning politeness.

What should I do if I get stuck halfway through the design?

If you are behind with two minutes left, stop expanding and close the loop: summarise the decisions you made, name the one you would revisit first and why, and say what you would design next. A clean handover shows the interviewer your reasoning held up even though time ran out.

Want a structured way to rehearse system design reasoning out loud?