Build a concurrent web crawler
Every page under a root, each fetched once, built in three stages on real threads: a breadth-first crawl, then worker threads that claim URLs under a lock, then an end decided by work finished and a fetch that raises survived, with the executor and asyncio versions beside it.
How do the worker threads of a crawler know the crawl is finished?
Count the work in flight, not the queue. A URL counts from the moment it is queued until the worker that fetched it has queued every new link on its page, so the count is zero only when nothing is waiting and nobody is fetching. An empty queue is not enough, because a worker that is still fetching may be about to add links. With queue.Queue the count is built in: call task_done in a finally after a page's links are queued, and join returns when every queued item has been marked done. Then the caller puts one stop marker per worker on the queue and joins the threads.
Why does the seen set need a lock if the GIL is there?
Because checking whether a URL is in the set and adding it are two steps, and the GIL only promises that one thread runs bytecode at a time, switching between instructions. Two threads can both find a URL missing and both add it, so the page is fetched twice. Hold one lock across the check and the add. The lock also has to cover the queueing and the count of work in flight, so they can never disagree with the set.
Should a crawler use threads or asyncio?
The work is almost all waiting on the network, and both handle that well. CPython releases the GIL while a thread waits on I/O, so threads overlap their fetches. Threads fit when the fetching library is blocking, which is most of them. asyncio fits when the client library is async and you want thousands of connections open at once, because a task costs far less than a thread. Under asyncio a check and an add with no await between them cannot be interleaved, so the seen set needs no lock at all.
How do you test a threaded crawler without flaky tests?
Pass the fetch function in, and let the test hand over a fake that serves a small graph from a dict and counts fetches per URL. The fake is also where a test holds threads still: a threading.Barrier inside the fetch of two pages makes both workers claim a shared link at the same moment, and an Event or a join inside one fetch holds that worker while another finds the queue empty. Assert only what no order of the threads can change, such as every page fetched exactly once, and give every wait a timeout.
What should a crawler do when a fetch fails?
Retry a bounded number of times, then record the URL as failed and carry on. The failure must never end the worker, or the pool quietly shrinks, and it must never be swallowed, or a page that failed looks like a page with no links. In a real crawler, retry only failures that can pass (a timeout, a reset connection, a server error) with a growing delay between tries, and record a missing page at once.