Before and After: Moving 12,000 API Calls From Processes to Threads in Python

    November 21, 2026
    9 min read
    By Silas Orvenwick
    Share this article

    Twelve thousand requests, each taking about 250 milliseconds to come back, add up to 50 minutes when a script sends them one at a time. That was the starting point of the job described below: a nightly enrichment script that sent product records to an external classification API and wrote the labels back to a database.

    Someone had read that Python's GIL stops threads from running in parallel, so the first rewrite used a pool of 8 processes. It finished in a little over 6 minutes and used 1.4 GB of RAM to do it.

    The second rewrite used 32 threads in one process. It finished in roughly 4 minutes and stayed under 200 MB. Nothing about the API changed between the two versions. What changed was matching the tool to what the program actually spends its time doing, which for API calls is waiting.

    Where an API call actually spends its 250 milliseconds

    Break one request down and the share of time your Python code holds the CPU is small. Building the JSON payload might take 0.2 ms. Serialising it, opening or reusing a TLS connection and writing bytes to the socket takes a little more.

    Then the process sits idle while the packet crosses the network, the provider queues it, a model or database on their side does its work and the response travels back. Parsing a 5 KB JSON reply takes another fraction of a millisecond. In the illustrative job above, well over 95% of each call was spent blocked on the socket.

    This is the fact that decides the threading question. CPython's global interpreter lock lets only one thread execute Python bytecode at a time, but a thread that enters a blocking system call, such as a socket read inside the standard library or inside urllib3, releases the lock first.

    While thread 1 waits for its response, threads 2 through 32 can build their payloads, send them and start waiting too. The GIL serialises the slivers of CPU work; it does not serialise the waiting. For I/O bound work, threads overlap almost perfectly.

    Multiprocessing solves a different problem. Each process has its own interpreter and its own GIL, so CPU heavy Python code really does run on several cores at once.

    The price is real: every worker is a full interpreter with its own copy of imported libraries, arguments and results travel between processes through pickling, and startup is slow, especially with the spawn method that macOS and Windows use by default. Python 3.14 also moved the default on Linux from fork to forkserver, so code that silently relied on fork copying global state may behave differently after an upgrade.

    So the honest framing of "multiprocessing vs threading for API calls" is a question about ratios. If your code spends 249 ms waiting and 1 ms computing per call, processes buy you nothing threads cannot, and they cost memory and complexity. If each response triggers 400 ms of pure Python parsing, scoring or image work, the picture flips, and that case gets its own section below.

    The before and after, measured on one illustrative job

    The mini-case is illustrative, assembled from a typical enrichment workload rather than a single published benchmark, but the arithmetic is the part worth copying.

    Before. A multiprocessing.Pool(8) mapped a function over 12,000 records. Each worker created its own requests.Session, so the job held 8 keep-alive connections. Throughput was bounded by 8 calls in flight: 12,000 calls divided by 8 equals 1,500 rounds, at 250 ms each that is 375 seconds, a bit over 6 minutes.

    Memory: 8 interpreters at roughly 170 MB each after importing pandas and the database driver, about 1.4 GB in total. Adding more processes to raise concurrency would have multiplied memory linearly, and at 32 processes the job would have needed more than 5 GB to do work that is mostly sleeping.

    After. A concurrent.futures.ThreadPoolExecutor(max_workers=32) in one process, with one requests.Session per thread and the adapter's pool_maxsize raised to match. With 32 calls in flight, the theoretical time drops to 12,000 / 32 × 0.25 s, about 94 seconds. The measured time was closer to 4 minutes, and the reason is instructive: the provider allowed 50 requests per second.

    At 50 rps the floor for 12,000 calls is 240 seconds no matter what concurrency model you pick. The threads hit the rate limit, received a stream of HTTP 429 responses and spent time in backoff until a token bucket limiter was added that paced requests at 45 per second.

    Three lessons came out of the comparison. The concurrency model mattered less than the rate limit, which was the true ceiling. Threads reached that ceiling with a seventh of the memory. And the default pool sizes in libraries quietly cap you: urllib3, which requests uses, keeps 10 connections per host by default, and above that it logs "Connection pool is full, discarding connection" and opens throwaway connections with a fresh TLS handshake each time.

    A short checklist distilled from the rewrite:

    • Measure one call: total latency vs CPU time (time.perf_counter vs time.process_time) before choosing a model.
    • Find the provider's rate limit and concurrency limit in their docs and design to 90% of it.
    • Size the HTTP connection pool to at least your worker count.
    • Reuse sessions; a fresh TLS handshake per call can add 50 to 150 ms.
    • Add timeouts on every request, for example 5 seconds connect and 30 seconds read.
    • Retry 429 and 5xx with exponential backoff and jitter, and honour Retry-After headers.
    • Keep results streaming to the database in batches of 500 or so, not one giant list at the end.

    When processes still win, and a decision tree for picking

    There are API workloads where multiprocessing is the right answer, and they share one trait: meaningful CPU time per call in pure Python. A scraper that runs BeautifulSoup over 2 MB HTML pages, a pipeline that resizes every image it downloads, or a job that runs a scikit-learn model over each response can spend more time computing than waiting.

    In those jobs 32 threads would queue behind one GIL and use a single core. Libraries such as NumPy, lxml or Pillow release the GIL inside many C routines, which blurs the line, so profile rather than guess.

    The common production answer is a hybrid: threads or asyncio do the network I/O, and a small ProcessPoolExecutor sized to the number of cores does the heavy parsing. Data flows from the I/O side to the CPU side through a queue, and each side scales on its own terms.

    Asyncio deserves a word here, because for very high concurrency it beats both. A single event loop with aiohttp or httpx can hold 500 or 1,000 requests in flight with far less overhead than 1,000 OS threads, each of which reserves its own stack.

    The cost is that the whole call path must be async; one blocking database call inside a coroutine stalls everything. For a script that needs 20 to 50 concurrent calls, threads are simpler and fast enough. Beyond a few hundred, asyncio usually pays for its learning curve.

    Run your job through these questions in order:

    1. Is CPU time per call more than about 20% of total latency? If yes, go to 2. If no, go to 3.
    2. Is that CPU work inside C extensions that release the GIL (NumPy, lxml, Pillow)? If yes, threads may still be enough: benchmark them first. If no, use processes for the CPU stage, ideally with threads or asyncio feeding them.
    3. Do you need more than about 200 requests in flight at once? If yes, use asyncio with a client like httpx or aiohttp. If no, go to 4.
    4. Does your code call blocking libraries, such as a synchronous database driver or an SDK without async support? If yes, use a ThreadPoolExecutor. If no, threads and asyncio both work; pick whichever your team already maintains.
    5. Whatever you picked: is the provider's rate limit lower than your concurrency × (1 / latency)? If yes, add a limiter, because the limit is your real ceiling.

    One trap sits outside the tree. ThreadPoolExecutor's default worker count is min(32, os.cpu_count() + 4). On a 4 core laptop that is 8 threads, which is fine for CPU work and too timid for network calls. For API calls, set max_workers explicitly from your target concurrency, not from core count.

    Free threading, async SDKs and where this question is heading

    The GIL argument that pushed the original team toward processes is weakening on two fronts. PEP 703 introduced an optional free-threaded build of CPython in version 3.13, shipped as a separate python3.13t binary, and PEP 779 moved it to officially supported status in 3.14. In that build threads can run Python bytecode on several cores at once.

    It is still opt-in, some C extensions are not yet compatible, and single-threaded code runs somewhat slower on it, so most production API jobs in 2026 still run on the default build. Over the next few releases the case for multiprocessing as a workaround for the GIL will keep shrinking; the case for it as process isolation (a crash in one worker does not kill the rest, memory leaks reset when a worker restarts) will remain.

    The second front is the client libraries. The official SDKs from OpenAI, Anthropic, Stripe and many cloud providers ship async clients alongside sync ones, and httpx gives the same API in both styles. That lowers the cost of moving to asyncio once a job outgrows a thread pool. Many of these SDKs also build retries and backoff in, which removes the most fragile part of hand-rolled concurrency, though you still need your own limiter to stay under account-level quotas.

    What does not change is the measurement that should start every one of these decisions. Time one call, split latency into waiting and computing, and read the provider's limits. In the before-and-after above, those three numbers would have pointed to threads on day one and saved a gigabyte of RAM. Expect free threading to make the wrong choice cheaper; it will not make the measurement optional.

    For teams building product features on top of model APIs, the more likely shift is architectural: background queues such as Celery, RQ or a cloud task service taking over the fan-out, with each worker using a modest thread pool. Concurrency then lives in two layers, worker count and threads per worker, and the provider's rate limit gets enforced in one shared place instead of in every script.

    Related Articles