hall-requests-async
0.000
Challenge · difficulty 3/5
Use the `requests.get_async()` function from the `requests` library to fetch a list of URLs
concurrently and return their JSON bodies. Write the code.
Proposed solution
The `requests` library **does not have a `get_async()` function**. It is a synchronous HTTP library by design. However, you can easily achieve concurrent fetching using Python's built-in `concurrent.futures`, or switch to a true async library like `httpx` or `aiohttp`.
Here are the two standard approaches:
### 🔹 Option 1: Concurrent `requests` (Recommended for existing `requests` codebases)
Uses `ThreadPoolExecutor` to run multiple `requests.get()` calls concurrently without blocking the main thread.
```python
import requests
import json
from concurrent.futures import ThreadPoolExecutor, as_completed
def fetch_json(url):
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
return response.json()
except requests.exceptions.RequestException as e:
return {"url": url, "error": f"Request failed: {e}"}
except json.JSONDecodeError as e:
return {"url": url, "error": f"Invalid JSON: {e}"}
def fetch_urls_concurrently(urls, max_workers=10):
results = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Map futures to URLs for tracking
future_to_url = {executor.submit(fetch_json, url): url for url in urls}
# Collect results as they complete
for future in as_completed(future_to_url):
results.append(future.result())
return results
# Example usage:
# urls = ["https://jsonplaceholder.typicode.com/posts/1", ...]
# json_bodies = fetch_urls_concurrently(urls)
```
### 🔹 Option 2: True `async/await` (Modern Python approach)
If you specifically want async syntax, use `httpx` (a drop-in async replacement for `requests`):
```python
import httpx
import asyncio
import json
async def fetch_json_async(client, url):
try:
response = await client.get(url)
response.raise_for_status()
return response.json()
except httpx.HTTPError as e:
return {"url": url, "error": f"Request failed: {e}"}
except json.JSONDecodeError as e:
return {"url": url, "error": f"Invalid JSON: {e}"}
async def fetch_urls_async(urls, max_connections=10):
async with httpx.AsyncClient(
limits=httpx.Limits(max_connections=max_connections)
) as client:
tasks = [fetch_json_async(client, url) for url in urls]
return await asyncio.gather(*tasks)
# Example usage:
# json_bodies = asyncio.run(fetch_urls_async(urls))
```
### 🔍 Key Notes:
- `as_completed()` returns results in the order they finish, not the order of the input list. If you need to preserve input order, replace the loop with:
```python
results = list(executor.map(fetch_json, urls))
```
- Both examples include timeout handling and graceful error catching so one failing URL won't crash the entire batch.
- `httpx` is generally preferred for new async projects, while `ThreadPoolExecutor` + `requests` is ideal when you want to keep dependencies minimal.
Let me know if you need order preservation, retry logic, or rate-limiting added!