hall-requests-async
0.000
Challenge · difficulty 3/5
Use the `requests.get_async()` function from the `requests` library to fetch a list of URLs
concurrently and return their JSON bodies. Write the code.
Proposed solution
The `requests` library in Python does not have an `async` method like `requests.get_async()`. The `requests` library is synchronous by design. However, you can achieve concurrent HTTP requests using the `requests` library with threading or multiprocessing, or you can use an asynchronous HTTP client library like `aiohttp` for true asynchronous requests.
Here’s how you can fetch a list of URLs concurrently using `requests` with threading:
```python
import requests
from concurrent.futures import ThreadPoolExecutor
def fetch_url(url):
try:
response = requests.get(url)
response.raise_for_status() # Raise an error for bad status codes
return response.json()
except requests.RequestException as e:
print(f"Error fetching {url}: {e}")
return None
def fetch_urls_concurrently(urls, max_workers=5):
results = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = [executor.submit(fetch_url, url) for url in urls]
for future in futures:
results.append(future.result())
return results
# Example usage:
urls = [
"https://jsonplaceholder.typicode.com/posts/1",
"https://jsonplaceholder.typicode.com/posts/2",
"https://jsonplaceholder.typicode.com/posts/3"
]
json_bodies = fetch_urls_concurrently(urls)
print(json_bodies)
```
### Using `aiohttp` for Asynchronous Requests:
If you want true asynchronous requests, you can use the `aiohttp` library. Here’s an example:
```python
import aiohttp
import asyncio
async def fetch_url(session, url):
try:
async with session.get(url) as response:
response.raise_for_status()
return await response.json()
except Exception as e:
print(f"Error fetching {url}: {e}")
return None
async def fetch_urls_concurrently(urls):
async with aiohttp.ClientSession() as session:
tasks = [fetch_url(session, url) for url in urls]
return await asyncio.gather(*tasks)
# Example usage:
urls = [
"https://jsonplaceholder.typicode.com/posts/1",
"https://jsonplaceholder.typicode.com/posts/2",
"https://jsonplaceholder.typicode.com/posts/3"
]
json_bodies = asyncio.run(fetch_urls_concurrently(urls))
print(json_bodies)
```
### Key Points:
1. The `requests` library is synchronous, so you need threading or multiprocessing for concurrency.
2. For true async, use `aiohttp` or `httpx` (which supports both sync and async).
3. The `aiohttp` example is more efficient for I/O-bound tasks like HTTP requests.