Advanced Async Techniques: aiohttp, asyncpg in Python

Table of Contents

• Introduction
• Why Advanced Async Techniques Matter
• Understanding aiohttp: Asynchronous HTTP Client and Server
  • Installing aiohttp
  • Making Asynchronous HTTP Requests with aiohttp
  • Building an Asynchronous Web Server with aiohttp
• Understanding asyncpg: High-Performance PostgreSQL Driver
  • Installing asyncpg
  • Connecting to a PostgreSQL Database Asynchronously
  • CRUD Operations Using asyncpg
• Combining aiohttp and asyncpg in a Single Project
• Error Handling and Best Practices
• Conclusion

Introduction

Asynchronous programming in Python, especially using the asyncio framework, has unlocked powerful ways to build scalable applications that can handle thousands of simultaneous connections. While basic asyncio tasks cover many needs, real-world applications often require more advanced techniques, especially for web services and database operations.

Two critical libraries that elevate Python’s async capabilities are aiohttp and asyncpg:

• aiohttp is an asynchronous HTTP client/server library.
• asyncpg is a fast and fully featured PostgreSQL driver.

This article provides a complete guide to mastering these tools to build high-performance, fully asynchronous Python applications.

Why Advanced Async Techniques Matter

When developing web applications or services, performance bottlenecks often arise from:

• Making numerous network calls (e.g., to APIs)
• Performing slow database queries

Blocking operations can severely degrade the responsiveness of your applications. Advanced asynchronous libraries like aiohttp and asyncpg help:

• Maximize I/O efficiency
• Handle thousands of concurrent requests
• Maintain responsiveness without spawning multiple threads or processes

Understanding and implementing these libraries properly can significantly enhance the performance and scalability of your applications.

Understanding aiohttp: Asynchronous HTTP Client and Server

Installing aiohttp

Before using aiohttp, install it via pip:

pip install aiohttp

Making Asynchronous HTTP Requests with aiohttp

As a client, aiohttp allows you to make non-blocking HTTP requests:

import aiohttp
import asyncio

async def fetch(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()

async def main():
    url = "https://www.example.com"
    html = await fetch(url)
    print(html)

asyncio.run(main())

Key Concepts:

• ClientSession manages and persists connections across requests.
• async with ensures proper closing of connections.
• await handles the asynchronous execution.

Use Cases:

• Web scraping
• API integrations
• Downloading multiple resources concurrently

Building an Asynchronous Web Server with aiohttp

aiohttp also provides a powerful, lightweight web server:

from aiohttp import web

async def handle(request):
    return web.Response(text="Hello, World!")

app = web.Application()
app.add_routes([web.get('/', handle)])

if __name__ == '__main__':
    web.run_app(app)

Highlights:

• aiohttp servers are event-driven and efficient for real-time applications.
• Supports WebSocket natively.
• Extensible with middlewares, sessions, and routing mechanisms.

Understanding asyncpg: High-Performance PostgreSQL Driver

Installing asyncpg

Install it using pip:

pip install asyncpg

Connecting to a PostgreSQL Database Asynchronously

import asyncpg
import asyncio

async def connect_to_db():
    conn = await asyncpg.connect(user='youruser', password='yourpassword',
                                 database='yourdb', host='127.0.0.1')
    await conn.close()

asyncio.run(connect_to_db())

Important Notes:

• Connections are coroutine-based.
• Fast connection times compared to traditional drivers like psycopg2.

CRUD Operations Using asyncpg

Insert Data Example:

async def insert_user(name, age):
    conn = await asyncpg.connect(user='youruser', password='yourpassword',
                                 database='yourdb', host='127.0.0.1')
    await conn.execute('''
        INSERT INTO users(name, age) VALUES($1, $2)
    ''', name, age)
    await conn.close()

Select Data Example:

async def fetch_users():
    conn = await asyncpg.connect(user='youruser', password='yourpassword',
                                 database='yourdb', host='127.0.0.1')
    rows = await conn.fetch('SELECT * FROM users')
    for row in rows:
        print(dict(row))
    await conn.close()

Update Data Example:

async def update_user(user_id, new_age):
    conn = await asyncpg.connect(user='youruser', password='yourpassword',
                                 database='yourdb', host='127.0.0.1')
    await conn.execute('''
        UPDATE users SET age=$1 WHERE id=$2
    ''', new_age, user_id)
    await conn.close()

Delete Data Example:

async def delete_user(user_id):
    conn = await asyncpg.connect(user='youruser', password='yourpassword',
                                 database='yourdb', host='127.0.0.1')
    await conn.execute('''
        DELETE FROM users WHERE id=$1
    ''', user_id)
    await conn.close()

asyncpg supports prepared statements, connection pooling, transactions, and sophisticated data types.

Combining aiohttp and asyncpg in a Single Project

One of the most powerful real-world patterns is to combine aiohttp (for web) and asyncpg (for database operations) into a single asynchronous stack.

Example: Simple API to fetch users from the database:

from aiohttp import web
import asyncpg
import asyncio

async def init_db():
    return await asyncpg.create_pool(user='youruser', password='yourpassword',
                                     database='yourdb', host='127.0.0.1')

async def handle_get_users(request):
    async with request.app['db'].acquire() as connection:
        users = await connection.fetch('SELECT * FROM users')
        return web.json_response([dict(user) for user in users])

async def create_app():
    app = web.Application()
    app['db'] = await init_db()
    app.add_routes([web.get('/users', handle_get_users)])
    return app

if __name__ == '__main__':
    web.run_app(create_app())

Here:

• create_pool is used for efficient connection management.
• API route /users fetches and returns users asynchronously.

Error Handling and Best Practices

Timeouts:

• Set timeouts when making HTTP requests with aiohttp to prevent indefinite hangs.

timeout = aiohttp.ClientTimeout(total=10)
async with aiohttp.ClientSession(timeout=timeout) as session:
    ...

Connection Management:

• Always close connections properly.
• Prefer using async with for automatic cleanup.

Pooling:

• Use asyncpg.create_pool() for database pooling instead of raw connections.
• It improves performance and resource utilization.

Exception Handling:

• Gracefully handle exceptions for both network and database operations.

try:
    ...
except aiohttp.ClientError as e:
    print(f"Network error: {e}")
except asyncpg.PostgresError as e:
    print(f"Database error: {e}")

Concurrency Limits:

• When dealing with thousands of requests, use semaphores to limit concurrency and avoid overloading the system.

Conclusion

Mastering advanced asynchronous libraries like aiohttp and asyncpg equips Python developers to build scalable, high-performance applications that can handle thousands of simultaneous users or requests. aiohttp enables efficient asynchronous HTTP operations, while asyncpg delivers fast, asynchronous PostgreSQL database access.

Combining them unlocks powerful full-stack async applications, particularly suited for microservices, real-time APIs, web scraping, financial applications, and large-scale data-driven platforms.