0%

43- Context Managers (with statement in depth)

Manage resources automatically. Open files, database connections, and network sockets safely. The with statement ensures cleanup even when errors occur.

You have used the with statement to open files. You know it closes the file automatically, even if an error occurs. But have you wondered how it works? And how you can create your own context managers? A context manager is an object that defines what happens when execution enters and exits a with block. It guarantees that cleanup code runs no matter what. This is perfect for managing resources: files, database connections, network sockets, locks, temporary directories, and more. This lesson covers context managers in depth. You will learn how the with statement works, how to create context managers using classes (with __enter__ and __exit__), how to create them using the contextlib module, and practical examples for real-world applications.

🕯️ Magic Note

The with statement was introduced in Python 2.5 (with the `__future__` import) and became standard in Python 2.6. It is one of Python’s most elegant features for resource management. It replaces the common try/finally pattern with cleaner, more readable code.

How the with Statement Works
The with statement calls two special methods on an object: __enter__() when the block starts, and __exit__() when the block ends (whether normally or due to an exception).

Python

# What happens behind the scenes

with something as variable:

do_something()

# Is equivalent to:

variable = something.__enter__()

try:

do_something()

finally:

something.__exit__(exc_type, exc_val, exc_tb)

🕯️ Magic Note

The __exit__ method receives exception information if an error occurred. If it returns True, the exception is suppressed (does not propagate). If it returns False or None, the exception continues to bubble up.

Creating a Context Manager with a Class
Define __enter__ and __exit__ methods on a class.

Python

class ManagedFile:

def __init__(self, filename, mode):

self.filename = filename

self.mode = mode

self.file = None

def __enter__(self):

print(f”Opening {self.filename}”)

self.file = open(self.filename, self.mode)

return self.file

def __exit__(self, exc_type, exc_val, exc_tb):

print(f”Closing {self.filename}”)

if self.file:

self.file.close()

# Return False to propagate exceptions (default)

return False

# Usage

with ManagedFile(“test.txt”, “w”) as f:

f.write(“Hello, World!”)

print(“Inside with block”)

# Output:

# Opening test.txt

# Inside with block

# Closing test.txt

💡 The __exit__ method receives three parameters: exception type, exception value, and traceback. If no exception occurred, all three are None. You can use this to handle specific exceptions differently.
Handling Exceptions in __exit__
You can suppress exceptions or handle them differently by returning True.

Python

class SuppressError:

def __enter__(self):

print(“Entering block”)

return self

def __exit__(self, exc_type, exc_val, exc_tb):

if exc_type is not None:

print(f”Suppressed error: {exc_val}”)

return True # Exception suppressed

print(“No error occurred”)

return False

with SuppressError():

print(“Doing something”)

raise ValueError(“Something went wrong”)

print(“This never runs”)

# Output:

# Entering block

# Doing something

# Suppressed error: Something went wrong

# Program continues (no crash)

⚠️ Suppressing exceptions can hide bugs. Only suppress exceptions you expect and know how to handle. For most context managers, return False (or nothing) to let exceptions propagate.
Creating Context Managers with contextlib
The contextlib module provides decorators and helpers for creating context managers more easily.

Python

from contextlib import contextmanager

@contextmanager

def managed_file(filename, mode):

print(f”Opening {filename}”)

f = open(filename, mode)

try:

yield f # The value that “as variable” receives

finally:

print(f”Closing {filename}”)

f.close()

# Usage (same as class-based version)

with managed_file(“test.txt”, “w”) as f:

f.write(“Hello from contextlib!”)

🕯️ Magic Note

The @contextmanager decorator turns a generator function into a context manager. The code before yield runs on entry. The code after yield (in finally) runs on exit. This is much simpler than writing a full class for simple context managers.

Practical Example: Database Connection Manager
Manage database connections automatically.

Python

import sqlite3

from contextlib import contextmanager

@contextmanager

def get_db_connection(db_name):

conn = sqlite3.connect(db_name)

print(“Database connected”)

try:

yield conn

finally:

conn.close()

print(“Database disconnected”)

# Usage

with get_db_connection(“example.db”) as conn:

cursor = conn.cursor()

cursor.execute(“CREATE TABLE IF NOT EXISTS users (id INTEGER, name TEXT)”)

cursor.execute(“INSERT INTO users VALUES (1, ‘Ali’)”)

conn.commit()

print(“Database operations completed”)

# Connection is automatically closed here

Practical Example: Timer Context Manager
Measure how long a block of code takes to execute.

Python

import time

from contextlib import contextmanager

@contextmanager

def timer(name=”Operation”):

start = time.perf_counter()

print(f”{name} started…”)

try:

yield

finally:

end = time.perf_counter()

duration = end – start

print(f”{name} took {duration:.4f} seconds”)

# Usage

with timer(“Data processing”):

# Simulate some work

time.sleep(1)

total = sum(range(1000000))

print(f”Sum calculated: {total}”)

# Output:

# Data processing started…

# Sum calculated: 499999500000

# Data processing took 1.0234 seconds

Practical Example: Temporary Directory
Create a temporary directory that cleans up automatically.

Python

import tempfile

import shutil

from pathlib import Path

from contextlib import contextmanager

@contextmanager

def temporary_directory():

temp_dir = tempfile.mkdtemp()

print(f”Created temporary directory: {temp_dir}”)

try:

yield Path(temp_dir)

finally:

shutil.rmtree(temp_dir)

print(“Temporary directory cleaned up”)

# Usage

with temporary_directory() as temp_dir:

# Create a file in the temp directory

file_path = temp_dir / “test.txt”

file_path.write_text(“Temporary content”)

print(f”File created at {file_path}”)

# Temp directory is deleted automatically

Practical Example: Redirecting Stdout
Temporarily capture or suppress print output.

Python

import sys

from io import StringIO

from contextlib import contextmanager

@contextmanager

def capture_output():

old_stdout = sys.stdout

captured = StringIO()

sys.stdout = captured

try:

yield captured

finally:

sys.stdout = old_stdout

# Usage

def noisy_function():

print(“This is printed”)

print(“This is also printed”)

return 42

with capture_output() as output:

result = noisy_function()

captured_text = output.getvalue()

print(f”Function returned: {result}”)

print(f”Captured output: {captured_text!r}”)

Built-in Context Managers You Already Know
Python has many built-in context managers beyond file opening.

Python

# 1. File handling (most common)

with open(“file.txt”, “r”) as f:

content = f.read()

# 2. Lock management (threading)

from threading import Lock

lock = Lock()

with lock:

# Critical section – lock is automatically released

pass

# 3. Temporary directories (Python 3.2+)

import tempfile

with tempfile.TemporaryDirectory() as tmpdir:

print(f”Working in {tmpdir}”)

# Directory is automatically deleted

# 4. Redirecting warnings

import warnings

with warnings.catch_warnings():

warnings.simplefilter(“ignore”)

# Warnings are suppressed here

pass

# 5. Changing directory temporarily (Python 3.11+)

import os

with os.chdir(“/tmp”):

print(f”Now in: {os.getcwd()}”)

print(f”Back to: {os.getcwd()}”)

Nesting Context Managers
You can nest multiple context managers, either explicitly or with commas.

Python

# Nested explicitly

with open(“input.txt”, “r”) as infile:

with open(“output.txt”, “w”) as outfile:

outfile.write(infile.read())

# Multiple context managers in one line (Python 3.1+)

with open(“input.txt”, “r”) as infile, open(“output.txt”, “w”) as outfile:

outfile.write(infile.read())

# Custom context managers can also be nested

from contextlib import ExitStack

# Dynamic number of context managers

with ExitStack() as stack:

files = [stack.enter_context(open(f”file_{i}.txt”, “w”)) for i in range(5)]

for i, f in enumerate(files):

f.write(f”Content for file {i}”)

# All files are automatically closed

🕯️ Magic Note

The ExitStack is a powerful tool for managing a dynamic or variable number of context managers. Each call to enter_context() adds a new context manager to the stack. When the with block exits, all are closed in reverse order.

Practical Example: Connection Pool Manager
A more advanced example combining multiple concepts.

Python

import time

from contextlib import contextmanager

from collections import deque

class ConnectionPool:

def __init__(self, size, create_connection_func):

self._pool = deque()

self._create_conn = create_connection_func

for _ in range(size):

self._pool.append(self._create_conn())

@contextmanager

def get_connection(self):

conn = self._pool.popleft()

print(f”Connection acquired ({len(self._pool)} left)”)

try:

yield conn

finally:

self._pool.append(conn)

print(f”Connection returned ({len(self._pool)} available)”)

# Simulated connection and usage

def create_mock_connection():

return {“id”: int(time.time() * 1000) % 10000}

pool = ConnectionPool(3, create_mock_connection)

with pool.get_connection() as conn1:

with pool.get_connection() as conn2:

with pool.get_connection() as conn3:

print(f”Using connections: {conn1[‘id’]}, {conn2[‘id’]}, {conn3[‘id’]}”)

# All connections are back in the pool

Common Mistakes with Context Managers
  • Forgetting to call __enter__ manually (always use with statement)
  • Not using contextlib.contextmanager for simple cases (unnecessary boilerplate)
  • Suppressing all exceptions in __exit__ by returning True unintentionally
  • Not re-raising exceptions after handling them in __exit__
  • Forgetting to close resources in the `finally` block of generator-based context managers
Check Your Understanding
  • Write a context manager that prints “Entering” before a block and “Exiting” after.
  • What is the difference between a class-based context manager and one using @contextmanager?
  • How do you suppress an exception in a context manager?
  • Write a context manager that temporarily changes the current working directory.
  • When would you use ExitStack?

⚡ Whisper

The with statement is a promise. “Enter this block, do what needs to be done, and when you leave (whether by success, error, or return), clean up.” Files close. Connections release. Locks unlock. Temp directories delete. This is not magic. This is __enter__ and __exit__. This is @contextmanager and yield. Context managers give your code integrity. They guarantee cleanup. They make resource management automatic and error-proof. Learn to write them. Use them for database connections. Use them for timing code. Use them for temporary changes. The with statement is not just for files. It is for any resource that needs setup and teardown. Master it, and your code becomes trustworthy. It always cleans up after itself. Always.

Related posts