🕯️ Magic Note
Generators are a form of lazy evaluation. They compute values only when needed. This is the opposite of eager evaluation (like lists, which compute everything immediately). Lazy evaluation can dramatically reduce memory usage and improve performance for large datasets.
Python
# A simple generator function
def count_up_to(n):
i = 1
while i <= n:
yield i
i += 1
# Create a generator object
counter = count_up_to(5)
# Use the generator in a loop
for num in counter:
print(num, end=” “)
# Output: 1 2 3 4 5
🕯️ Magic Note
When the generator reaches yield, it produces a value and pauses. The next time next() is called, it resumes right after the yield. This ability to pause and resume is what makes generators unique.
Python
import sys
# List: stores all numbers in memory
list_of_numbers = [x for x in range(1000000)]
print(f”List size: {sys.getsizeof(list_of_numbers)} bytes”)
# About 8 million bytes (8 MB)
# Generator: produces numbers one at a time
def number_generator(n):
for i in range(n):
yield i
gen = number_generator(1000000)
print(f”Generator size: {sys.getsizeof(gen)} bytes”)
# About 112 bytes (tiny!)
Python
def simple_generator():
print(“First yield”)
yield 1
print(“Second yield”)
yield 2
print(“Third yield”)
yield 3
print(“Generator finished”)
gen = simple_generator()
print(next(gen)) # First yield // 1
print(next(gen)) # Second yield // 2
print(next(gen)) # Third yield // 3
# print(next(gen)) # StopIteration (generator exhausted)
Python
# List comprehension (eager, stores all)
squares_list = [x ** 2 for x in range(10)]
print(type(squares_list)) # <class ‘list’>
print(squares_list) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
# Generator expression (lazy, produces on demand)
squares_gen = (x ** 2 for x in range(10))
print(type(squares_gen)) # <class ‘generator’>
print(list(squares_gen)) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
# Generator with condition
even_gen = (x for x in range(20) if x % 2 == 0)
for num in even_gen:
print(num, end=” “)
# 0 2 4 6 8 10 12 14 16 18
🕯️ Magic Note
The syntax is identical to list comprehensions, just with parentheses. Generator expressions are incredibly useful for processing large datasets without memory overhead. They are also faster for single-use iteration.
Python
def infinite_counter(start=0):
while True:
yield start
start += 1
# This generator never ends
counter = infinite_counter()
print(next(counter)) # 0
print(next(counter)) # 1
print(next(counter)) # 2
# Can loop with a break condition
for num in infinite_counter():
if num > 10:
break
print(num, end=” “)
# 0 1 2 3 4 5 6 7 8 9 10
🕯️ Magic Note
Infinite generators are perfect for sequences that are theoretically infinite: Fibonacci numbers, prime numbers, random numbers, or any stream of data that never ends.
Python
def fibonacci():
“””Infinite generator of Fibonacci numbers.”””
a, b = 0, 1
while True:
yield a
a, b = b, a + b
# Print first 10 Fibonacci numbers
fib = fibonacci()
for _ in range(10):
print(next(fib), end=” “)
# 0 1 1 2 3 5 8 13 21 34
# Find first Fibonacci number greater than 1000
fib = fibonacci()
for num in fib:
if num > 1000:
print(f”\nFirst Fibonacci > 1000: {num}”) # 1597
break
Python
def read_lines(filename):
“””Generator that yields lines from a file.”””
with open(filename, “r”) as f:
for line in f:
yield line.strip()
def filter_lines(lines, keyword):
“””Generator that yields only lines containing a keyword.”””
for line in lines:
if keyword in line:
yield line
def transform_lines(lines):
“””Generator that transforms each line to uppercase.”””
for line in lines:
yield line.upper()
# Create a processing pipeline
lines = read_lines(“data.txt”)
filtered = filter_lines(lines, “important”)
transformed = transform_lines(filtered)
# Process the pipeline (lazy evaluation)
for result in transformed:
print(result)
# Each line is read, filtered, transformed, and printed one at a time
🕯️ Magic Note
Generator pipelines are memory-efficient because only one item is processed at a time. The entire dataset is never loaded into memory. This is incredibly powerful for processing large log files, CSV files, or streaming data.
Python
# Without yield from (manual iteration)
def flatten_manual(nested):
for sublist in nested:
for item in sublist:
yield item
# With yield from (cleaner)
def flatten_yield_from(nested):
for sublist in nested:
yield from sublist
nested = [[1, 2], [3, 4], [5, 6]]
flat = list(flatten_yield_from(nested))
print(flat) # [1, 2, 3, 4, 5, 6]
# Recursive flattening with yield from
def deep_flatten(nested):
for item in nested:
if isinstance(item, list):
yield from deep_flatten(item)
else:
yield item
deep = [[1, [2, 3]], 4, [5, [6, 7]]]
flat_deep = list(deep_flatten(deep))
print(flat_deep) # [1, 2, 3, 4, 5, 6, 7]
Python
def interactive_gen():
value = 0
while True:
received = yield value
if received is not None:
value = received
else:
value += 1
gen = interactive_gen()
print(next(gen)) # 0 (yield value = 0)
print(next(gen)) # 1 (auto-incremented)
print(gen.send(100)) # 100 (value set to 100)
print(gen.send(200)) # 200 (value set to 200, then yielded)
print(gen.throw(ValueError(“Something went wrong”))) # Raises exception
🕯️ Magic Note
The send() method sends a value back into the generator, becoming the value of the yield expression. This allows two-way communication, making generators like lightweight coroutines.
Python
import csv
def read_csv_generator(filename):
“””Generator that yields rows from a CSV file one at a time.”””
with open(filename, “r”) as f:
reader = csv.DictReader(f)
for row in reader:
yield row
def filter_by_column(rows, column, value):
“””Generator that yields only rows matching a column condition.”””
for row in rows:
if row.get(column) == value:
yield row
def transform_row(rows):
“””Generator that transforms each row.”””
for row in rows:
row[“full_name”] = f”{row[‘first_name’]} {row[‘last_name’]}”
yield row
# Pipeline (no memory overhead)
data = read_csv_generator(“users.csv”)
filtered = filter_by_column(data, “country”, “Iran”)
transformed = transform_row(filtered)
for user in transformed:
print(user[“full_name”], user[“email”])
| Use a Generator When... | Use a List When... |
|---|---|
| Processing large datasets (>memory limit) | Dataset fits comfortably in memory |
| You only need to iterate once | You need to access items multiple times |
| Creating infinite sequences | You need random access (indexing) |
| Building data processing pipelines | You need to store and modify the data |
| You want lazy evaluation for performance | You need all data immediately available |
| Working with streaming data (logs, sensors) | The data is naturally fixed and small |
- Trying to reuse a generator (they are exhausted after one iteration)
- Converting a generator to a list unnecessarily (defeats memory benefits)
- Using return inside a generator without understanding it ends the generator
- Expecting generators to support indexing or slicing (they do not)
- Calling len() on a generator (not supported)
- Not handling StopIteration when using next() directly
- Write a generator that yields even numbers up to a given limit.
- What is the main advantage of a generator over a list?
- How do you create a generator expression?
- What is the difference between yield and return?
- Write an infinite generator that yields powers of 2 (1, 2, 4, 8, 16, …).
- What does the yield from statement do?
⚡ Whisper
A generator is a patient worker. It waits. You ask for a value. It computes one, hands it to you, and waits again. It does not fill your memory with unneeded results. It does not rush. It gives you what you need, when you need it. This is not slowness. This is efficiency. When processing a billion log lines, a generator reads one line at a time. When generating an infinite sequence, a generator never stops. When building a pipeline, each generator passes data to the next, item by item, never holding the whole stream. This is lazy evaluation. This is memory wisdom. Learn to think in terms of streams, not collections. When you see a large list, ask: “Could this be a generator?” Often, the answer is yes. Your memory will thank you. Your code will be faster. And you will have learned one of Python’s most elegant features.
