The itertools module provides a collection of fast, memory-efficient tools for working with iterators. These functions are implemented in C, making them extremely fast. They work lazily—they produce values one at a time without storing everything in memory.
This lesson covers the most useful itertools functions: infinite iterators like count(), cycle(), repeat(); iterators that terminate like chain(), compress(), dropwhile(), takewhile(); and combinatoric iterators like product(), permutations(), combinations(), and combinations_with_replacement().
🕯️ Magic Note
The itertools module is written in C and is extremely efficient. Functions like chain() and islice() create iterators that yield values without creating intermediate lists. This makes them perfect for processing large or infinite data streams.
Python
import itertools
# count(start, step) – infinite counter
counter = itertools.count(10, 2)
for i in range(5):
print(next(counter), end=” “)
# 10 12 14 16 18
# cycle(iterable) – repeat the iterable forever
colors = itertools.cycle([“red”, “green”, “blue”])
for i in range(7):
print(next(colors), end=” “)
# red green blue red green blue red
# repeat(object, times) – repeat an object multiple times
repeated = itertools.repeat(“Python”, 3)
print(list(repeated)) # [‘Python’, ‘Python’, ‘Python’]
# repeat without times (infinite)
infinite_repeat = itertools.repeat(“echo”)
# Use with islice to get finite slice
echoes = list(itertools.islice(infinite_repeat, 4))
print(echoes) # [‘echo’, ‘echo’, ‘echo’, ‘echo’]
Python
import itertools
# chain(*iterables) – combine iterables sequentially
result = list(itertools.chain([1, 2, 3], “ABC”, range(3)))
print(result) # [1, 2, 3, ‘A’, ‘B’, ‘C’, 0, 1, 2]
# compress(data, selectors) – filter items where selector is truthy
data = [“apple”, “banana”, “cherry”, “date”]
selectors = [True, False, True, False]
result = list(itertools.compress(data, selectors))
print(result) # [‘apple’, ‘cherry’]
# dropwhile(predicate, iterable) – drop while predicate is true
numbers = [1, 2, 3, 4, 5, 1, 2, 3]
result = list(itertools.dropwhile(lambda x: x < 3, numbers))
print(result) # [3, 4, 5, 1, 2, 3] (drops 1,2)
# takewhile(predicate, iterable) – take while predicate is true
result = list(itertools.takewhile(lambda x: x < 3, numbers))
print(result) # [1, 2] (takes 1,2 then stops)
# filterfalse(predicate, iterable) – opposite of filter
evens = list(itertools.filterfalse(lambda x: x % 2, range(10)))
print(evens) # [0, 2, 4, 6, 8]
🕯️ Magic Note
dropwhile() and takewhile() are perfect for processing data streams where you want to skip a prefix or take a prefix based on a condition. They stop processing as soon as the condition changes, making them efficient.
Python
import itertools
# groupby(key) – group consecutive equal items
data = [“apple”, “apple”, “banana”, “apple”, “apple”, “cherry”]
for key, group in itertools.groupby(data):
print(f”{key}: {list(group)}”)
# apple: [‘apple’, ‘apple’]
# banana: [‘banana’]
# apple: [‘apple’, ‘apple’]
# cherry: [‘cherry’]
# groupby with key function
words = [“cat”, “dog”, “car”, “bird”, “cow”, “duck”]
for key, group in itertools.groupby(sorted(words), key=lambda x: x[0]):
print(f”Words starting with {key}: {list(group)}”)
# Words starting with b: [‘bird’]
# Words starting with c: [‘car’, ‘cat’, ‘cow’]
# Words starting with d: [‘dog’, ‘duck’]
# islice(iterable, start, stop, step) – slice iterator lazily
infinite = itertools.count()
sliced = itertools.islice(infinite, 5, 15, 2)
print(list(sliced)) # [5, 7, 9, 11, 13]
# tee(iterable, n) – create n independent iterators
original = range(5)
iter1, iter2 = itertools.tee(original, 2)
print(list(iter1)) # [0, 1, 2, 3, 4]
print(list(iter2)) # [0, 1, 2, 3, 4]
Python
import itertools
# product(*iterables, repeat=1) – Cartesian product
suits = [“Hearts”, “Diamonds”]
ranks = [“A”, “K”]
cards = list(itertools.product(suits, ranks))
print(cards) # [(‘Hearts’, ‘A’), (‘Hearts’, ‘K’), (‘Diamonds’, ‘A’), (‘Diamonds’, ‘K’)]
# product with repeat (cartesian product of iterable with itself)
dice = list(itertools.product([1, 2, 3, 4, 5, 6], repeat=2))
print(f”Two dice outcomes: {len(dice)} possibilities”) # 36
# permutations(iterable, r) – ordered arrangements (order matters)
items = [“A”, “B”, “C”]
perms = list(itertools.permutations(items, 2))
print(perms) # [(‘A’, ‘B’), (‘A’, ‘C’), (‘B’, ‘A’), (‘B’, ‘C’), (‘C’, ‘A’), (‘C’, ‘B’)]
# combinations(iterable, r) – unordered combinations (order doesn’t matter)
combs = list(itertools.combinations(items, 2))
print(combs) # [(‘A’, ‘B’), (‘A’, ‘C’), (‘B’, ‘C’)]
# combinations_with_replacement – allows repeating elements
combs_wr = list(itertools.combinations_with_replacement(items, 2))
print(combs_wr) # [(‘A’, ‘A’), (‘A’, ‘B’), (‘A’, ‘C’), (‘B’, ‘B’), (‘B’, ‘C’), (‘C’, ‘C’)]
🕯️ Magic Note
Combinatoric iterators are essential for password generation, test case generation, lottery simulations, and many combinatorial problems. They are memory-efficient because they yield tuples one at a time instead of generating all possibilities at once.
Python
import itertools
items = [1, 2, 3]
r = 2
print(“Permutations (order matters):”)
for p in itertools.permutations(items, r):
print(p)
# (1,2), (1,3), (2,1), (2,3), (3,1), (3,2)
print(f”Count: {len(list(itertools.permutations(items, r)))}”) # P(3,2)=6
print(“\nCombinations (order doesn’t matter):”)
for c in itertools.combinations(items, r):
print(c)
# (1,2), (1,3), (2,3)
print(f”Count: {len(list(itertools.combinations(items, r)))}”) # C(3,2)=3
Python
import itertools
import string
def generate_passwords(charset, min_length, max_length):
“””Generate all possible passwords in a length range.”””
for length in range(min_length, max_length + 1):
for password_tuple in itertools.product(charset, repeat=length):
yield “”.join(password_tuple)
# Example: short numeric PINs
chars = string.digits
for pin in generate_passwords(chars, 2, 4):
print(pin)
# Stop after a few to avoid flooding
if int(pin) > 10:
break
Python
import itertools
def chunked_iterable(iterable, size):
“””Yield chunks of the specified size from an iterable.”””
it = iter(iterable)
while True:
chunk = list(itertools.islice(it, size))
if not chunk:
break
yield chunk
# Process a large file line by line in chunks
def process_large_file(filename, chunk_size=1000):
with open(filename, “r”) as f:
for chunk in chunked_iterable(f, chunk_size):
# Process each chunk (e.g., insert into database)
print(f”Processing chunk of {len(chunk)} lines”)
# batch_insert(chunk)
# Process a large list in chunks
data = list(range(1000000))
for chunk in chunked_iterable(data, 10000):
print(f”Chunk sum: {sum(chunk)}”)
🕯️ Magic Note
The chunked_iterable pattern is essential for processing large datasets without loading everything into memory. It works with any iterable, not just lists.
Python
import itertools
from collections import deque
def rolling_average(data_stream, window_size):
“””Compute rolling average of a data stream.”””
window = deque(maxlen=window_size)
for value in data_stream:
window.append(value)
yield sum(window) / len(window)
# Simulate sensor data
sensor_data = [10, 12, 11, 15, 14, 13, 16, 18, 17, 19]
averages = rolling_average(sensor_data, 3)
print(list(averages)) # [10.0, 11.0, 11.0, 12.67, 13.33, 14.0, 14.33, 15.67, 17.0, 18.0]
Python
import itertools
import time
# Manual chain using list concatenation
def manual_chain(lists):
result = []
for lst in lists:
result.extend(lst)
return result
# itertools chain (lazy)
def itertools_chain(lists):
return itertools.chain(*lists)
lists = [list(range(10000)) for _ in range(10)]
# When you need to iterate once, itertools is memory efficient
for item in itertools_chain(lists):
pass # Processes items one by one, no intermediate list
- Calling list() on infinite iterators without limiting (memory exhaustion)
- Forgetting that groupby() only groups consecutive items
- Consuming the same iterator multiple times (use tee() if needed)
- Assuming islice() modifies the original iterator (it creates a new one)
- Using cycle() without a break condition (infinite loop)
- Write a function that generates the first 10 powers of 2 using itertools.count() and itertools.islice().
- How do you combine two lists into pairs of all combinations?
- What is the difference between combinations() and permutations()?
- Write a function that processes a large file in chunks of 1000 lines.
- How would you create an infinite iterator that repeats the pattern [1, 2, 3] forever?
- What does itertools.tee() do?
⚡ Whisper
Itertools is a treasure chest of lazy, efficient, and elegant iteration tools. chain() connects streams. cycle() repeats forever. count() generates numbers. groupby() finds runs. islice() slices the infinite. product(), permutations(), combinations() explore possibilities. Each function is a tool. Each tool solves a problem without creating huge lists. Use them to process large files, generated sequences, combinatorial searches, and data streams. They are fast, memory-efficient, and Pythonic. Master itertools, and you master iteration. The loop becomes a whisper. The data flows. The memory stays low. The code stays clear. Itertools is not just a module. It is a way of thinking. Think lazily. Think efficiently. Think iteratively.
