sdlc.dev

Python path · Functions and control flow · Topic 5 of 12

Iterators, generators and lazy evaluation

Not started

A for loop in Python does not know about lists or indexes. It asks the object for an iterator and then calls next() on it until the iterator says it is finished. Generators are the easy way to write iterators: a function containing yield that pauses at each value and resumes where it left off. Together they let you describe a whole pipeline of work that only happens when someone asks for the next item.

Why it matters: Generators let you stream a 10 GB file or an endless feed in constant memory, and they are the machinery that asyncio's coroutines grew out of.

  1. 1Two methods make a for loop

    An iterable's `__iter__` returns an iterator; the iterator's `__next__` returns items until it raises StopIteration.

  2. 2yield pauses a function

    Calling a generator function runs nothing. Each next() resumes the frame until the following yield.

  3. 3Lazy pipelines

    Chained generators pull one item at a time through every stage, so memory stays flat however long the input is.

  4. 4One pass only

    An iterator is used up after one pass. Iterate it again and you get nothing, silently.

1

How it works

The explanation, with examples you can run.

The iterator protocol

An iterable is anything with an __iter__ method that returns an iterator. An iterator has a __next__ method that returns the next item or raises StopIteration. Iterators are themselves iterable (their __iter__ returns self), which is why you can loop over one directly.

nums = [10, 20]it = iter(nums)print(next(it), next(it))   # 10 20try:    next(it)except StopIteration:    print("exhausted")

for x in nums: is exactly this, with the StopIteration caught for you. A list is iterable but not an iterator, so every for loop over it gets a fresh iterator and starts from the beginning. A file, a map(), a zip() or a generator is an iterator, so it can only be walked once.

Generators

A function with yield in it becomes a generator function. Calling it creates a generator object without running any of the body. Each next() runs the frame until the next yield, hands out that value, and freezes the frame, local variables and all:

def countdown(n):    print("starting")    while n > 0:        yield n        n -= 1    print("done") gen = countdown(3)          # nothing printed yet: the body hasn't startedprint(next(gen))            # starting, then 3print(list(gen))            # [2, 1], after printing done

When the function returns, the generator raises StopIteration and the loop ends. yield from other delegates to another iterator, which makes it easy to split a generator into smaller ones.

Lazy pipelines

Because each stage pulls from the one before only when asked, you can chain generators over input that would never fit in memory, or that never ends:

import itertools def read_lines():    for i in range(1_000_000_000):        yield f"line {i}" errors = (line for line in read_lines() if line.endswith("7"))print(list(itertools.islice(errors, 3)))   # ['line 7', 'line 17', 'line 27']

Only 28 lines were ever produced; islice stopped pulling after three matches. A list comprehension in the same place would try to build a billion strings first. Generator expressions (parentheses instead of square brackets) are the one-line version, and itertools has the building blocks: islice, chain, groupby, batched (3.12+), pairwise and more.

The price of laziness is that iterators are single-use and have no length:

squares = (x * x for x in range(3))print(sum(squares), sum(squares))   # 5 0

The second sum gets an exhausted iterator and silently returns 0. If you need several passes or len(), materialise the data with list().

Generators as coroutines

Generators can also receive values: value = yield item pauses, and gen.send(x) resumes with x as the result of the yield. That two-way pausing is the idea Python's async/await coroutines are built on. They are a separate type now, but an event loop still drives them by resuming each one until it next has to wait.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.What does this print?
    squares = (x * x for x in range(3))print(sum(squares), sum(squares))
  2. 2.`gen = countdown(3)` where countdown contains `print("starting")` before its first yield. When is "starting" printed?
  3. 3.Why does `max(line for line in open(path))` work on a file bigger than RAM?

0 of 3 right so far. Get all three to finish this topic.

Go deeper