Python Iterators, Generators, and yield Explained: The Complete 2026 Guide

If you’ve ever used a `for` loop in Python, you’ve used an iterator. If you’ve ever written `for x in range(10):`, you’ve consumed a generator. Despite how common they are, the underlying mechanism — and how to build your own — is one of Python’s most elegant and powerful concepts.

In this tutorial, you will learn to:

• Explain the three-layer relationship between iterable, iterator, and generator
• Implement the iterator protocol from scratch
• Build generator functions using `yield` and generator expressions
• Use `yield from` to delegate to sub-generators
• Apply generators to real-world problems: file processing, data pipelines, and API calls
• Use `.send()` for two-way generator communication
• Avoid common generator pitfalls including the PEP 479 `StopIteration` behavior

Prerequisites

This article assumes you are comfortable with Python functions and understand how `for` loops iterate over sequences. If you are new to Python, start with the Python lists guide first.

The Three Layers: Iterable, Iterator, Generator

These three terms are often used interchangeably, but they represent distinct concepts:

Iterable  →  An object you can iterate over (has __iter__)
    ↓
Iterator  →  The object doing the iterating (has __next__, remembers position)
    ↓
Generator  →  A convenient way to create iterators (uses yield, or generator expression)

What Is an Iterable?

An iterable is any object that can produce an iterator. It has an `__iter__()` method that returns a fresh iterator:

# Lists, strings, dicts, files, sets — all iterables
my_list = [1, 2, 3]
my_string = "hello"
my_dict = {"a": 1, "b": 2}

# iter() gets an iterator from an iterable
it = iter(my_list)
print(type(it))  # <class 'list_iterator'>
print(next(it))  # 1
print(next(it))  # 2
print(next(it))  # 3
print(next(it))  # StopIteration raised!

What Is an Iterator?

An iterator is the active agent — it holds state and knows the current position. It has:

• `__iter__()` — returns `self`
• `__next__()` — returns next item, raises `StopIteration` when done

# A manual iterator — the hard way
class CountUp:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        val = self.current
        self.current += 1
        return val

counter = CountUp(3)
for n in counter:
    print(n)
# 0, 1, 2

What Is a Generator?

A generator is a simpler way to build an iterator — no class, no `__next__`, no manual state management. Just a function with `yield`:

# The same CountUp, but as a generator
def count_up(limit):
    current = 0
    while current < limit:
        yield current
        current += 1

for n in count_up(3):
    print(n)
# 0, 1, 2

Fibonacci Generator — A Classic Example

def fibonacci(limit):
    """Yield Fibonacci numbers up to a limit."""
    a, b = 0, 1
    while a < limit:
        yield a
        a, b = b, a + b

fib_gen = fibonacci(100)
for _ in range(10):
    print(next(fib_gen), end=" ")
# 0 1 1 2 3 5 8 13 21 34

Still not exhausted! Can continue:

for _ in range(5):
    print(next(fib_gen), end=" ")
# 55 89 144 233 377

Generator Expressions — Comprehensions as Iterators

List comprehensions create all items immediately (eager). Generator expressions create items on demand (lazy):

# List comprehension — all in memory at once
squares_list = [x ** 2 for x in range(10)]
print(squares_list)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

# Generator expression — one at a time
squares_gen = (x ** 2 for x in range(10))
print(squares_gen)   # <generator object at 0x...>
print(next(squares_gen))  # 0
print(next(squares_gen))  # 1

Generator expressions are memory-efficient for large (or infinite) sequences:

# Sum of first 1 billion squares — no problem with generator!
total = sum(x ** 2 for x in range(1_000_000_000))
# vs list: [x**2 for x in range(1_000_000_000)] — would allocate 8GB!

# sum() accepts a generator directly — no intermediate list needed

yield from — Delegating to Sub-Generators

def integers(start=1):
    """Yield all integers from start."""
    current = start
    while True:
        yield current
        current += 1

def first_n_evens(n):
    """Yield the first n even integers."""
    count = 0
    num = 2
    while count  n:
            break
        yield val
    for val in first_n_evens(n):
        yield val

print(list(combined(5)))
# [1, 2, 3, 4, 5, 2, 4, 6, 8, 10]

# With yield from — cleaner delegation
def combined_yield_from(n):
    yield from integers(1)
    yield from first_n_evens(n)

print(list(combined_yield_from(5)))
# [1, 2, 3, 4, 5, 2, 4, 6, 8, 10]

---

Real-World Generator Patterns

1. Processing Large Files Line-by-Line

def read_large_file(filepath, chunk_size=1024):
    """Memory-efficient: never loads entire file."""
    with open(filepath) as f:
        while chunk := f.read(chunk_size):  # read chunk by chunk
            yield chunk

# Count lines in a 10GB file — uses ~constant memory
def count_lines(filepath):
    count = 0
    for chunk in read_large_file(filepath):
        count += chunk.count('n')
    return count

# Process CSV rows lazily
import csv

def csv_rows(filepath):
    with open(filepath, newline='') as f:
        reader = csv.DictReader(f)
        for row in reader:
            yield row

for row in csv_rows("big_data.csv"):
    process(row)  # One row at a time in memory

2. Streaming Data Pipeline

def fetch_prices(symbols):
    """Simulate streaming price data."""
    import random
    import time
    for symbol in symbols:
        yield {"symbol": symbol, "price": round(random.uniform(10, 500), 2)}
        time.sleep(0.1)

def filter_above(prices, threshold):
    """Filter prices above a threshold — lazy."""
    for record in prices:
        if record["price"] > threshold:
            yield record

def transform_symbol(prices, prefix="STOCK:"):
    """Transform each record — lazy."""
    for record in prices:
        record["symbol"] = prefix + record["symbol"]
        yield record

# Build a lazy pipeline — nothing runs until consumed
pipeline = transform_symbol(
    filter_above(
        fetch_prices(["AAPL", "GOOG", "MSFT", "AMZN"]),
        threshold=200
    ),
    prefix="NASDAQ:"
)

for record in pipeline:
    print(record)
# Only runs when iterated — memory-efficient chaining

3. Batched API Calls

def paginated_fetch(get_page):
    """Yield all results from a paginated API."""
    page = 1
    while True:
        results = get_page(page)
        if not results:
            break
        yield from results
        page += 1

# Simulated API
def get_user_page(page):
    # Returns empty list when no more pages
    if page > 3:
        return []
    return [{"id": i + (page-1)*10, "name": f"User-{i+(page-1)*10}"}
            for i in range(1, 11)]

all_users = list(paginated_fetch(get_user_page))
print(f"Total users fetched: {len(all_users)}")  # 30

4. Stateless Parsing with .send()

Generators can receive values back via `.send()` — useful for bidirectional communication:

def running_average():
    """Calculate running average — receives values from outside."""
    total = 0
    count = 0
    average = None
    while True:
        value = yield average  # yield current average, receive new value
        total += value
        count += 1
        average = total / count

ra = running_average()
next(ra)  # Prime the generator (advance to first yield)

ra.send(10)  # Send 10 → average = 10.0
ra.send(20)  # Send 20 → average = 15.0
ra.send(30)  # Send 30 → average = 20.0

5. contextlib.contextmanager — Generator as Context Manager

The contextlib.contextmanager decorator turns a generator into a context manager, cleanly handling setup and teardown:

from contextlib import contextmanager

@contextmanager
def managed_resource():
    print("Acquiring resource")
    resource = allocate()
    try:
        yield resource
    finally:
        print("Releasing resource")
        resource.release()

with managed_resource() as res:
    print(f"Using: {res}")

Generator State — What’s Inside

gen = (x ** 2 for x in range(5))

print(gen.gi_frame.f_locals)  # Shows current frame's local variables
print(gen.gi_code.co_name)    # Name of the generator function
print(gen.gi_running)         # False (not currently executing)

next(gen)  # Advance
print(gen.gi_frame.f_lasti)  # Instruction pointer offset in bytecode

Checking if an Object Is an Iterator or Generator

from collections.abc import Iterator, Generator, Iterable

def gen():
    yield 1

g = gen()

print(isinstance(g, Iterator))     # True
print(isinstance(g, Generator))    # True
print(isinstance(g, Iterable))    # False — generators are NOT iterables!
                                  # (they don't have __iter__ returning something)

# Iterables have __iter__
print(isinstance([1,2,3], Iterator))  # False — list is iterable, not iterator
print(isinstance(iter([1,2,3]), Iterator))  # True — list's iterator is an iterator

Common Mistakes and Gotchas

• Forgetting to prime coroutine-like generators: A generator that uses .send() or expects values must be advanced to its first yield by calling next(gen) before sending values. Failure to do so causes a TypeError.
• Exhausted generators can’t restart: Once a generator raises StopIteration, it cannot be reused. Create a fresh generator instance instead.
• PEP 479 — StopIteration becomes RuntimeError: Since Python 3.7, if a generator raises StopIteration internally (not via return), it gets converted to a RuntimeError. Use return instead of raise StopIteration.
• Variables captured in closures may surprise you: Generator expressions that capture loop variables in nested scopes may hold references to the outer variable’s final value. Use a default argument to capture the value at definition time.
• Accidental assignment instead of comparison: Writing while x = 10: instead of while x < 10: silently assigns 10 to x instead of comparing — a common typo that goes undetected in simple cases.

---

Summary

Concept	What It Is	Key Feature
Iterable	Object with __iter__()	Can produce iterators
Iterator	Object with __next__()	Remembers position
Generator	Function with yield	Simplest way to create iterators
Generator expression	(x for x in iterable)	Lazy list-like comprehension
yield from	Delegation	Pass through values from sub-generator
.send(value)	Two-way communication	Send values into a paused generator

Generators are the backbone of Python’s lazy evaluation model. They power everything from reading large files, to streaming API responses, to building memory-efficient data pipelines. The rule of thumb: if you’re processing a sequence and don’t need random access, a generator is almost always the right choice.

Summary and Next Steps

You now understand the three-layer model (iterable → iterator → generator), how to build custom iterators, and how generator functions with yield simplify iterator creation. You have seen five practical generator patterns covering files, data pipelines, API calls, two-way communication, and context managers.

If you want to go deeper, here are related posts worth reading:

• Python type() Function Explained — Python’s built-in functions and the type system
• Python Lists for Beginners — working with sequences and iteration
• uv Python Package Manager Guide — modern Python tooling

Further Reading

• Python Docs — Iterator Types
• Python 3.7 What’s New — PEP 479
• PEP 479 — StopIteration handling in generators