What are and how to use Generators in Python

In Python, generators are a tool for efficiently handling sequences of data.

Unlike lists, generators do not store all elements in memory, but produce them on the fly.

This makes them very useful for working with large datasets or continuous data streams, as elements are generated only when needed, which can save time.

Additionally, they are much more memory-efficient than lists because they do not store all elements in memory at once.

What are generators

A generator is a function that uses the yield keyword instead of return. yield could be interpreted as “give way”.

The yield keyword allows a function to produce a value while “remembering” its state. This allows execution to be resumed at that point on the next call.

Let’s see it with an example,

def simple_generator():
    yield 1
    yield 2
    yield 3

When simple_generator() is called, the code inside the function does not execute immediately. Instead, it returns a generator object that can be iterated to produce the values.

The created generator object is not invocable by itself. Instead, the next value is “requested” with next(). For example like this,

gen = simple_generator()
print(next(gen))  # Output: 1
print(next(gen))  # Output: 2
print(next(gen))  # Output: 3

In this example

• We use the generator function simple_generator() to create a generator object gen
• next(gen) is used to get the next value produced by the generator
• When there are no more values to produce, the generator raises a StopIteration exception

Generators with Loops

Most generators are created using loops to produce a sequence of values.

def count_to(n):
    counter = 1
    while counter <= n:
        yield counter
        counter += 1

This generator function

• Starts by returning counter with a value of 1
• Each time the function is called, it “wakes up” in the while loop. It increments the value, does the loop, and returns the new counter
• When counter reaches 5, the loop ends, and the sequence finishes.

It is also very common to use generators from a loop. For example, it is very usual to use them with for.

for number in count_to(5):
    print(number)

# output: 1 2 3 4 5

Combined Generators

Generators can be nested to handle more complex sequences.

def nested_generator():
    yield from range(3)
    yield from range(10, 13)

for number in nested_generator():
    print(number)

# output: 0 1 2 10 11 12

In this example, yield from delegates the production of values to another generator or iterable.

• First, the function yields the values generated by range(3)
• When finished, it supplies the values from range(10, 13)
• When both finish, the sequence ends

Generators with Expressions

Python also allows creating generators using a syntax similar to list comprehensions, known as “generator comprehension”.

gen = (x**2 for x in range(10))

for number in gen:
    print(number)

# output: 0 1 4 9 16 25 36 49 64 81