Python Generators

A generator in python is a kind of function which does not use return an object and you could use next() function to iterate through it. The object returned in such a way is iterator. If the iteration is finished then it raises StopIteration exception.

In Python you have specific keyword to achieve a generator implementation, it is called yield. A generator object is generated only once, but its code not run all at once. The calls made to next() execute ( part of ) the code. Execution of the code stops once a yield statement has been reached, upon which it returns the value. The next call to the next() will execute the state in which the generator was left if there are more yields. The point is a function could have more than one yields.

Generators are highly efficient. For large computations you could use generators. Instead of writing a function which returns a list of values, you could write a generator which generates values on the fly. Generators allow for a natural way to describe infinite streams. Generators are memory efficient too.

Consider this example of a generator,

def my_generator():
yield ‘this’
yield ‘is’
yield ‘a’
yield ‘generator’
yield ‘example’

now to iterate over it using next()

generator = my_generator()

next(generator)

# will output

this

next(generator)

# will give output

is

# and so on ..

If you want to iterate it till the end,

for word in my_generator():

print word

You could also create generators using expressions,

>>> generator = (x for x in range(10))

Generators are flexible and efficient way to work with.

The basic implementation of Python generator is stack frames, which is a frame on top of current stacks. The stack frame includes space allocated for variables local to a function including the arguments passed to that function. When we call a function, the program counter or current point of execution is pushed onto the stack, and a new stack is created. Execution then transfers to the beginning of the function being called. With regular functions the stack is popped after the function returns a value.

The key advantage of generators is that the state of the function is preserved, unlike with regular functions where each time the stack frame is discarded, due to which you lose all the state. Another point is that with generator functions the overhead of creating and deleting stack frames is avoided, which is a very trivial advantage.

Although there are scenarios where you would want to avoid generators,

  • When you need to access the data multiple times.
  • When you need to randomly access data.
  • You need to join strings, because using list is more faster than generators in this case.