So far, we have used the for loop to iterate over sequences. So, the for loop is an iterator. In this chapter, we will learn to create our own iterators. However, creating a custom iterator is not that common in programming, but a good programmer should know how to create one. There can be several benefits of creating our own iterator like saving memory which we will discuss later in the chapter.
We know that sequences like lists, tuples, strings, etc can be iterated using the for loop. Thus, the for loop is the iterator and the sequence over which it iterates is called the iterable.
For example, we can iterate over a list using a for loop as shown below.
mylist = [1, 2, 3, 4, 5, 6]
for num in mylist:
print(num)
We are well aware of how for loop is used to iterate over a sequence like a list. The list mylist is a sequence of numbers. The for loop iterates over the list and prints each element of the list. Here, the for loop is the iterator and the list mylist is the iterable.
for loop is the most common iterator used in Python.
Technically speaking, an iterator is an object that iterates. It is used to return the data of an iterable by returning one element in each iteration. Whereas, an iterable is an object which is a collection of data.
Till now, we have used only for loop as the iterator. Let’s see how we can create a new iterator.
Creating a Python Iterator
We can create a new iterator using the built-in iter() function in Python. This function takes the iterable which we want to iterate over as the argument and returns the iterator as the output.
Python also provides another built-in function next() which the iterator returned by the iter() function uses to do iteration over the iterable.
Seems confusing? Ok, it must have been. Just understand that the iter() function is used to create an iterator and that iterator uses the next() function to iterate over an iterable like list, tuple, etc.
Let’s create a simple iterator.
mylist = [1, 2, 3, 4] # iterable
# creating an iterator named my_iterator
my_iterator = iter(mylist)
# iterate over the list mylist using next()
# 1st iteration
print(next(my_iterator))
# 2nd iteration
print(next(my_iterator))
# 3rd iteration
print(next(my_iterator))
# 4th iteration
print(next(my_iterator))
The iter() function takes the list mylist as argument and returns an iterator. We named the returned iterator as my_iterator (You can give any other name to the iterator). Hence, we got our iterator.
Now, to iterate over the list, the iterator uses the next() function. When the first time next() is called by writing next(my_iterator), it returns the first element of the list. On calling next() the second time, it returns the second element of the list. This goes on for each call of next().
It was this simple to create an iterator.
If in the above example, we call next() one more time, then it will throw the StopIteration exception because there is no more element in the list to iterate through. This is shown below.
mylist = [1, 2, 3, 4] # iterable
# creating an iterator named my_iterator
my_iterator = iter(mylist)
# iterate over the list mylist using next()
# 1st iteration
print(next(my_iterator))
# 2nd iteration
print(next(my_iterator))
# 3rd iteration
print(next(my_iterator))
# 4th iteration
print(next(my_iterator))
# this will throw error
print(next(my_iterator))
So you now know how to create an iterator using the iter() function. Let’s create an iterator to iterate over a string.
mystring = "Hey there!" # iterable
# creating an iterator named my_iterator
my_iterator = iter(mystring)
# iterate over the list mylist using next()
# 1st iteration
print(next(my_iterator))
# 2nd iteration
print(next(my_iterator))
# 3rd iteration
print(next(my_iterator))
# 4th iteration
print(next(my_iterator))
# 5th iteration
print(next(my_iterator))
We created an iterator named my_iterator for the string mystring, and called the next() function only five times, thus printing the first five characters of the string.
Python __iter__() and __next__()
We can also use the special functions __iter__() and __next__() in place of iter() and next() respectively. However, using iter() and next() is preferred due to shorter syntax.
mylist = [1, 2, 3, 4] # iterable
# creating an iterator named my_iterator
my_iterator = mylist.__iter__()
# iterate over the list mylist using next()
# 1st iteration
print(next(my_iterator))
# 2nd iteration
print(next(my_iterator))
# 3rd iteration
print(my_iterator.__next__())
# 4th iteration
print(my_iterator.__next__())
From this example, you can see that replacing next(my_iterator) by my_iterator.__iter__() and iter(my_iterator) by my_iterator.__next__() doesn’t alter the result.
Python Working of for Loop as Iterator
We know that the for loop can iterate over an iterable like list, tuple, etc., automatically (without calling next() again and again).
Internally, the for loop also uses the iter() and next() functions for creating an iterator and for iteration. Let’s take a look at how the for loop is implemented internally. Note that this information is just for your understanding.
Look at the following syntax which we use to implement the for loop over an iterable.
for element in iterable:
# perform some task
This is implemented internally as shown below.
# creating an iterator from iterable
iterator_obj = iter(iterable)
# iterating over the iterable
while True:
try:
# returning the next element
element = next(iterator_obj)
# perform some task
except StopIteration:
# if StopIteration exception is thrown, break the loop
break
iterator_obj = iter(iterable) → Internally in the for loop, iter() takes the iterable (for example, a list) as an argument and returns the iterator (let’s name it iterator_obj).
Then an infinite while loop is run. Inside the loop, there is a try-except clause. Inside the try clause, next(iterator_obj) returns the next element of the iterable, after which any code which is written inside the body of our normal for loop is written. Inside the except clause, we are breaking the loop if the StopIteration exception is raised.
Don’t worry if you didn’t understand the above syntax clearly. The following examples will make it clear.
First of all, let’s see an example in which a for loop prints all the elements of a list.
mylist = [1, 2, 3, 4, 5, 6]
for num in mylist:
print(num)
Now, let’s look at how the for loop implements this internally.
mylist = [1, 2, 3, 4, 5, 6]
iterator_obj = iter(mylist)
while True:
try:
num = next(iterator_obj)
print(num)
except StopIteration:
break
Hope you understood it now.
Python Creating a Custom Iterator
Suppose we have a list of numbers and we created an iterator for the list using iter(). By default, whenever next() is called, it returns the next element of the list.
Now, suppose we want that, calling next() should return the next to next element instead of the next element. In that case, we can create our own custom iterator.
If you learn to create an iterator once, then you can create any other iterator also because the syntax almost remains the same. So, let’s create an iterator for all the numbers from 10 to 14.
class Range:
# constructor
def __init__(self, max):
self.max = max
# creates iterator
def __iter__(self):
self.x = 10
return self
# moves to next element
def __next__(self):
x = self.x
# if x becomes greater than max, stops iteration
if x > self.max:
raise StopIteration
# else increments the old value and returns it
self.x = x + 1;
return x
# creating an iterator
obj = Range(14)
iterator_obj = obj.__iter__()
# 1st iteration
print(iterator_obj.__next__())
# 2nd iteration
print(iterator_obj.__next__())
# 3rd iteration
print(iterator_obj.__next__())
# 4th iteration
print(iterator_obj.__next__())
# 5th iteration
print(iterator_obj.__next__())
# 6th iteration
print(iterator_obj.__next__())
In this example, we created a class Range with the methods __iter__() and __next__(). We will be implementing the desired logic of iter() and next() inside these functions.
obj = Range(14) → While creating the object obj of the Range class, we passed 14 to the constructor, thus making its parameter max equal to 14. Therefore, the attribute max became equal to 14.
iterator_obj = obj.__iter__() → The object obj called the method __iter__() of the class. Inside the __iter__() method, an object whose attribute x is initialized to 10 is returned. This returned object having the attributes x (= 10) and max (= 14) is assigned to iterator_obj. This is our iterator object.
iterator_obj.__next__() → The object iterator_obj called the method __next__() of the class. Inside the __next__() method, it is checked if the value of the attribute x is less than that of the attribute max. If x is less than max, then the value of x is incremented by 1 and returned.
In the first iteration, x is 10 and max is 14. Therefore, x is incremented to 11 and returned and printed.
In the second iteration, x is 11 and max is 14. Therefore, again x is incremented to 12 and returned.
Going on, when the value of x becomes 15, x becomes greater than 14. In that case, the StopIteration exception is thrown.
So, you can see that the above example behaves similar to how an iterator behaves. In the Range class, the __iter__() method creates and returns an iterator object with the desired attributes. Similarly, the __next__() method returns the next element based on the desired conditions.
It is fine if you are not able to write the code to create your own custom connector because it is not required much in Python. However, it is important that you understand the flow.
We can also use the for loop to create the iterator object and to iterate as shown below.
class Range:
# constructor
def __init__(self, max):
self.max = max
# creates iterator
def __iter__(self):
self.x = 10
return self
# moves to next element
def __next__(self):
x = self.x
# if x becomes greater than max, stops iteration
if x > self.max:
raise StopIteration
# else increments the old value and returns it
self.x = x + 1;
return x
# creating an iterator
obj = Range(14)
for num in obj:
print(num)
Do you remember that in the internal implementation of the for loop, the iterator object calls the __iter__() and __next__() functions. Yes, this is how these methods got called when using the for loop in the above example.
Now let’s create another iterator for all the even numbers from 10 to 20 (included).
class Range:
# constructor
def __init__(self, max):
self.max = max
# creates iterator
def __iter__(self):
self.x = 10
return self
# moves to next element
def __next__(self):
x = self.x
# if x becomes greater than max, stops iteration
if x > self.max:
raise StopIteration
# else increments the old value and returns it
self.x = x + 2;
return x
# creating an iterator
obj = Range(20)
for num in obj:
print(num)
You must have understood the changes we did here. So, as we said, the syntax of the class remains almost the same when creating different custom iterators.
Python Infinite Iterator
We can also create iterators that never get exhausted. This means that we will never get the StopIteration exception while iterating. This can be done by not raising the StopIteration exception when creating a custom iterator.
Let’s quickly create an infinite iterator for all the even positive numbers.
class Even:
# creates iterator
def __iter__(self):
self.x = 2
return self
# moves to next element
def __next__(self):
x = self.x
# else increments the old value and returns it
self.x = x + 2;
return x
# creating an iterator
obj = Even()
iterator_obj = obj.__iter__()
# 1st iteration
print(iterator_obj.__next__())
# 2nd iteration
print(iterator_obj.__next__())
# 3rd iteration
print(iterator_obj.__next__())
In this example, we will always get the next element whenever we call next(). In the class Even, we are not raising the StopIteration exception and we also removed the constructor because we no longer need the max attribute.
However, it is suggested to keep to avoid using infinite loops unless required.
The significance of using iterators can be that it helps us save memory. For example, if we want to display all the numbers from 1 to 100, we need not store them all in the memory at once. Instead, only one element is stored in the memory at once.
An iterator is a powerful tool which when used at the correct time can save us a lot of memory. So use iterators with caution.
