.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/basics/generators.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_basics_generators.py: ================= 1.19 generators ================= This lesson explains the concept of generators in python and the use of keyword ``yield``. .. GENERATED FROM PYTHON SOURCE LINES 10-11 We know that we can do list comprehension as below, .. GENERATED FROM PYTHON SOURCE LINES 11-14 .. code-block:: default print([i for i in range(10)]) .. rst-class:: sphx-glr-script-out .. code-block:: none [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. GENERATED FROM PYTHON SOURCE LINES 15-16 Similarly we can also do tuple comprehension .. GENERATED FROM PYTHON SOURCE LINES 16-19 .. code-block:: default print((i for i in range(10))) .. rst-class:: sphx-glr-script-out .. code-block:: none at 0x7f13697a8e40> .. GENERATED FROM PYTHON SOURCE LINES 20-21 However, we see that a tuple comprehension returns a generator. .. GENERATED FROM PYTHON SOURCE LINES 23-24 .. code-block:: default a = (i**i for i in range(10)) .. GENERATED FROM PYTHON SOURCE LINES 25-26 We can unpack generators using ``*`` as following .. GENERATED FROM PYTHON SOURCE LINES 26-28 .. code-block:: default print(*a) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 1 4 27 256 3125 46656 823543 16777216 387420489 .. GENERATED FROM PYTHON SOURCE LINES 29-32 iterating over generator ========================== We can use generators in a for loop because we can iterate over them. .. GENERATED FROM PYTHON SOURCE LINES 32-39 .. code-block:: default a = (i for i in range(10)) for i in a: print(i) .. rst-class:: sphx-glr-script-out .. code-block:: none 0 1 2 3 4 5 6 7 8 9 .. GENERATED FROM PYTHON SOURCE LINES 40-42 The ``next`` function on a generator is akin to running the for loop for one iteration. It means we want to get the next value from generator. .. GENERATED FROM PYTHON SOURCE LINES 42-46 .. code-block:: default a = (i**i for i in range(10)) next(a) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 .. GENERATED FROM PYTHON SOURCE LINES 47-50 .. code-block:: default next(a) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 .. GENERATED FROM PYTHON SOURCE LINES 51-54 .. code-block:: default next(a) .. rst-class:: sphx-glr-script-out .. code-block:: none 4 .. GENERATED FROM PYTHON SOURCE LINES 55-56 We can call the ``next`` function as long as the function is not exhausted. .. GENERATED FROM PYTHON SOURCE LINES 58-60 Running a for loop on a generator actually calls ``next`` function on the generator until it is exhausted. .. GENERATED FROM PYTHON SOURCE LINES 60-64 .. code-block:: default for i in a: print(i) .. rst-class:: sphx-glr-script-out .. code-block:: none 27 256 3125 46656 823543 16777216 387420489 .. GENERATED FROM PYTHON SOURCE LINES 65-67 Since all the iterations are complete, our generator is now exhausted. If we try to get next value from generator, it will throw an error. .. GENERATED FROM PYTHON SOURCE LINES 67-71 .. code-block:: default # uncomment the following line # next(a) .. GENERATED FROM PYTHON SOURCE LINES 72-75 .. code-block:: default for i in a: print(i) .. GENERATED FROM PYTHON SOURCE LINES 76-80 The above for loop was run on exhausted generated, therefore no iteration was run. However, it should be noted that StopIteration exception was not raised by for loop. This is because, for loop automatically detects the StopIteration and stop its iteration instead of raising the error/exception. .. GENERATED FROM PYTHON SOURCE LINES 82-85 Above, when we created generator using tuple comprehension, we were just returning ``i`` but we can do a complicated or computationally heavy stuff e.g. reading a file at each iteration. .. GENERATED FROM PYTHON SOURCE LINES 85-97 .. code-block:: default def read_file(idx): print(f"reading file {idx}") return idx reader = (read_file(i) for i in range(10)) for _ in reader: pass .. rst-class:: sphx-glr-script-out .. code-block:: none reading file 0 reading file 1 reading file 2 reading file 3 reading file 4 reading file 5 reading file 6 reading file 7 reading file 8 reading file 9 .. GENERATED FROM PYTHON SOURCE LINES 98-99 or a memory intensive computation at each step. .. GENERATED FROM PYTHON SOURCE LINES 99-105 .. code-block:: default a = (i**i for i in range(10)) for i in a: print(i) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 1 4 27 256 3125 46656 823543 16777216 387420489 .. GENERATED FROM PYTHON SOURCE LINES 106-115 Working of generator ====================== A generator actually breaks the computation flow in a for loop and executes the commands inside the for loop one by one. This helps in executing the memory intensive work one by one instead of executing all the code at once. For example, instead of reading all the 10 files at once and then processing them, we read one file at one time, process it and then read the next file. .. GENERATED FROM PYTHON SOURCE LINES 117-120 yield -------- How to generate a generator using ``yield`` keyword? .. GENERATED FROM PYTHON SOURCE LINES 120-132 .. code-block:: default def read_files(num_files): file_content = [] for f in range(num_files): _file_content = read_file(f) file_content.append(_file_content) return read_files(10) .. rst-class:: sphx-glr-script-out .. code-block:: none reading file 0 reading file 1 reading file 2 reading file 3 reading file 4 reading file 5 reading file 6 reading file 7 reading file 8 reading file 9 .. GENERATED FROM PYTHON SOURCE LINES 133-138 Above we are reading all the files and saving their content in a list at once. If the files are large and we don't need all the files at once, then we would like to read one file, use/process its contents and then read the next file. In such a case, we would like to write a function, which reads one file at a time. This can be accomplished by using ``yield`` keyword. .. GENERATED FROM PYTHON SOURCE LINES 140-151 .. code-block:: default def read_files(num_files): for f in range(num_files): yield read_file(f) reader = read_files(10) print(type(reader)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 152-153 Let's iterate through the generator .. GENERATED FROM PYTHON SOURCE LINES 153-157 .. code-block:: default for _ in reader: pass .. rst-class:: sphx-glr-script-out .. code-block:: none reading file 0 reading file 1 reading file 2 reading file 3 reading file 4 reading file 5 reading file 6 reading file 7 reading file 8 reading file 9 .. GENERATED FROM PYTHON SOURCE LINES 158-160 We can not have any statement in a function after ``return`` keyword. However, we can have statements after ``yield``. These statements are executed at the next iteration. .. GENERATED FROM PYTHON SOURCE LINES 160-174 .. code-block:: default def read_files(num_files): print("entering read_files function") for f in range(num_files): print(f"at iteration {f}") yield read_file(f) print(f"at iteration {f} after yield") reader = read_files(10) next(reader) .. rst-class:: sphx-glr-script-out .. code-block:: none entering read_files function at iteration 0 reading file 0 0 .. GENERATED FROM PYTHON SOURCE LINES 175-178 We see that the string **at iteration {f} after yield** is not printed yet. This is because only first iteration of for loop is complete. However, this string will be printed at the start of next iteration. .. GENERATED FROM PYTHON SOURCE LINES 178-182 .. code-block:: default next(reader) .. rst-class:: sphx-glr-script-out .. code-block:: none at iteration 0 after yield at iteration 1 reading file 1 1 .. GENERATED FROM PYTHON SOURCE LINES 183-186 When a function has ``yield`` keyword, we can have statements even outside the for loop. All the statements outside the for loop will be executed after the last iteration of the generator. .. GENERATED FROM PYTHON SOURCE LINES 186-201 .. code-block:: default def read_files(num_files): print("entering read_files function") for f in range(num_files): print(f"at iteration {f}") yield read_file(f) print(f"at iteration {f} after yield") print('end of for loop') return reader = read_files(10) print(type(reader)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 202-206 .. code-block:: default for _ in reader: pass .. rst-class:: sphx-glr-script-out .. code-block:: none entering read_files function at iteration 0 reading file 0 at iteration 0 after yield at iteration 1 reading file 1 at iteration 1 after yield at iteration 2 reading file 2 at iteration 2 after yield at iteration 3 reading file 3 at iteration 3 after yield at iteration 4 reading file 4 at iteration 4 after yield at iteration 5 reading file 5 at iteration 5 after yield at iteration 6 reading file 6 at iteration 6 after yield at iteration 7 reading file 7 at iteration 7 after yield at iteration 8 reading file 8 at iteration 8 after yield at iteration 9 reading file 9 at iteration 9 after yield end of for loop .. GENERATED FROM PYTHON SOURCE LINES 207-208 We wee that the string `end of for loo` is printed only once at the end. .. GENERATED FROM PYTHON SOURCE LINES 210-213 yield from ---------- Consider the case where we have a list which further consists of one or more lists .. GENERATED FROM PYTHON SOURCE LINES 213-216 .. code-block:: default my_list = [1, 2, [3, 4], 5] .. GENERATED FROM PYTHON SOURCE LINES 217-219 If we want to yield one element from this list by flattening it, the naive approach would be following .. GENERATED FROM PYTHON SOURCE LINES 219-235 .. code-block:: default def flatten(elements): for elem in elements: if isinstance(elem, list): for _elem in elem: yield _elem else: yield elem flattener = flatten(my_list) for i in flattener: print(i) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 2 3 4 5 .. GENERATED FROM PYTHON SOURCE LINES 236-240 Above we iterate over each value (elem) of my_list. Whenever, ``elem`` is itself a list, we make another ``for`` loop. However, we can achieve the same thing using ``yield from`` keyword. .. GENERATED FROM PYTHON SOURCE LINES 240-250 .. code-block:: default def flatten(elements): for elem in elements: if isinstance(elem, list): yield from elem else: yield elem .. GENERATED FROM PYTHON SOURCE LINES 251-253 Above, instead of writing another ``for`` loop for ``elem``, we are using the keyword ``yield from``. .. GENERATED FROM PYTHON SOURCE LINES 253-259 .. code-block:: default flattener = flatten([1, 2, [3, 4], 5]) for i in flattener: print(i) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 2 3 4 5 .. GENERATED FROM PYTHON SOURCE LINES 260-262 What if we have list inside list of another list or a variation of such nested lists? .. GENERATED FROM PYTHON SOURCE LINES 264-269 .. code-block:: default flattener = flatten([1, 2, [3, 4], 5, [6, [7, 8]]]) for i in flattener: print(i) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 2 3 4 5 6 [7, 8] .. GENERATED FROM PYTHON SOURCE LINES 270-272 ``[7,8]`` is printed in same line, which means this inner list is not flattened by our function. .. GENERATED FROM PYTHON SOURCE LINES 274-278 We may be tempted to add another ``if`` statement for checking whether any member in ``elem`` is a list or not. However, we can call the ``flatten`` function from inside so that the recursion continues until we flatten the list to its last/innermost member. .. GENERATED FROM PYTHON SOURCE LINES 278-292 .. code-block:: default def flatten(elements): for elem in elements: if isinstance(elem, list): yield from flatten(elem) # we call the function again else: yield elem flattener = flatten([1, 2, [3, 4], 5, [6, [7, 8]]]) for i in flattener: print(i) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 2 3 4 5 6 7 8 .. GENERATED FROM PYTHON SOURCE LINES 293-294 Now the list is flattened to its innermost member. .. GENERATED FROM PYTHON SOURCE LINES 296-300 .. code-block:: default flattener = flatten([1, 2, [3, 4], 5, [6, [7, [8, 9, 10]]]]) for i in flattener: print(i) .. rst-class:: sphx-glr-script-out .. code-block:: none 1 2 3 4 5 6 7 8 9 10 .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 0.011 seconds) .. _sphx_glr_download_auto_examples_basics_generators.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/AtrCheema/python-seekho/master?urlpath=lab/tree/notebooks/auto_examples/basics/generators.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: generators.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: generators.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_