.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/basics/io_operations.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_basics_io_operations.py: =========================== 1.17 read/write operations =========================== This lesson shows how to read/write files using python. .. GENERATED FROM PYTHON SOURCE LINES 10-14 creating new file ------------------- If we want to create a new file we can make use of ``open`` function. The second argument here ``w`` defines that we would like to create a new file. .. GENERATED FROM PYTHON SOURCE LINES 14-17 .. code-block:: default open("NewFile.txt", "w") .. rst-class:: sphx-glr-script-out .. code-block:: none <_io.TextIOWrapper name='NewFile.txt' mode='w' encoding='UTF-8'> .. GENERATED FROM PYTHON SOURCE LINES 18-20 The first argument to ``open`` must be complete path of the file. If only file name is given, it means the file will be created at current working directory. .. GENERATED FROM PYTHON SOURCE LINES 22-23 The ``open`` function returns a file handler which can be useful. .. GENERATED FROM PYTHON SOURCE LINES 23-26 .. code-block:: default new_file = open("NewFile.txt", "w") print(type(new_file)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 27-29 Using the file handler, we can check whether a file is open or closed. if a file is open, we can write in it. If it is closed, we can not. .. GENERATED FROM PYTHON SOURCE LINES 29-32 .. code-block:: default print(new_file.closed) .. rst-class:: sphx-glr-script-out .. code-block:: none False .. GENERATED FROM PYTHON SOURCE LINES 33-34 We must close the file once we are done working with a file. .. GENERATED FROM PYTHON SOURCE LINES 34-39 .. code-block:: default new_file.close() print(new_file.closed) .. rst-class:: sphx-glr-script-out .. code-block:: none True .. GENERATED FROM PYTHON SOURCE LINES 40-43 Instead of manually closing the file everytime, we can automatically close it using ``with`` keyword. ``with`` is called context manager which makes sure that whenever we are out of it, the file is closed. .. GENERATED FROM PYTHON SOURCE LINES 43-49 .. code-block:: default with open("NewFile.txt", "w") as fp: pass print(fp.closed) .. rst-class:: sphx-glr-script-out .. code-block:: none True .. GENERATED FROM PYTHON SOURCE LINES 50-52 Even if there is an error during writing/reading the file, the context manager makes sure that the file is closed despite the error. .. GENERATED FROM PYTHON SOURCE LINES 52-58 .. code-block:: default # uncomment following lines # with open("NewFile.txt", 'w') as fp: # fp.write("Bajwa chor hai") # raise NotImplementedError(f"You are not allowed to utter this.") .. GENERATED FROM PYTHON SOURCE LINES 59-61 If we uncomment and run the above cell, it will result in error but we can confirm that the file is still closed. We can verify this using following statement .. GENERATED FROM PYTHON SOURCE LINES 61-63 .. code-block:: default print(fp.closed) # --> True .. rst-class:: sphx-glr-script-out .. code-block:: none True .. GENERATED FROM PYTHON SOURCE LINES 64-71 writing to a new file ---------------------- If we want to write to a new file, we can make use of ``open`` function but with ``w`` as second argument. Context manager also returns us a file handler which can be used to read/write the file .. GENERATED FROM PYTHON SOURCE LINES 71-75 .. code-block:: default with open("NewFile.txt", "w") as fp: fp.write("My Name is Ali") .. GENERATED FROM PYTHON SOURCE LINES 76-80 Here, ``fp`` is file handler, which can can use to modify file. The ``write`` function is for writing a string. If we want to write a list of strings, we can make use of ``writelines`` function. .. GENERATED FROM PYTHON SOURCE LINES 80-86 .. code-block:: default lines = ['His name was Ali', 'He was very brave'] with open("NewFile.txt", "w") as fp: fp.writelines(lines) .. GENERATED FROM PYTHON SOURCE LINES 87-90 If we write something to an existing file and open the file with ``w``, the previous file will be overwritten. This can also cause loss of data. .. GENERATED FROM PYTHON SOURCE LINES 90-96 .. code-block:: default new_lines = ['His name was Hussain.\n', 'He was very brave\n'] with open("NewFile.txt", "w") as fp: fp.writelines(new_lines) .. GENERATED FROM PYTHON SOURCE LINES 97-99 If we see the `NewFile.txt` we will see that it has `new_lines` and not `lines. This is because the previous file was deleted altogether. .. GENERATED FROM PYTHON SOURCE LINES 101-105 writing to already existing file ---------------------------------- If want to write to an already existing file, the second argument to ``open`` function must be ``a`` .. GENERATED FROM PYTHON SOURCE LINES 105-111 .. code-block:: default lines = ['He was killed in 661'] with open("NewFile.txt", "a") as fp: fp.writelines(lines) .. GENERATED FROM PYTHON SOURCE LINES 112-115 writing with specific separator --------------------------------- Consider that we want to write following data into a file. .. GENERATED FROM PYTHON SOURCE LINES 115-117 .. code-block:: default lines = ["1 2 3", "1 2 3", "1 2 3"] .. GENERATED FROM PYTHON SOURCE LINES 118-119 We can do this using following lines of code. .. GENERATED FROM PYTHON SOURCE LINES 121-126 .. code-block:: default with open("NewFile", 'w') as fp: for line in lines: line = ','.join(line.split()) fp.write(line + '\n') .. GENERATED FROM PYTHON SOURCE LINES 127-132 Even though we have not defined the extension of file i.e., the file name is `NewFile`, but this file is still a text file which can be opened by any text editor. Above we used comma ``,`` to separate each value in a line. However, we can use any other separator that we wish e.g. a tab. .. GENERATED FROM PYTHON SOURCE LINES 132-138 .. code-block:: default with open("NewFile", 'w') as fp: for line in lines: line = '\t'.join(line.split()) fp.write(line + '\n') .. GENERATED FROM PYTHON SOURCE LINES 139-143 reading a file ------------------- If we want to read a file, the second argument to ``open`` function must be ``r``. However, the file must exist at the location which is specified. .. GENERATED FROM PYTHON SOURCE LINES 143-168 .. code-block:: default text = """Nb C O Cr 1.0 9.461699 0.0 0.0 0.590249 0.31933 29.99 Nb C O Cr 18 9 18 1 Cartesian 0.13 1.87 11.074 -1.44 4.60 11.076 -3.02 7.33 11.075 3.28 1.85 11.040""" with open('NewFile', 'w') as fp: fp.write(text) lines = [] with open('NewFile', 'r') as fp: for line_num, line in enumerate(fp.readlines()): if line_num > 6: line = line.split() # split the line based upon spaces and put all members into a list lines.append([float(num) for num in line]) print(lines) .. rst-class:: sphx-glr-script-out .. code-block:: none [[0.13, 1.87, 11.074], [-1.44, 4.6, 11.076], [-3.02, 7.33, 11.075], [3.28, 1.85, 11.04]] .. GENERATED FROM PYTHON SOURCE LINES 169-174 We are using ``readlines`` function for reading all the lines lin the file. The ``readlines()`` actually returns us a ``list`` on which we can iterate. Then we are iterating over these lines one by one. Next we are saving the lines in `lines` list after line number 7. Above the first argument is only file name. This means that the file must exists in current folder (working directory). .. GENERATED FROM PYTHON SOURCE LINES 176-181 reading large files ---------------------- The problem with above methodology is that it reads all the lines into memory. If however, the file is large (in GigaBytes), we may not wish to read whole file into memory. In that case we can read line by line. .. GENERATED FROM PYTHON SOURCE LINES 181-186 .. code-block:: default with open('NewFile', 'r') as fp: for idx, line in enumerate(fp): pass .. GENERATED FROM PYTHON SOURCE LINES 187-189 At every iteration, the previous ``line`` from memory is overwritten by the new ``line``. .. GENERATED FROM PYTHON SOURCE LINES 192-197 writing to a specific line --------------------------- If we want to write to a specific line in a file or modify a specific line in a file, we can achieve this by reading the whole lines and then adding/changing those specific lines and then writing the modified lines back to the same file. .. GENERATED FROM PYTHON SOURCE LINES 197-206 .. code-block:: default with open('NewFile', 'r') as fp: lines = fp.readlines() lines.insert(9, '1.111 2.2222 3.33333\n') # 9 means line number 10 with open('NewFile', 'w') as fp: fp.writelines(lines) .. GENERATED FROM PYTHON SOURCE LINES 207-211 writing json format --------------------- `json` is a human readable file format. This file format is similar to python dictionary. .. GENERATED FROM PYTHON SOURCE LINES 211-219 .. code-block:: default import json data = {"name": "Ali", "age": 63, 'is_bold': True} with open("NewFile.json", "w") as fp: json.dump(data, fp) .. GENERATED FROM PYTHON SOURCE LINES 220-222 The data that we wrote above, was a dictionary but we can write any other data to json file as far as it is of native python type. .. GENERATED FROM PYTHON SOURCE LINES 222-226 .. code-block:: default with open("NewFile.json", "w") as fp: json.dump([1, 2, 3], fp) .. GENERATED FROM PYTHON SOURCE LINES 227-229 By setting the ``indent`` keyword argument, we can make sure that all the data is not saved on a single line. This makes the json file more readable. .. GENERATED FROM PYTHON SOURCE LINES 229-233 .. code-block:: default with open("NewFile.json", "w") as fp: json.dump(data, fp, indent=True) .. GENERATED FROM PYTHON SOURCE LINES 234-235 We can sort the keys of saved dictionary in json file by setting ``sort_keys`` to True. .. GENERATED FROM PYTHON SOURCE LINES 235-239 .. code-block:: default with open("NewFile.json", "w") as fp: json.dump(data, fp, indent=True, sort_keys=True) .. GENERATED FROM PYTHON SOURCE LINES 240-242 However, the json file can save only native python types. If the data is not native python type, we get the ``TypeError`` .. GENERATED FROM PYTHON SOURCE LINES 242-251 .. code-block:: default import numpy as np np_data = np.array([2]) # uncomment following two lines, they will return in TypeError # with open("jsonfile.json", "w") as fp: # json.dump(np_data, fp) # -> TypeError: Object of type ndarray is not JSON serializable .. GENERATED FROM PYTHON SOURCE LINES 252-255 The error message very explicit says that the data we are trying to save is `ndarray `_ and it can not be `serialized `_ . .. GENERATED FROM PYTHON SOURCE LINES 257-258 We can verify this by checking the type of `data`. .. GENERATED FROM PYTHON SOURCE LINES 258-261 .. code-block:: default print(type(np_data)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 262-264 `data` is an array, which means it consists of multiple values. We can get the first value of `data` by slicing it using slice operator ``[]``. .. GENERATED FROM PYTHON SOURCE LINES 264-269 .. code-block:: default np_data_0 = np_data[0] print(np_data_0) .. rst-class:: sphx-glr-script-out .. code-block:: none 2 .. GENERATED FROM PYTHON SOURCE LINES 270-271 Now If we try to save it in json file, .. GENERATED FROM PYTHON SOURCE LINES 271-276 .. code-block:: default # Uncomment following two lines, they will result in TypeError # with open("jsonfile.json", "w") as fp: # json.dump(np_data_0, fp) # -> TypeError: Object of type int32 is not JSON serializable .. GENERATED FROM PYTHON SOURCE LINES 277-279 The above error message says that ``int32`` is also not serializable. This is because the first member of ``data`` is ``int32`` type which is from numpy library. .. GENERATED FROM PYTHON SOURCE LINES 279-282 .. code-block:: default print(type(np_data_0)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 283-287 This is because ``int32`` is also not python's native type but is from numpy library. We can convert ``int32`` into python's ``int`` type and then we can save it into json file format. .. GENERATED FROM PYTHON SOURCE LINES 287-291 .. code-block:: default np_data_0_int = int(np_data_0) print(type(np_data_0_int)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 292-296 .. code-block:: default with open("NewFile.json", "w") as fp: json.dump(np_data_0_int, fp) .. GENERATED FROM PYTHON SOURCE LINES 297-304 .. code-block:: default np_data = np.array([2, 3, 4]) # Uncomment following two lines, they will result in TypeError # with open("jsonfile.json", "w") as fp: # json.dump(np_data, fp) .. GENERATED FROM PYTHON SOURCE LINES 305-307 The ``tolist`` method of numpy array converts the numpy array into list which is python native type and can be saved as json. .. GENERATED FROM PYTHON SOURCE LINES 307-313 .. code-block:: default data_list = np_data.tolist() with open("NewFile.json", "w") as fp: json.dump(data_list, fp) .. GENERATED FROM PYTHON SOURCE LINES 314-318 reading json format --------------------- In order to read the json file, we can make use of ``json.load()`` function. The first argument must be file path. .. GENERATED FROM PYTHON SOURCE LINES 318-324 .. code-block:: default with open("NewFile.json", "r") as fp: data = json.load(fp) print(data) .. rst-class:: sphx-glr-script-out .. code-block:: none [2, 3, 4] .. GENERATED FROM PYTHON SOURCE LINES 325-326 The type of the data is preserved when we load the json file. .. GENERATED FROM PYTHON SOURCE LINES 326-329 .. code-block:: default print(type(data)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 330-342 writing in binary format ------------------------- Above when we wrote the data, the saved file was human readable. This means that you can open the file and see/read the data. However, there is a cost of doing this. If the data is large, the file size gets extremely large and the process of reading and writing becomes slow. This can be avoided by writing the data into binary format. The downside here is that the written file is not human readable unless you have specific software e.g `HDFView `_ . These software actually convert the binary data into human readable and show it. Saving the data into binary format is a very large topic and there are many built-in and third-party libraries in python for it. However, here we will only cover basics of ``pickle`` module of python. .. GENERATED FROM PYTHON SOURCE LINES 342-345 .. code-block:: default import pickle .. GENERATED FROM PYTHON SOURCE LINES 346-349 When we want to save the data into binary format/file, the second argument to ``open`` function must be ``wb``. Here, ``w`` means that we are creating a new file and ``b`` represents that the data will be written in binary format. .. GENERATED FROM PYTHON SOURCE LINES 349-354 .. code-block:: default my_bytes = [120, 3, 255, 0, 1000] with open("NewFile", "wb") as my_pickle_file: pickle.dump(my_bytes, my_pickle_file) .. GENERATED FROM PYTHON SOURCE LINES 355-356 Above all the elements in list were integer, however they can also be float .. GENERATED FROM PYTHON SOURCE LINES 356-361 .. code-block:: default my_bytes = [120, 3, 255, 0, 1000.0] with open("NewFile", "wb") as my_pickle_file: pickle.dump(my_bytes, my_pickle_file) .. GENERATED FROM PYTHON SOURCE LINES 362-363 also string data can be saved as binary using pickle module. .. GENERATED FROM PYTHON SOURCE LINES 363-368 .. code-block:: default my_bytes = [120, 3, 255, 0, 1000.0, 'a'] with open("NewFile", "wb") as my_pickle_file: pickle.dump(my_bytes, my_pickle_file) .. GENERATED FROM PYTHON SOURCE LINES 369-370 Similarly we can write ``tuple`` or ``dictionary`` data into binary format. .. GENERATED FROM PYTHON SOURCE LINES 370-375 .. code-block:: default my_bytes = [120, 3, 255, 0, 1000.0, 'a', (1, 2), None] with open("NewFile", "wb") as my_pickle_file: pickle.dump(my_bytes, my_pickle_file) .. GENERATED FROM PYTHON SOURCE LINES 376-381 .. code-block:: default my_bytes = [-1200, 3, 255, 0, 1000.0, 'a', {'a': 1}, True] with open("NewFile", "wb") as my_pickle_file: pickle.dump(my_bytes, my_pickle_file) .. GENERATED FROM PYTHON SOURCE LINES 382-386 reading binary format ----------------------- If we want to read binary file, we can use ``rb`` keyword in ``open`` function as second argument. .. GENERATED FROM PYTHON SOURCE LINES 386-392 .. code-block:: default with open("NewFile", "rb") as my_pickle_file: my_bytes = pickle.load(my_pickle_file) print(my_bytes) .. rst-class:: sphx-glr-script-out .. code-block:: none [-1200, 3, 255, 0, 1000.0, 'a', {'a': 1}, True] .. GENERATED FROM PYTHON SOURCE LINES 393-394 The pickle module can read/write a wide range of data types. .. GENERATED FROM PYTHON SOURCE LINES 394-404 .. code-block:: default my_bytes = np.array([120, 3, 255, 0, 1000.0]) with open("NewFile", "wb") as my_pickle_file: pickle.dump(my_bytes, my_pickle_file) with open("NewFile", "rb") as my_pickle_file: my_bytes = pickle.load(my_pickle_file) print(type(my_bytes)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 405-407 Above we wrote numpy data type which is not python's native data type. Moreover, when we read binary file, we still got numpy data type. .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 0.017 seconds) .. _sphx_glr_download_auto_examples_basics_io_operations.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/AtrCheema/python-seekho/master?urlpath=lab/tree/notebooks/auto_examples/basics/io_operations.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: io_operations.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: io_operations.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_