5.1 introduction
Contents
Note
Click here to download the full example code or to run this example in your browser via Binder
5.1 introduction#
import numpy as np
import pandas as pd
print(pd.__version__, np.__version__)
1.5.3 1.26.3
Suppose we have an array [0.4, 0.3, 0.5, 0.2, 0.6, 0.3]. Let’s say
the values in this array represent concentrations in water measured
every hour from 13 pm to 19 pm. However, with just an array, we don’t
have the ability to encode this information. If we want to add the (temporal) reference of each value
we have to add it ourself for example by saving that in a separate array.
Pandas comes with this in-built ability that we can add reference or labels to arrays.
Every array in pandas has two kinds of references. The reference for the rows which
is called index and the reference for the columns which is called columns.
Therefore we can call pandas a library which have referenced/labelled arrays.
The core data structure in pandas is DataFrame which consists of one or more
columns. A single column in a DataFrame is a Series.
df = pd.DataFrame(np.random.random((10, 3)))
print(df)
0 1 2
0 0.119385 0.212646 0.773571
1 0.651526 0.231232 0.662007
2 0.921970 0.731259 0.769609
3 0.777924 0.019730 0.053925
4 0.571768 0.795749 0.763418
5 0.506534 0.737717 0.469336
6 0.727143 0.713029 0.610969
7 0.738030 0.211512 0.747559
8 0.619142 0.135396 0.195000
9 0.793140 0.076853 0.739200
The data in columns is stored as numpy arrays. Therefore, a DataFrames and Series have a lot of characteristics similar to that of numpy arrays.
print(df.shape)
(10, 3)
By default the columns names are just integers starting from 0, however we can define the column names ourselves as well.
df = pd.DataFrame(np.random.random((10, 3)), columns=['a', 'b', 'c'])
print(df)
a b c
0 0.558031 0.024179 0.344116
1 0.028144 0.726968 0.899165
2 0.099216 0.579136 0.979136
3 0.004508 0.528733 0.324257
4 0.906279 0.327970 0.950071
5 0.044544 0.855949 0.192976
6 0.434093 0.375051 0.047678
7 0.599005 0.791190 0.892243
8 0.875665 0.873897 0.455042
9 0.247549 0.855402 0.947856
print(df.columns)
Index(['a', 'b', 'c'], dtype='object')
The columns are list like structures. However they are not exactly lists.
type(df.columns)
We can however, convert the columns to list though.
['a', 'b', 'c']
type(df.columns.to_list())
The default label for the rows i.e. index consists of numbers starting from 0.
print(df.index)
RangeIndex(start=0, stop=10, step=1)
However, we can set index of our choice as well.
df = pd.DataFrame(np.random.random((10, 3)),
columns=['a', 'b', 'c'],
index=[2000+i for i in range(10)])
print(df)
a b c
2000 0.487772 0.597249 0.566402
2001 0.804949 0.076963 0.096471
2002 0.660040 0.491927 0.858581
2003 0.454847 0.286776 0.638591
2004 0.855336 0.337863 0.592056
2005 0.768659 0.863738 0.042553
2006 0.583440 0.951762 0.296148
2007 0.857781 0.667254 0.118941
2008 0.335035 0.182315 0.431538
2009 0.742475 0.894606 0.594587
print(df.index)
Int64Index([2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009], dtype='int64')
The default name of index is None.
print(df.index.name)
None
However, we can set the name of index as well.
df.index.name = 'years'
print(df)
a b c
years
2000 0.487772 0.597249 0.566402
2001 0.804949 0.076963 0.096471
2002 0.660040 0.491927 0.858581
2003 0.454847 0.286776 0.638591
2004 0.855336 0.337863 0.592056
2005 0.768659 0.863738 0.042553
2006 0.583440 0.951762 0.296148
2007 0.857781 0.667254 0.118941
2008 0.335035 0.182315 0.431538
2009 0.742475 0.894606 0.594587
print(df.index.name)
years
type(df)
df = pd.DataFrame(np.random.randint(0, 10, (10, 1)),
columns=['a'],
index=[2000+i for i in range(10)])
print(df)
a
2000 7
2001 1
2002 9
2003 1
2004 5
2005 1
2006 8
2007 3
2008 6
2009 0
type(df)
print(df.columns)
Index(['a'], dtype='object')
Series#
A Series consists of a single column. It can be constructed using pd.Series.
s = pd.Series(np.random.random(10))
print(s)
0 0.221007
1 0.149426
2 0.002763
3 0.095406
4 0.880150
5 0.646121
6 0.897058
7 0.265881
8 0.267116
9 0.703218
dtype: float64
type(s)
print(s.shape)
(10,)
print(s.name)
None
s = pd.Series(np.random.random(10),
name="a")
print(s)
0 0.706505
1 0.009507
2 0.203233
3 0.947896
4 0.712221
5 0.162364
6 0.148728
7 0.603879
8 0.441128
9 0.193225
Name: a, dtype: float64
print(s.name)
a
the Series is literally the data structure for a single column of a DataFrame.
df = pd.DataFrame(np.random.random((10, 3)),
columns=['a', 'b', 'c'],
index=[2000+i for i in range(10)])
print(df)
a b c
2000 0.173367 0.818694 0.626084
2001 0.723843 0.645404 0.353689
2002 0.336572 0.022853 0.629518
2003 0.173588 0.183761 0.423902
2004 0.344363 0.588909 0.161421
2005 0.155139 0.144516 0.815570
2006 0.963709 0.382972 0.967488
2007 0.138722 0.984276 0.991498
2008 0.626236 0.333986 0.653095
2009 0.170549 0.018864 0.122101
A single column in a DataFrame is a Series.
print(type(df['a']))
<class 'pandas.core.series.Series'>
s = pd.Series(np.random.random(10),
index=[2000+i for i in range(10)],
name="a")
print(s)
2000 0.506574
2001 0.615551
2002 0.507256
2003 0.434326
2004 0.206546
2005 0.299607
2006 0.616760
2007 0.691642
2008 0.277358
2009 0.642254
Name: a, dtype: float64
Since pandas is based upon numpy arrays. We can extract actual numpy arrays from DataFrame using .values method.
print(df.values)
[[0.1733671 0.81869419 0.62608396]
[0.72384345 0.64540401 0.35368909]
[0.33657217 0.02285333 0.62951771]
[0.17358839 0.18376112 0.42390209]
[0.34436257 0.58890914 0.16142116]
[0.15513896 0.14451641 0.81556994]
[0.9637094 0.38297155 0.96748806]
[0.13872247 0.98427641 0.99149836]
[0.62623596 0.3339857 0.65309479]
[0.17054927 0.01886448 0.1221009 ]]
type(df.values)
df = pd.DataFrame(np.random.randint(0, 14, (10, 3)),
columns=['a', 'b', 'c'],
index=[2000+i for i in range(10)])
print(df)
a b c
2000 3 1 10
2001 13 0 8
2002 0 11 11
2003 9 10 4
2004 4 3 13
2005 1 8 0
2006 0 2 2
2007 7 8 8
2008 5 8 4
2009 1 3 6
type(df.values)
print(df.values.shape)
(10, 3)
df.head()
df.head(8)
Get the last N rows of a DataFrame
df.tail()
df.tail(7)
df.mean()
a 4.3
b 5.4
c 6.6
dtype: float64
{'a': {2000: 3, 2001: 13, 2002: 0, 2003: 9, 2004: 4, 2005: 1, 2006: 0, 2007: 7, 2008: 5, 2009: 1}, 'b': {2000: 1, 2001: 0, 2002: 11, 2003: 10, 2004: 3, 2005: 8, 2006: 2, 2007: 8, 2008: 8, 2009: 3}, 'c': {2000: 10, 2001: 8, 2002: 11, 2003: 4, 2004: 13, 2005: 0, 2006: 2, 2007: 8, 2008: 4, 2009: 6}}
df.to_dict('list')
{'a': [3, 13, 0, 9, 4, 1, 0, 7, 5, 1], 'b': [1, 0, 11, 10, 3, 8, 2, 8, 8, 3], 'c': [10, 8, 11, 4, 13, 0, 2, 8, 4, 6]}
df['d'] = np.random.randint(0, 10, (10,))
print(df)
a b c d
2000 3 1 10 0
2001 13 0 8 2
2002 0 11 11 8
2003 9 10 4 2
2004 4 3 13 1
2005 1 8 0 4
2006 0 2 2 1
2007 7 8 8 7
2008 5 8 4 3
2009 1 3 6 7
a b c
2000 3 1 10
2001 13 0 8
2002 0 11 11
2003 9 10 4
2004 4 3 13
2005 1 8 0
2006 0 2 2
2007 7 8 8
2008 5 8 4
2009 1 3 6
df.columns = ['x', 'y', 'z']
print(df)
x y z
2000 3 1 10
2001 13 0 8
2002 0 11 11
2003 9 10 4
2004 4 3 13
2005 1 8 0
2006 0 2 2
2007 7 8 8
2008 5 8 4
2009 1 3 6
row count of pandas dataframe
len(df.index)
10
print(df.shape[0])
10
change the order of DataFrame columns
z x y
2000 10 3 1
2001 8 13 0
2002 11 0 11
2003 4 9 10
2004 13 4 3
2005 0 1 8
2006 2 0 2
2007 8 7 8
2008 4 5 8
2009 6 1 3
drop rows of Pandas DataFrame whose value in a certain column is NaN
df = pd.DataFrame(np.random.randn(6,3))
print(df)
0 1 2
0 0.412635 -2.039483 0.610389
1 -2.010406 1.486636 -1.388038
2 0.419472 0.351581 0.665132
3 -0.054123 1.298534 1.455114
4 0.394011 -1.569920 -1.175858
5 -0.733748 0.387037 -2.159622
0 1 2
0 NaN -2.039483 NaN
1 -2.010406 1.486636 -1.388038
2 NaN 0.351581 0.665132
3 -0.054123 1.298534 NaN
4 NaN -1.569920 NaN
5 -0.733748 0.387037 -2.159622
dropping all rows having NaN values
dropping NaN in specific columns
0 1 2
1 -2.010406 1.486636 -1.388038
2 NaN 0.351581 0.665132
5 -0.733748 0.387037 -2.159622
count the NaN values in a column in DataFrame
df = pd.DataFrame(np.random.randn(6,3))
df.iloc[::2,0] = np.nan; df.iloc[::4,2] = np.nan; df.iloc[::3,2] = np.nan
print(df)
0 1 2
0 NaN -0.425574 NaN
1 0.704846 1.779992 0.204399
2 NaN 1.344717 0.348908
3 0.213708 1.009525 NaN
4 NaN 0.793162 NaN
5 1.311316 1.664391 2.299337
df.isna().sum()
0 3
1 0
2 3
dtype: int64
for columns
df.isnull().sum(axis = 0)
0 3
1 0
2 3
dtype: int64
for rows
df.isnull().sum(axis = 1)
0 2
1 0
2 1
3 1
4 2
5 0
dtype: int64
check if any value is NaN in a DataFrame
df = pd.DataFrame(np.random.randn(6,3))
df.iloc[::2,0] = np.nan; df.iloc[::4,2] = np.nan; df.iloc[::3,2] = np.nan
print(df)
0 1 2
0 NaN 0.796718 NaN
1 0.425959 -1.341634 -1.609487
2 NaN -1.204189 -1.416200
3 -0.189283 -0.018870 NaN
4 NaN 0.171775 NaN
5 -0.190172 0.673441 -1.075047
how many NaN
column wise
df.isnull().any()
0 True
1 False
2 True
dtype: bool
if there is any NaN in entire data
df.isnull().any().any()
True
replace NaN values by Zeroes in a column of a Dataframe?
df = pd.DataFrame(np.random.randn(6,3))
df.iloc[::2,0] = np.nan; df.iloc[::4,2] = np.nan; df.iloc[::3,2] = np.nan
print(df)
0 1 2
0 NaN -1.407393 NaN
1 0.029791 0.728372 -1.683737
2 NaN -1.013543 0.927217
3 -0.113928 1.111284 NaN
4 NaN -0.861306 NaN
5 0.810280 0.322339 -1.948100
df.fillna(0)
To fill the NaNs in only one column
0 1 2
0 NaN -1.407393 0.000000
1 0.029791 0.728372 -1.683737
2 NaN -1.013543 0.927217
3 -0.113928 1.111284 0.000000
4 NaN -0.861306 0.000000
5 0.810280 0.322339 -1.948100
check if a column exists in Pandas
df = pd.DataFrame(np.random.randn(6,3))
print(df)
0 1 2
0 -0.482179 -0.291660 0.268568
1 0.933887 -0.399380 0.354771
2 -0.179937 -0.838072 -0.264850
3 -0.051832 0.459479 0.399571
4 -0.830171 -0.428108 -1.246793
5 -0.690748 0.772872 -0.768607
if 0 in df.columns:
print("true")
true
Python dict into a dataframe
d = {
'2012-06-08': 388,
'2012-06-09': 388,
'2012-06-10': 388,
'2012-06-11': 389,
'2012-06-12': 389,
'2012-06-13': 389,
'2012-06-14': 389,
'2012-06-15': 389,
'2012-06-16': 389,
'2012-06-17': 389,
'2012-06-18': 390,
'2012-06-19': 390,
'2012-06-20': 390,
}
pd.DataFrame(d.items())
pd.DataFrame(d.items(), columns=['Date', 'DateValue'])
uncomment following line pd.DataFrame(d) # ValueError: If using all scalar values, you must pass an index
pd.DataFrame([d])
pd.DataFrame.from_dict(d, orient='index', columns=['DateVaue'])
Count the frequency that a value occurs in a dataframe column
df = pd.DataFrame(np.random.randint(0, 14, (10, 3)),
columns=['a', 'b', 'c'],
index=[2000+i for i in range(10)])
df['a'].value_counts()
7 2
1 2
9 2
4 1
5 1
11 1
8 1
Name: a, dtype: int64
for index, row in df.iterrows():
print(index, row, '\n')
2000 a 5
b 9
c 8
Name: 2000, dtype: int64
2001 a 4
b 11
c 2
Name: 2001, dtype: int64
2002 a 7
b 7
c 2
Name: 2002, dtype: int64
2003 a 9
b 9
c 5
Name: 2003, dtype: int64
2004 a 11
b 7
c 13
Name: 2004, dtype: int64
2005 a 1
b 11
c 2
Name: 2005, dtype: int64
2006 a 1
b 3
c 3
Name: 2006, dtype: int64
2007 a 7
b 4
c 0
Name: 2007, dtype: int64
2008 a 9
b 1
c 2
Name: 2008, dtype: int64
2009 a 8
b 9
c 7
Name: 2009, dtype: int64
df = pd.DataFrame(np.random.randint(0, 14, (10, 3)),
columns=['a', 'b', 'c'])
print(df)
a b c
0 12 8 7
1 8 9 13
2 0 5 3
3 7 10 6
4 7 7 9
5 5 7 7
6 11 9 3
7 6 11 4
8 13 8 1
9 0 8 7
0 1.500000
1 0.888889
2 0.000000
3 0.700000
4 1.000000
5 0.714286
6 1.222222
7 0.545455
8 1.625000
9 0.000000
dtype: float64
add an empty column to a dataframe?
a b c d
0 12 8 7
1 8 9 13
2 0 5 3
3 7 10 6
4 7 7 9
5 5 7 7
6 11 9 3
7 6 11 4
8 13 8 1
9 0 8 7
print(df['d'])
0
1
2
3
4
5
6
7
8
9
Name: d, dtype: object
a b c d
0 12 8 7 NaN
1 8 9 13 NaN
2 0 5 3 NaN
3 7 10 6 NaN
4 7 7 9 NaN
5 5 7 7 NaN
6 11 9 3 NaN
7 6 11 4 NaN
8 13 8 1 NaN
9 0 8 7 NaN
What does axis in pandas mean?
df.mean(axis=0)
a 6.9
b 8.2
c 6.0
d NaN
dtype: float64
df.mean(axis=1)
0 9.000000
1 10.000000
2 2.666667
3 7.666667
4 7.666667
5 6.333333
6 7.666667
7 7.000000
8 7.333333
9 5.000000
dtype: float64
Replace NaN with blank/empty string
df.replace(9, np.nan)
df.replace(np.nan, '')
Rename specific column(s) in pandas
df = pd.DataFrame(np.random.randint(0, 14, (10, 3)), columns=['a', 'b', 'c'])
print(df)
a b c
0 9 8 7
1 11 8 8
2 6 12 2
3 0 2 5
4 11 10 4
5 8 4 1
6 0 2 9
7 10 7 4
8 5 10 6
9 7 9 11
log(A) b c
0 9 8 7
1 11 8 8
2 6 12 2
3 0 2 5
4 11 10 4
5 8 4 1
6 0 2 9
7 10 7 4
8 5 10 6
9 7 9 11
print DataFrame without index
print(df)
log(A) b c
0 9 8 7
1 11 8 8
2 6 12 2
3 0 2 5
4 11 10 4
5 8 4 1
6 0 2 9
7 10 7 4
8 5 10 6
9 7 9 11
/home/docs/checkouts/readthedocs.org/user_builds/python-seekho/checkouts/latest/scripts/pandas/dataframe_vs_series.py:476: FutureWarning: this method is deprecated in favour of `Styler.hide(axis="index")`
df.style.hide_index()
replace nan values with average of columns
retrieve the number of columns in a dataframe?
len(df.columns)
3
print(df.shape[1])
3
We can create empty DataFrame by telling how many columns should exist or how many rows should exist.
df = pd.DataFrame(columns=['A','B','C','D','E','F','G'])
print(df)
Empty DataFrame
Columns: [A, B, C, D, E, F, G]
Index: []
print(df.shape)
(0, 7)
df = pd.DataFrame(index=range(1,8))
print(df)
Empty DataFrame
Columns: []
Index: [1, 2, 3, 4, 5, 6, 7]
print(df.shape)
(7, 0)
Total running time of the script: ( 0 minutes 0.487 seconds)
