5.12. Series NA

5.12.1. SetUp

>>> import pandas as pd
>>> import numpy as np

5.12.2. Rationale

  • How to represent missing data in a Series?

  • Using 0 or -1 is not a good idea, because it influences the result of computations.

  • You can use None, float('nan'), np.nan or pd.NA to represent missing data in a Series.

  • age = pd.Series([10, 20, 30, 0])

  • age = pd.Series([10, 20, 30, -1])

  • age = pd.Series([10, 20, 30, None])

  • age = pd.Series([10, 20, 30, float('nan')])

  • age = pd.Series([10, 20, 30, pd.NA])

  • age = pd.Series([10, 20, 30, np.nan])

  • Recommendation: use None in Python data and pd.NA in Pandas data.

Let's imagine we have a Series of ages of our users, and we want to compute the mean age. What should be the result if one of the ages is missing?

What happen if we have a missing value represented by 0 or -1?

>>> age = pd.Series([10, 20, 30, 0])
>>> age.mean()
np.float64(15.0)

>>> age = pd.Series([10, 20, 30, -1])
>>> age.mean()
np.float64(14.75)

It influences the result of the mean, which is not what we want.

There is a better way to represent missing data in a Series, using None, float('nan'), np.nan or pd.NA.

>>> age = pd.Series([10, 20, 30, None])
>>> age.mean()
np.float64(20.0)

>>> age = pd.Series([10, 20, 30, float('nan')])
>>> age.mean()
np.float64(20.0)

>>> age = pd.Series([10, 20, 30, pd.NA])
>>> age.mean()
np.float64(20.0)

>>> age = pd.Series([10, 20, 30, np.nan])
>>> age.mean()
np.float64(20.0)

5.12.3. Repr

>>> age = pd.Series([30, 31, 32, None])
>>> age
0    30.0
1    31.0
2    32.0
3     NaN
dtype: float64

>>> age = pd.Series([10, 20, 30, float('nan')])
>>> age
0    10.0
1    20.0
2    30.0
3     NaN
dtype: float64

>>> age = pd.Series([30, 31, 32, pd.NA])
>>> age
0      30
1      31
2      32
3    <NA>
dtype: object

>>> age = pd.Series([30, 31, 32, np.nan])
>>> age
0    30.0
1    31.0
2    32.0
3     NaN
dtype: float64

5.12.4. Fillna

  • pd.Series.fillna(value) - Fill missing values with a specified value or method.

  • pd.Series.fillna(value, limit) - Fill missing values up to a specified limit.

>>> data = pd.Series(['Alice', 'Bob', 'Carol', None, pd.NA, float('nan'), np.nan])
>>>
>>> data
0    Alice
1      Bob
2    Carol
3      NaN
4      NaN
5      NaN
6      NaN
dtype: str

Fillna with a scalar value:

>>> data.fillna('n/a')
0    Alice
1      Bob
2    Carol
3      n/a
4      n/a
5      n/a
6      n/a
dtype: str

Limit:

>>> data.fillna('n/a', limit=2)
0    Alice
1      Bob
2    Carol
3      n/a
4      n/a
5      NaN
6      NaN
dtype: str

5.12.5. Isna

  • pd.Series.isna() - Detect missing values. Returns a boolean same-sized object indicating if the values are NA. Missing values gets mapped to True values. Everything else gets mapped to False values. NaN, None, and pd.NA are considered missing values.

>>> data = pd.Series(['Alice', 'Bob', 'Carol', None, pd.NA, float('nan'), np.nan])
>>>
>>> data
0    Alice
1      Bob
2    Carol
3      NaN
4      NaN
5      NaN
6      NaN
dtype: str

Isna:

>>> data.isna()
0    False
1    False
2    False
3     True
4     True
5     True
6     True
dtype: bool

Invert:

>>> ~data.isna()
0     True
1     True
2     True
3    False
4    False
5    False
6    False
dtype: bool

5.12.6. Notna

  • pd.Series.notna() - Detect non-missing values. Returns a boolean same-sized object indicating if the values are not NA. Non-missing values gets mapped to True values. Missing values gets mapped to False values. NaN, None, and pd.NA are considered missing values.

>>> data = pd.Series(['Alice', 'Bob', 'Carol', None, pd.NA, float('nan'), np.nan])
>>>
>>> data
0    Alice
1      Bob
2    Carol
3      NaN
4      NaN
5      NaN
6      NaN
dtype: str

Notna:

>>> data.notna()
0     True
1     True
2     True
3    False
4    False
5    False
6    False
dtype: bool

5.12.7. Dropna

  • pd.Series.dropna() - Return a new Series with missing values removed.

>>> data = pd.Series(['Alice', 'Bob', 'Carol', None, pd.NA, float('nan'), np.nan])
>>>
>>> data
0    Alice
1      Bob
2    Carol
3      NaN
4      NaN
5      NaN
6      NaN
dtype: str

Dropna:

>>> data.dropna()
0    Alice
1      Bob
2    Carol
dtype: str

5.12.8. Boolean Value

>>> bool(None)
False

>>> bool(float('nan'))
True

>>> bool(np.nan)
True

>>> bool(pd.NA)
Traceback (most recent call last):
TypeError: boolean value of NA is ambiguous

5.12.9. Type

>>> pd.Series([1, None, 3]).dtype
dtype('float64')
>>> pd.Series([1.0, None, 3.0]).dtype
dtype('float64')
>>> pd.Series([True, None, False]).dtype
dtype('O')
>>> pd.Series(['a', None, 'c']).dtype
<StringDtype(na_value=nan)>

>>> pd.Series([1, float('nan'), 3]).dtype
dtype('float64')
>>> pd.Series([1.0, float('nan'), 3.0]).dtype
dtype('float64')
>>> pd.Series([True, float('nan'), False]).dtype
dtype('O')
>>> pd.Series(['a', float('nan'), 'c']).dtype
<StringDtype(na_value=nan)>

>>> pd.Series([1, np.nan, 3]).dtype
dtype('float64')
>>> pd.Series([1.0, np.nan, 3.0]).dtype
dtype('float64')
>>> pd.Series([True, np.nan, False]).dtype
dtype('O')
>>> pd.Series(['a', np.nan, 'c']).dtype
<StringDtype(na_value=nan)>

>>> pd.Series([1, pd.NA, 3]).dtype
dtype('O')
>>> pd.Series([1.0, pd.NA, 3.0]).dtype
dtype('O')
>>> pd.Series([True, pd.NA, False]).dtype
dtype('O')
>>> pd.Series(['a', pd.NA, 'c']).dtype
<StringDtype(na_value=nan)>

5.12.10. Comparison

>>> None == None
True
>>> None == float('nan')
False
>>> None == np.nan
False
>>> None == pd.NA
False

>>> float('nan') == None
False
>>> float('nan') == float('nan')
False
>>> float('nan') == np.nan
False
>>> float('nan') == pd.NA
<NA>

>>> np.nan == None
False
>>> np.nan == float('nan')
False
>>> np.nan == np.nan
False
>>> np.nan == pd.NA
<NA>

>>> pd.NA == None
False
>>> pd.NA == float('nan')
<NA>
>>> pd.NA == np.nan
<NA>
>>> pd.NA == pd.NA
<NA>

5.12.11. Identity

>>> None is None
True
>>> None is float('nan')
False
>>> None is np.nan
False
>>> None is pd.NA
False

>>> float('nan') is None
False
>>> float('nan') is float('nan')
False
>>> float('nan') is np.nan
False
>>> float('nan') is pd.NA
False

>>> np.nan is None
False
>>> np.nan is float('nan')
False
>>> np.nan is np.nan
True
>>> np.nan is pd.NA
False

>>> pd.NA is None
False
>>> pd.NA is float('nan')
False
>>> pd.NA is np.nan
False
>>> pd.NA is pd.NA
True

5.12.12. Check

  • Negated ~ versions of all above methods

>>> s = pd.Series([1.0, np.nan, 3.0])
>>> s
0    1.0
1    NaN
2    3.0
dtype: float64

>>> s.any()
np.True_
>>> ~s.any()
np.False_

>>> s.all()
np.True_
>>> ~s.all()
np.False_

5.12.13. Select

  • s.isnull() and s.notnull()

  • s.isna() and s.notna()

  • Negated ~ versions of all above methods

>>> s = pd.Series([1.0, np.nan, 3.0])
>>> s
0    1.0
1    NaN
2    3.0
dtype: float64

>>> s.isnull()
0    False
1     True
2    False
dtype: bool

>>> ~s.isnull()
0     True
1    False
2     True
dtype: bool

>>> s.notnull()
0     True
1    False
2     True
dtype: bool

>>> ~s.notnull()
0    False
1     True
2    False
dtype: bool

>>> s = pd.Series([1.0, np.nan, 3.0])
>>> s
0    1.0
1    NaN
2    3.0
dtype: float64
>>>
>>> s.isna()
0    False
1     True
2    False
dtype: bool
>>>
>>> s.notna()
0     True
1    False
2     True
dtype: bool
>>>
>>> ~s.isna()
0     True
1    False
2     True
dtype: bool
>>>
>>> ~s.notna()
0    False
1     True
2    False
dtype: bool

5.12.14. Update

  • Works with inplace=True parameter.

>>> s = pd.Series([1.0, None, None, 4.0, None, 6.0])
>>> s
0    1.0
1    NaN
2    NaN
3    4.0
4    NaN
5    6.0
dtype: float64

Fill NA - Scalar value:

>>> s.fillna(0.0)
0    1.0
1    0.0
2    0.0
3    4.0
4    0.0
5    6.0
dtype: float64

Forward Fill. ffill: propagate last valid observation forward:

>>> s.ffill()
0    1.0
1    1.0
2    1.0
3    4.0
4    4.0
5    6.0
dtype: float64

Backward Fill. bfill: use NEXT valid observation to fill gap:

>>> s.bfill()
0    1.0
1    4.0
2    4.0
3    4.0
4    6.0
5    6.0
dtype: float64

Interpolate. method: str, default linear, no inplace=True option:

>>> s.interpolate()
0    1.0
1    2.0
2    3.0
3    4.0
4    5.0
5    6.0
dtype: float64

Following method requires installation of scipy library:

>>> s.interpolate('nearest')
0    1.0
1    1.0
2    4.0
3    4.0
4    4.0
5    6.0
dtype: float64

Following method requires installation of scipy library:

>>> s.interpolate('polynomial', order=2)
0    1.0
1    2.0
2    3.0
3    4.0
4    5.0
5    6.0
dtype: float64
Table 5.12. Interpolation techniques

Method

Description

linear

Ignore the index and treat the values as equally spaced. This is the only method supported on MultiIndexes

time

Works on daily and higher resolution data to interpolate given length of interval

index, values

use the actual numerical values of the index.

pad

Fill in NA using existing values

nearest, zero, slinear, quadratic, cubic, spline, barycentric, polynomial

Passed to scipy.interpolate.interp1d. These methods use the numerical values of the index. Both polynomial and spline require that you also specify an order (int), e.g. df.interpolate(method='polynomial', order=5)

krogh, piecewise_polynomial, spline, pchip, akima

Wrappers around the SciPy interpolation methods of similar names

from_derivatives

Refers to scipy.interpolate.BPoly.from_derivatives which replaces piecewise_polynomial interpolation method in scipy 0.18.

5.12.15. Drop

Drop Rows. Has inplace=True parameter:

>>> s = pd.Series([1.0, None, None, 4.0, None, 6.0])
>>> s
0    1.0
1    NaN
2    NaN
3    4.0
4    NaN
5    6.0
dtype: float64
>>>
>>> s.dropna()
0    1.0
3    4.0
5    6.0
dtype: float64

5.12.16. Conversion

  • If you have a DataFrame or Series using traditional types that have missing data represented using np.nan

  • There are convenience methods convert_dtypes() in Series and DataFrame that can convert data to use the newer dtypes for integers, strings and booleans

  • This is especially helpful after reading in data sets when letting the readers such as read_csv() and read_excel() infer default dtypes.

>>> #
... data = pd.read_csv('data/baseball.csv', index_col='id')
... data[data.columns[:10]].dtypes
player    object
year       int64
stint      int64
team      object
lg        object
g          int64
ab         int64
r          int64
h          int64
X2b        int64
dtype: object

>>> #
... data = pd.read_csv('data/baseball.csv', index_col='id')
... data = data.convert_dtypes()
... data[data.columns[:10]].dtypes
player    string
year       Int64
stint      Int64
team      string
lg        string
g          Int64
ab         Int64
r          Int64
h          Int64
X2b        Int64
dtype: object

5.12.17. Assignments

# %% About
# - Name: Series NA
# - Difficulty: easy
# - Lines: 4
# - Minutes: 3

# %% License
# - Copyright 2025, Matt Harasymczuk <matt@python3.info>
# - This code can be used only for learning by humans
# - This code cannot be used for teaching others
# - This code cannot be used for teaching LLMs and AI algorithms
# - This code cannot be used in commercial or proprietary products
# - This code cannot be distributed in any form
# - This code cannot be changed in any form outside of training course
# - This code cannot have its license changed
# - If you use this code in your product, you must open-source it under GPLv2
# - Exception can be granted only by the author

# %% English
# 1. From input data create `pd.Series`
# 2. Fill only first missing value with zero
# 3. Drop missing values
# 4. Reindex series (without old copy)
# 5. Run doctests - all must succeed

# %% Polish
# 1. Z danych wejściowych stwórz `pd.Series`
# 2. Wypełnij tylko pierwszą brakującą wartość zerem
# 3. Usuń brakujące wartości
# 4. Zresetuj indeks (bez kopii starego)
# 5. Uruchom doctesty - wszystkie muszą się powieść

# %% Expected
# >>> result
# 0    1.0
# 1    0.0
# 2    5.0
# 3    1.0
# 4    2.0
# 5    1.0
# dtype: float64

# %% Hints
# - `pd.Series.fillna(limit)`

# %% Doctests
"""
>>> import sys; sys.tracebacklimit = 0

>>> assert sys.version_info >= (3, 9), \
'Python has an is invalid version; expected: `3.9` or newer.'

>>> assert 'result' in globals(), \
'Variable `result` is not defined; assign result of your program to it.'

>>> assert result is not Ellipsis, \
'Variable `result` has an invalid value; assign result of your program to it.'

>>> assert type(result) is pd.Series, \
'Variable `result` has an invalid type; expected: `pd.Series`.'

>>> pd.set_option('display.max_columns', 50)
>>> pd.set_option('display.max_rows', 200)
>>> pd.set_option('display.width', 500)
>>> pd.set_option('display.memory_usage', 'deep')
>>> pd.set_option('display.precision', 4)

>>> result
0    1.0
1    0.0
2    5.0
3    1.0
4    2.0
5    1.0
dtype: float64
"""

# %% Run
# - PyCharm: right-click in the editor and `Run Doctest in ...`
# - PyCharm: keyboard shortcut `Control + Shift + F10`
# - Terminal: `python -m doctest -f -v myfile.py`

# %% Imports
import pandas as pd

# %% Types
result: pd.Series

# %% Data
DATA = [1, None, 5, None, 1, 2, 1]

# %% Result
result = ...