# Markov Decision Process (MDP) Toolbox: util module¶

The util module provides functions to check that an MDP is validly described. There are also functions for working with MDPs while they are being solved.

## Available functions¶

check()
Check that an MDP is properly defined
checkSquareStochastic()
Check that a matrix is square and stochastic
getSpan()
Calculate the span of an array
isNonNegative()
Check if a matrix has only non-negative elements
isSquare()
Check if a matrix is square
isStochastic()
Check if a matrix is row stochastic
mdptoolbox.util.check(P, R)[source]

Check if P and R define a valid Markov Decision Process (MDP).

Let S = number of states, A = number of actions.

Parameters: P (array) – The transition matrices. It can be a three dimensional array with a shape of (A, S, S). It can also be a one dimensional arraye with a shape of (A, ), where each element contains a matrix of shape (S, S) which can possibly be sparse. R (array) – The reward matrix. It can be a three dimensional array with a shape of (S, A, A). It can also be a one dimensional array with a shape of (A, ), where each element contains matrix with a shape of (S, S) which can possibly be sparse. It can also be an array with a shape of (S, A) which can possibly be sparse.

Notes

Raises an error if P and R do not define a MDP.

Examples

>>> import mdptoolbox, mdptoolbox.example
>>> P_valid, R_valid = mdptoolbox.example.rand(100, 5)
>>> mdptoolbox.util.check(P_valid, R_valid) # Nothing should happen
>>>
>>> import numpy as np
>>> P_invalid = np.random.rand(5, 100, 100)
>>> mdptoolbox.util.check(P_invalid, R_valid) # Raises an exception
Traceback (most recent call last):
...
StochasticError:...

mdptoolbox.util.checkSquareStochastic(matrix)[source]

Check if matrix is a square and row-stochastic.

To pass the check the following conditions must be met:

• The matrix should be square, so the number of columns equals the number of rows.
• The matrix should be row-stochastic so the rows should sum to one.
• Each value in the matrix must be positive.

If the check does not pass then a mdptoolbox.util.Invalid

Parameters: matrix (numpy.ndarray, scipy.sparse.*_matrix) – A two dimensional array (matrix).

Notes

Returns None if no error has been detected, else it raises an error.

mdptoolbox.util.getSpan(array)[source]

Return the span of array

span(array) = max array(s) - min array(s)

mdptoolbox.util.isNonNegative(matrix)[source]

Check that matrix is row non-negative.

Returns: is_stochastic – True if matrix is non-negative, False otherwise. bool
mdptoolbox.util.isSquare(matrix)[source]

Check that matrix is square.

Returns: is_square – True if matrix is square, False otherwise. bool
mdptoolbox.util.isStochastic(matrix)[source]

Check that matrix is row stochastic.

Returns: is_stochastic – True if matrix is row stochastic, False otherwise. bool