pytest

Pytest

Original author(s)	Krekel et al.
Stable release	7.1.1
Repository	github.com/pytest-dev/pytest
Written in	Python
Platform	macOS, Windows, POSIX
Type	Software testing framework.
License	MIT License
Website	pytest.org

Pytest is a software testing framework based on the Python programming language. It can be used to write various types of software tests, including unit tests, integration tests, end-to-end tests, and functional tests. Its features include parametrized testing, fixtures, and assert re-writing.

Pytest has been described as a scalable framework with less boilerplate code than other testing frameworks. Pytest fixtures provide the contexts for tests by passing in parameter names in test cases; its parametrization eliminates duplicate code for testing multiple sets of input and output; and its re-writed assert statements provides detailed output for causes of failures.

Many technology projects have switched to pytest from other testing frameworks, including those of Mozilla and Dropbox. According to developer security platform Snyk, Pytest is classified as a key ecosystem project in Python due to its high degree of popularity.

History

Pytest had began as part of PyPy, an alternative implementation of Python to CPython. It later separated from PyPy into its own project and was named pytest.

Since early 2003 when PyPy emerged, it held a heavy emphasis on testing. PyPy had unit tests for newly written code, regressions tests for bugs, and integration tests using CPython's test suite. In mid 2004, a testing framework called utest emerged, and contributors to PyPy began converting existing test cases to utest. Meanwhile, for EuroPython 2004, a complementary standard library for testing named std was created, which layed out the principles, such as assert rewriting, of what later became pytest.

In late 2004, the std project was renamed to py and std.utest became py.test, afterwhich the py library was separated from PyPy. In November 2010, pytest 2.0.0 was released as a package separate from py, being still called py.test until August 2016 with the release of pytest 3.0.0 when the recommended command line entry point became pytest.

Notable features

Parametrized testing

It is a common pattern in software testing to send values through test functions and check for correct output. In many cases, in order to thoroughly test functionalities, one needs to test multiple sets of input/outputs, and writing such cases separately would cause duplicate code as most of the actions would remain the same, only differing in input/output values. Pytest's parametrized testing feature eliminates such duplicate code by combining different iterations into one test case, then runs these iterations and displays each test's result separately.

This feature is preferred by some developers over unittest's parametrizing method where multiple test results are grouped into one, in which case if one test fails and the rest passes, unittest only displays a single failing result for the entire group, which contrasts with pytest's individual test reporting.

Parametrized tests in pytest are marked by the @pytest.mark.parametrize(argnames, argvalues) decorator, where the first parameter, argnames, is a string of comma-separated names, and argvalues is a list of values to pass into argnames. When there are multiple names in argnames, argvalues would be a list of tuples where values in each tuple corresponds to the names in argnames by index. The names in argnames are then passed into the test function makred by the decorator as parameters. When pytest runs such decorated tests, each pair of argnames and argvalues would constitute a separate run with its own test output and unique identifier. The identifier can then be used to run individual data pairs.

Assert rewriting

When writing software tests, the assert statement is a primary means for communicating test failure, where expected values are compared to actual values. While Python's builtin assert keyword would only raise AssertionError with no details in cases of failure, pytest rewrites Python's assert keyword and provides detailed output for the causes of failures, such as what expressions in the assert statement evaluate to. Its concision and readability drive many to use pytest over other testing frameworks. For instance, a comparison can be made with unittest's assert statements:

pytest	unittest
assert x	assertTrue(x)
assert x == y	assertEqual(x, y)
assert x >= y	assertLessEqual(x, y)

unittest adheres to a more verbose syntax because it is written with inspiration from JUnit in Java, as are most unit testing libraries. However, being considered by some Python developers as a more pythonic approach, pytest takes advantage of Python's ability for user defined libraries to achieve what in other languages would require compiler modifications to display traceback that provides information on the exact reasons of failed assertions, hence this assert rewriting feature.

Pytest fixtures

Pytest fixtures provide the context for tests. They can be used to set a system in to known state and to pass dataset into test functions. Fixtures practically constitute the arrange phase in the anatomy of a test (AAA, short for arrange, act, assert). Pytest fixtures can run before test cases as setup or after test cases for clean up, though they are different from unittest and nose's setup and teardowns. Functions declared as pytest fixtures are marked by the @pytest.fixture() decorator, whose names can then be passed into test functions as parameters. When pytest finds the fixtures names in test functions parameters, it first searches in the same module for such fixtures, and if not found, it searches for such fixtures in the conftest.py file.

For example:

import pytest

@pytest.fixture()
def dataset():
    """Return some data to test functions"""
    return {'data1': 1, 'data2': 2}

def test_dataset(dataset):
    """test and confirm fixture value"""
    assert dataset == {'data1': 1, 'data2': 2}

In the above example, pytest fixture dataset returns a dictionary, which is then passed into test function test_dataset for assertion. In addition to fixture detection within the same file as test cases, pytest fixtures can also be placed in the conftest.py file in the tests directory. There can be multiple conftest.py files, each placed within a tests directory for fixtures to be detected for each sebset of tests.

Fixture scopes

In pytest, fixure scopes let the user define when a fixure should be called. There are four fixture scopes: function scope, class scope, module scope, and session scope. Function-scoped fixtures are default for all pytest fixtures, which are called every time a function having the fixture as a parameter runs. The goal of specifying a broader fixture scope is to eliminate repeated fixture calls, which could slow down test execution. Class-scoped fixtures are called once per test class, regardless of the number of times they are called, and the same logic goes for all other scopes. When changing fixture scope, one need only add the scope parameter to fixture decorators, for example, @pytest.fixture(scope="class").

Test filtering

Another notable feature of pytest is its ability to filter through tests, where only desired tests are selected to run, or behave in a certain way as desired by the developer.

With the "k" option (e.g. pytest -k some_name), pytest would only runs tests whose names include some_name. The opposite is true, where one can run pytest -k "not some_name", and pytest will run all tests whose names do not include some_name.

Pytest's markers can, in addition to altering test behaviour, also filter tests. Pytest's markers are Python decorators starting with the @pytest.mark.<markername> syntax placed on top of test functions. With different arbitrarily named markers, running pytest -m <markername> on the command line will only run those tests decorated with such markers. All available markers can be listed by the pytest --markers along with their descriptions; custom markers can also be defined by users and registered in pytest.ini, in which case pytest --markers will also list those custom markers along with builtin markers.

Popularity

Pytest was developed as part of an effort by third-party packages to address Python's built-in module unittest's shortcomings, and has been classified by developer security platform Snyk as one of the key ecosystem projects in Python due to its popularity. Some well-known cases of projects switching to pytest from unittest and nose (another testing package) include those of Mozilla and Dropbox.

Installation and running tests

One needs to run pip install pytest from the command line. After entering pytest on the command line, pytest then recursively detects and runs tests that have either test_ or _test as function names in files within the working directory. Files containing tests are named according to the following format: test_*.py or *_test.py; Test classes are named such that they either begin or end with Test. This is the default naming convention pytest uses for detection, though it can be customized in pytest's configuration file pytest.ini.

For instance, one can define a test in a file named test_pass.py as follows:

def test_pass():
    assert 1 == 1

Then, enter pytest test_pass.py on the command line within the project directory where the test function resides. The output is as follows:

===================== test session starts ======================
collected 1 items
test_one.py .
=================== 1 passed in 0.01 seconds ===================

Pytest uses a . beside the file under test to signify a passing test, and F for a failing test, along with various other test output behaviours which are all customizable.

See also

• JUnit, well-known software testing framework based on Java
• Doctest, well-known testing framework in Python for docstrings
• List of unit testing frameworks

External links

• https://pypi.python.org/pypi/pytest
• https://docs.pytest.org