`test_actions` - Testing Actions¶

This module is not part of the ProMod3 binary distribution. That is the productive bit running to produce models. It is only part of the source distribution intended to help developing ProMod3. Basically it supports you creating new actions along immediate tests, which will be stored as unit tests and stay available to monitor later changes.

Note

A couple of different paths will be mentioned in the following. To make things easier to tell apart, a prefix <SOURCE> refers to the code repository, <BUILD> to the build directory tree.

Inside the development environment, the module is only available to unit tests in the <SOURCE>/actions/tests directory. There is one special thing about using it in your tests for an action, emerging from the way make runs unit tests as set up via CMake. Python modules are imported from the source directory, here this is <SOURCE>/actions/tests, while the tests run inside <BUILD>/tests, here this is <BUILD>/tests/actions. When Python imports a module, its usually compiled into bytecode. This new file would clutter up the source repository, it would always show up as untracked file on git status. To prevent this, tell Python to stop producing bytecode right at the beginning of your test-script:

import sys

# this is needed so there will be no test_actions.pyc created in the source
# directory
sys.dont_write_bytecode = True

Line 5 does the trick. This needs to be set by you in every action unit test file since Python only recognises it before the module is imported. Otherwise a module could disable bytecoding for all other modules loaded.

Testing actions, basically those are commands run in a shell, is very similar across various actions. Additionally, there are some things that should be tested for all actions like exit codes. That is why this module exists.

When developing an action, you will try it in the shell during the process. You have to check that its doing what you intend, that it delivers the right output, that it just behaves right on various kinds of input. This module supports you by providing functionality to run scripts out of Python. The goal is to not trigger test runs manually in a shell but have a script that does it for you. From there, you do not need to remember all the calls you punched into the command line a year ago, when you come back to change something, add new functionality, etc..

Creating an Action Unit Test Script¶

In the next couple of paragraphs, we will walk through setting up a new unit test script for an imaginary action. We will continuously extend the file started above, so keep an eye on line numbers. Lets just assume your action is called do-awesome for the rest of this section.

The Test Script¶

The script to supervise your action needs to be placed in <SOURCE>/actions/tests and follow the naming convention test_action_<NAME>.py, where <NAME> is the name for your action. So here we create a file test_action_do_awesome.py (recognise the underscore between do and awesome instead of a hyphen, that’s PEP 8).

$ touch <SOURCE>/actions/tests/test_action_do_awesome.py
$

As a starter, we disable bytecode compilation in the script:

import sys

# this is needed so there will be no test_actions.pyc created in the source
# directory
sys.dont_write_bytecode = True

CMake Integration¶

As always, when introducing new material to ProMod3, it has to be announced to the CMake build system. For action unit tests, fire up <SOURCE>/actions/tests/CMakeLists.txt in your favourite text editor and add your new script:

set(ACTION_UNIT_TESTS
  test_action_help.py
  test_action_do_awesome.py
  test_actions.py # leave this as last item so it will be executed first!
)

promod3_unittest(MODULE actions SOURCES "${ACTION_UNIT_TESTS}" TARGET actions)

The important thing is to leave test_actions.py as last item in the list. This script contains the tests around the test_actions.ActionTestCase class, which is the foundation of the tests for your action. If this class is broken, we are lost. Putting it as the last element in the list, CMake will execute this script first, before any other action test script is run.

Creating a Test Subclass¶

test_actions.ActionTestCase is sort of a template class for your tests. By spawning off from this you inherit a bunch of useful methods for your testing. To make it work, the childclass needs to be set up properly. But first, test_actions.py has to be loaded as a module:

6import test_actions

To showcase, the test cases, we explain how one would (and does) test the help action of pm. First, we create the childclass for the action. Go for <NAME>ActionTests as a naming scheme:

class HelpActionTests(test_actions.ActionTestCase):
    def __init__(self, *args, **kwargs):
        test_actions.ActionTestCase.__init__(self, *args, **kwargs)
        self.pm_action = 'help'

Pay attention that in your own class, you must set pm_action to make everything work. Also __init__() needs certain parameters, as everything is derived from the unittest.TestCase class.

Must Have Tests¶

What needs testing without exclusion are the exit codes of actions. Those states will be placed in the userlevel documentation. This topic is already covered in test_actions.ActionTestCase by RunExitStatusTest(). As an example, testing for $?=0 could work like this:

    def testExit0(self):
        self.RunExitStatusTest(0, list())

That will call the action stored in pm_action with the provided list of parameters and check that 0 is returned on the command line.

In a more general way, you need to test that your action is working as intended. Do not forget some negative testing, with the idea in mind what happens if a user throws dirty input data in.

Making the Script Executable¶

In ProMod3, unit tests are run via OST’s ost.testutils and Python’s unittest.TestCase. Those are called when the test module is executed as a script:

if __name__ == "__main__":
    from ost import testutils
    testutils.RunTests()

These three lines should be the same for all unit tests.

Running the Test Script¶

Unit tests are executed via make check and so are ProMod3 action tests. But for every test script, we also provide a private make target, ending with _run. To solely run the tests for the awesome action, hit

$ make test_action_do_awesome.py_run

Output Of `test_actions.ActionTestCase`¶

When running the test script you will notice that its not really talkative. Basically you do not see output to stdout/ stderr of your action, while the test script fires it a couple of times. That is by design. When running the full unit test suite, usually nobody wants to see the output of everything tested and working. The interesting bits are where we fail. But for developing a new application you certainly need all the output you can get. For this, some functions in test_actions.ActionTestCase have a parameter verbose. That triggers specific functions to flush captured output onto the command line. The idea is to turn it on for development, but once done, disable it to keep output of unit tests low.

To get the test for exit code 0 talking to you, just do

    def testExit0(self):
        self.RunExitStatusTest(0, list(), verbose=True)

and

    def testExit0(self):
        self.RunExitStatusTest(0, list())

keeps it silent (verbose is set to False by default). If enabled, output will be separated into stdout and stderr:

$ make test_action_do_awesome.py_run
<Lots of output from the build process>
running checks test_action_do_awesome.py
stdout of '<BUILD>/stage/bin/pm do-awesome'
------
<Output to stdout>
------
stderr of '<BUILD>/stage/bin/pm do-awesome'
------
<Output to stderr>
------
<More output from unit test runner>

Unit Test Actions API¶

class test_actions.ActionTestCase(*args, **kwargs)¶

Class to help developing actions. Comes with a lot of convenience wrappers around what should be tested and serves as a recorder for test calls… just for in two years when you come back to rewrite the whole action…

While inheriting this class, pm_action needs to be defined. Otherwise the whole idea does not work.

pm_bin¶

This is the path of the pm binary. Automatically set by calling __init__() inside the initialisation of your class.

Type:: str

pm_action¶

The action to be tested. Needs to be set by your initialisation routine, after calling __init__() from here. Skip the “pm-” in front of the action name.

Type:: str

RunAction(arguments, verbose=False)¶

Call an action, return the exit status ($? shell variable). May be set to verbose to print the actions terminal output. The action to be executed needs to be stored in pm_action first.

If in verbose mode, output to stdout of the action will be printed first followed by stderr.

Parameters:

arguments (list) – A list of arguments for the call.
verbose (bool) – If True, report output of the action.

Returns:

The exit code of the action (int).

RunExitStatusTest(exit_code, arguments, verbose=False)¶

Run the action with given arguments and check the exit code.

Parameters:

exit_code (int) – The expected return code, $? in a shell.
arguments (list) – A list of arguments for the call.
verbose (bool) – If True, report output of the action.

testPMExists()¶: This is an internal test, executed when the source code of the test class is run as unit test. Verifies that pm_bin is an existing file (also complains if a directory is found instead).

`test_actions` - Testing Actions¶

Creating an Action Unit Test Script¶

The Test Script¶

CMake Integration¶

Creating a Test Subclass¶

Must Have Tests¶

Making the Script Executable¶

Running the Test Script¶

Output Of `test_actions.ActionTestCase`¶

Unit Test Actions API¶

Search

Contents

test_actions - Testing Actions¶

Creating an Action Unit Test Script¶

The Test Script¶

CMake Integration¶

Creating a Test Subclass¶

Must Have Tests¶

Making the Script Executable¶

Running the Test Script¶

Output Of test_actions.ActionTestCase¶

Unit Test Actions API¶

Search

Contents

`test_actions` - Testing Actions¶

Output Of `test_actions.ActionTestCase`¶