test_actions - Testing Actions¶

The Test Script¶

The script to supervise your action needs to be placed in <SOURCE>/actions/tests and follow the naming convention test_action_<NAME>.py, where <NAME> is the name for your action. So here we create a file test_action_do_awesome.py (recognise the underscore between do and awesome instead of a hyphen, that’s PEP 8).

$ touch <SOURCE>/actions/tests/test_action_do_awesome.py
$

As a starter, we disable bytecode compilation in the script:

1
2
3
4
5

import sys

# this is needed so there will be no test_actions.pyc created in the source
# directory
sys.dont_write_bytecode = True

CMake Integration¶

As always, when introducing new material to ProMod3, it has to be announced to the CMake build system. For action unit tests, fire up <SOURCE>/actions/tests/CMakeLists.txt in your favourite text editor and add your new script:

1
2
3
4
5
6
7

set(ACTION_UNIT_TESTS
  test_action_help.py
  test_action_do_awesome.py
  test_actions.py # leave this as last item so it will be executed first!
)

promod3_unittest(MODULE actions SOURCES "${ACTION_UNIT_TESTS}" TARGET actions)

The important thing is to leave test_actions.py as last item in the list. This script contains the tests around the test_actions.ActionTestCase class, which is the foundation of the tests for your action. If this class is broken, we are lost. Putting it as the last element in the list, CMake will execute this script first, before any other action test script is run.

Creating a Test Subclass¶

test_actions.ActionTestCase is sort of a template class for your tests. By spawning off from this you inherit a bunch of useful methods for your testing. To make it work, the childclass needs to be set up properly. But first, test_actions.py has to be loaded as a module:

6

import test_actions

To showcase, the test cases, we explain how one would (and does) test the help action of pm. First, we create the childclass for the action. Go for <NAME>ActionTests as a naming scheme:

 7
 8
 9
10

class HelpActionTests(test_actions.ActionTestCase):
    def __init__(self, *args, **kwargs):
        test_actions.ActionTestCase.__init__(self, *args, **kwargs)
        self.pm_action = 'help'

Pay attention that in your own class, you must set pm_action to make everything work. Also __init__() needs certain parameters, as everything is derived from the unittest.TestCase class.

Must Have Tests¶

What needs testing without exclusion are the exit codes of actions. Those states will be placed in the userlevel documentation. This topic is already covered in test_actions.ActionTestCase by RunExitStatusTest(). As an example, testing for $?=0 could work like this:

11
12

    def testExit0(self):
        self.RunExitStatusTest(0, list())

That will call the action stored in pm_action with the provided list of parameters and check that 0 is returned on the command line.

In a more general way, you need to test that your action is working as intended. Do not forget some negative testing, with the idea in mind what happens if a user throws dirty input data in.

Making the Script Executable¶

In ProMod3, unit tests are run via OST‘s ost.testutils and Python‘s unittest.TestCase. Those are called when the test module is executed as a script:

13
14
15

if __name__ == "__main__":
    from ost import testutils
    testutils.RunTests()

These three lines should be the same for all unit tests.

Running the Test Script¶

Unit tests are executed via make check and so are ProMod3 action tests. But for every test script, we also provide a private make target, ending with _run. To solely run the tests for the awesome action, hit

$ make test_action_do_awesome.py_run

Output Of test_actions.ActionTestCase¶

When running the test script you will notice that its not really talkative. Basically you do not see output to stdout/ stderr of your action, while the test script fires it a couple of times. That is by design. When running the full unit test suite, usually nobody wants to see the output of everything tested and working. The interesting bits are where we fail. But for developing a new application you certainly need all the output you can get. For this, some functions in test_actions.ActionTestCase have a parameter verbose. That triggers specific functions to flush captured output onto the command line. The idea is to turn it on for development, but once done, disable it to keep output of unit tests low.

To get the test for exit code 0 talking to you, just do

11
12

    def testExit0(self):
        self.RunExitStatusTest(0, list(), verbose=True)

and

11
12

    def testExit0(self):
        self.RunExitStatusTest(0, list())

keeps it silent (verbose is set to False by default). If enabled, output will be separated into stdout and stderr:

$ make test_action_do_awesome.py_run
<Lots of output from the build process>
running checks test_action_do_awesome.py
stdout of '<BUILD>/stage/bin/pm do-awesome'
------
<Output to stdout>
------
stderr of '<BUILD>/stage/bin/pm do-awesome'
------
<Output to stderr>
------
<More output from unit test runner>