test_actions - Testing Actions¶
This module is not part of the ProMod3 binary distribution. That is the productive bit running to produce models. It is only part of the source distribution intended to help developing ProMod3. Basically it supports you creating new actions along immediate tests, which will be stored as unit tests and stay available to monitor later changes.
Note
A couple of different paths will be mentioned in the following. To make
things easier to tell apart, a prefix <SOURCE> refers to the code
repository, <BUILD> to the build directory tree.
Inside the development environment, the module is only available to unit tests
in the <SOURCE>/actions/tests directory. There is one special thing
about using it in your tests for an action, emerging from the way make runs
unit tests as set up via CMake. Python modules are imported from the source
directory, here this is <SOURCE>/actions/tests, while the tests run
inside <BUILD>/tests, here this is <BUILD>/tests/actions. When
Python imports a module, its usually compiled into bytecode. This new file
would clutter up the source repository, it would always show up as untracked
file on git status. To prevent this, tell Python to stop producing
bytecode right at the beginning of your test-script:
1 import sys
2
3 # this is needed so there will be no test_actions.pyc created in the source
4 # directory
5 sys.dont_write_bytecode = True
Line 5 does the trick. This needs to be set by you in every action unit test file since Python only recognises it before the module is imported. Otherwise a module could disable bytecoding for all other modules loaded.
Testing actions, basically those are commands run in a shell, is very similar across various actions. Additionally, there are some things that should be tested for all actions like exit codes. That is why this module exists.
When developing an action, you will try it in the shell during the process. You have to check that its doing what you intend, that it delivers the right output, that it just behaves right on various kinds of input. This module supports you by providing functionality to run scripts out of Python. The goal is to not trigger test runs manually in a shell but have a script that does it for you. From there, you do not need to remember all the calls you punched into the command line a year ago, when you come back to change something, add new functionality, etc..
Creating an Action Unit Test Script¶
In the next couple of paragraphs, we will walk through setting up a new unit
test script for an imaginary action. We will continuously extend the file
started above, so keep an eye on line numbers. Lets just assume your action is
called do-awesome for the rest of this section.
The Test Script¶
The script to supervise your action needs to be placed in
<SOURCE>/actions/tests and follow the naming convention
test_action_<NAME>.py, where <NAME> is the name for your
action. So here we create a file test_action_do_awesome.py (recognise
the underscore between do and awesome instead of a hyphen, that’s
PEP 8).
$ touch <SOURCE>/actions/tests/test_action_do_awesome.py
$
As a starter, we disable bytecode compilation in the script:
1 import sys
2
3 # this is needed so there will be no test_actions.pyc created in the source
4 # directory
5 sys.dont_write_bytecode = True
CMake Integration¶
As always, when introducing new material to ProMod3, it has to be announced
to the CMake build system. For action unit tests, fire up
<SOURCE>/actions/tests/CMakeLists.txt in your favourite text editor and
add your new script:
1 set(ACTION_UNIT_TESTS
2 test_action_help.py
3 test_action_do_awesome.py
4 test_actions.py # leave this as last item so it will be executed first!
5 )
6
7 promod3_unittest(MODULE actions SOURCES "${ACTION_UNIT_TESTS}" TARGET actions)
The important thing is to leave test_actions.py as last item in the
list. This script contains the tests around the
test_actions.ActionTestCase class, which is the foundation of the
tests for your action. If this class is broken, we are lost. Putting it as the
last element in the list, CMake will execute this script first, before any
other action test script is run.
Creating a Test Subclass¶
test_actions.ActionTestCase is sort of a template class for your
tests. By spawning off from this you inherit a bunch of useful methods for your
testing. To make it work, the childclass needs to be set up properly. But
first, test_actions.py has to be loaded as a module:
6 import test_actions
To showcase, the test cases, we explain how one would (and does) test the
help action of pm.
First, we create the childclass for the action.
Go for <NAME>ActionTests as a naming scheme:
7 class HelpActionTests(test_actions.ActionTestCase):
8 def __init__(self, *args, **kwargs):
9 test_actions.ActionTestCase.__init__(self, *args, **kwargs)
10 self.pm_action = 'help'
Pay attention that in your own class, you must set pm_action to make
everything work. Also __init__() needs certain parameters, as everything
is derived from the unittest.TestCase class.
Must Have Tests¶
What needs testing without exclusion are the exit codes of actions. Those
states will be placed in the userlevel documentation. This topic is already
covered in test_actions.ActionTestCase by RunExitStatusTest().
As an example, testing for $?=0 could work like this:
11 def testExit0(self):
12 self.RunExitStatusTest(0, list())
That will call the action stored in pm_action with the provided list of
parameters and check that 0 is returned on the command line.
In a more general way, you need to test that your action is working as intended. Do not forget some negative testing, with the idea in mind what happens if a user throws dirty input data in.
Making the Script Executable¶
In ProMod3, unit tests are run via OST’s ost.testutils and Python’s
unittest.TestCase. Those are called when the test module is executed
as a script:
13 if __name__ == "__main__":
14 from ost import testutils
15 testutils.RunTests()
These three lines should be the same for all unit tests.
Running the Test Script¶
Unit tests are executed via make check and so are ProMod3 action tests.
But for every test script, we also provide a private make target, ending
with _run. To solely run the tests for the awesome action, hit
$ make test_action_do_awesome.py_run
Output Of test_actions.ActionTestCase¶
When running the test script you will notice that its not really talkative.
Basically you do not see output to stdout/ stderr of your
action, while the test script fires it a couple of times. That is by design.
When running the full unit test suite, usually nobody wants to see the output
of everything tested and working. The interesting bits are where we fail.
But for developing a new application you certainly need all the output you can
get. For this, some functions in test_actions.ActionTestCase have a
parameter verbose. That triggers specific functions to flush captured
output onto the command line. The idea is to turn it on for development, but
once done, disable it to keep output of unit tests low.
To get the test for exit code 0 talking to you, just do
11 def testExit0(self):
12 self.RunExitStatusTest(0, list(), verbose=True)
and
11 def testExit0(self):
12 self.RunExitStatusTest(0, list())
keeps it silent (verbose is set to False by default). If enabled,
output will be separated into stdout and stderr:
$ make test_action_do_awesome.py_run
<Lots of output from the build process>
running checks test_action_do_awesome.py
stdout of '<BUILD>/stage/bin/pm do-awesome'
------
<Output to stdout>
------
stderr of '<BUILD>/stage/bin/pm do-awesome'
------
<Output to stderr>
------
<More output from unit test runner>