ProMod3 Setup¶

Git Branches¶

Basically we have two, sometimes three major branches. master, develop and in front of a new release a dedicated release branch. For bugs, hotfix branches of a rather short life are used.

master is the stable branch, corresponding to a released version. It is solely fed by a release or hotfix branch.

Release branches, usually labelled release-<VERSION>, are branched of develop to fix features and thoroughly test them before a new major release. Once everything looks trustworthy, such a branch is merged into master and since there should be a few bug fixes in, master is merged into develop. Bugs are fixed in dedicated hotfix branches, which should only exist for the fix and testing itself. Those are forged from release branches or master. If created for master, they are also merged back into develop.

The develop branch exists to introduce new features up to the level of whole projects extending ProMod3 and see that they work seamlessly together with the rest of the system. There do exist a couple of rather strict rules for what goes into this branch:

• Your code must have been (briefly) reviewed by others
• There have to be unit tests
• It needs to pass make check including doctest & linkcheck
• Your project needs documentation
• It must not break the ability of out-of-source builds

The reason to be a bit restrictive on branches which end up in actual releases, should be mostly obvious: ProMod3 is used by productive services as a third party toolbox. There it is not an item of active development and people probably have no insight in its internals. So messing up a new release creates a lot of extra work for a lot of people. First for the developer of a service to find out that ProMod3 has turned malicious, then for the maintainer of this package to figure out that its your contribution messing things up and in the end for you, fixing the problems.

The place where you may get messy is your own Git branch within the ProMod3 repository. This is basically where you should develop your project. Once you created something that could go into a release, tidy things up according to the rules from above and merge it into develop. From there it will automatically find its way into the next release.

To set up your own branch, start from a current develop branch:

$ git checkout develop         # switch to branch develop
$ git pull --rebase            # update branch develop
$ git checkout -b <BRANCHNAME> # create branch <BRANCHNAME> and switch to it

Over time, develop may recognise some changes, e.g. new features, which you want to make use of in your project. Keeping your branch up to date is a three step process. Git does not allow updates on top of changed code, so either changes have to be committed, or if in the middle of implementing something, stored away temporarily. Making commits is straight forward:

$ git commit -m '<DESCRIPTION>' # commit changes including a comment

Hiding your changes away from Git just for updating files is a bit more involved. Everything is easily stored on an internal stack and needs to be fetched from there, once the branch was updated. One major problem in the past was a possible loss of code by those operations. If the update changes a file you have changed, too, and stashed away, this may end up in a non-resolvable merge conflict and your changes are lost. Usually the log tells you, which files were recently modified. Moving all current changes to the stack is achieved by:

$ git stash save

To revive them, use:

$ git stash pop

After cleaning up your branch, switch to develop, update it and switch back:

$ git checkout develop
$ git pull --rebase
$ git checkout <BRANCHNAME>

Now for actually updating your branch, there are two different ways: merging and rebasing. A rebase may only be done, if you never pushed your branch to the origin of the repository (otherwise you will mess up history, in the worst case develop may be unusable once you merge):

$ git rebase develop

For branches which are available to others, do a proper merge:

$ git merge develop

This may require some manual conflict solving and will end up in a merge commit.

Git Hooks¶

Git hooks are scripts invoked by Git in connection to certain commands. ProMod3 currently provides one for commit. It is installed by

$ cp extras/pre_commit/pre-commit .git/hooks/

Its task is applying coding standards and doing a bunch of other checks on the files involved in a commit. Everything around the script is hosted in extras/pre_commit/. The checks can be manually executed with

$ python .git/hooks/pre-commit

If you ever have to skip the hook,

$ git commit --no-verify

does the trick. But checks are always run on the complete file containing changes, not only on the lines changed. This means if you opt out of an issue, it will reappear next time that very file changes.

For checking Python code, the pre-commit hook employs Pylint, to make sure we stay close to PEP 8. If you feel the need to make changes to the Pylint call, please make sure you fully understand what the complaints are. Sometimes PEP 8 sounds overly restrictive but it may help with performance and compatibility with Python 3. For ProMod3 it is also important that the code looks similar throughout the various modules. So do not disable a check because it just seems inconvenient or you do not understand why Pylint is croaking at what looks like ‘working’ code. But then there are also cases where Pylint is not smart enough to cope with valid PEP 8 code. For changes with valid cause, the configuration flushed into Pylint may be found at extras/pre_commit/pm3_csc/filecheck/pylintrc and extras/pre_commit/pm3_csc/filecheck/pylint-unittest-rc. The latter one is invoked on unit test code, where we may go a little bit less restrictive.

Directory Structure¶

The directory structure of the ProMod3 repository is supposed to ‘keep everything together that belongs together’. That is, code, documentation and extra data should be gathered on a per-module basis immediately in the repository root. The directory structure of your module should look like this:

promod3.git/                            Project folder
      your_module/                      Module directory
            CMakeLists.txt              CMake configuration
            data/                       Extra data (if needed)
                  CMakeLists.txt        CMake configuration
                  ...
            doc/                        Documentation
                  CMakeLists.txt        CMake configuration
                  your_module.rst       Overview/frame of your module
                  ...
            pymod/                      Python code
                  CMakeLists.txt        CMake configuration
                  __init__.py           Init file needed for import
                  submodule1.py         Code
                  ...
            src/                        C/ C++ code
                  CMakeLists.txt        CMake configuration
                  source1.cc            C++ code
                  source2.hh            Header
                  ...
            tests/                      Unit tests
                  CMakeLists.txt        CMake configuration
                  data/                 Test data (if needed)
                        ...
                  test_your_module.py   Unit tests for your_module
                  test_submodule1.py    Unit tests for submodule1
                  ...

Additionally to the module directories there are a few extra folders:

• actions: Scripts callable as pm <ACTION_NAME>. See here for details.
• cmake_support: Helper functions for CMake. See here for details.
• doc: High-level documentation, test scripts (doc/tests) and a copy of the generated html documentation (doc/html). The latter must be kept up-to-date at least on the master branch. See here for details.
• extras: Extra data and information that doesn’t fit anywhere else (e.g. Git hooks or scripts to recreate the binary files).
• scripts: Input for scripts that end up in stage/bin

CMake¶

The attentive reader may have noticed all the CMakeLists.txt files in the directory structure. Those are needed to configure the build system, e.g. tell it which files have to be considered packaging, compiling, etc.. Also Python modules are declared there as well as which files belong to the documentation. CMake is a rather complex topic (unfortunately all usable build systems seem to be) so we skip a detailed view, here, and just advice you to go by example. There is a tiny bit of documentation on our additions to CMake here. If you really need to make changes to the build system, other than adding new files and modules, you have to dive into CMake documentation all by yourself and on your own responsibility. You have been warned.

The stage Directory¶

Once you hit make in your build directory, a directory stage in this path will be populated. It just resembles a directory structure as of a usual Unix file system filled with the build products of ProMod3. The stage directory tree can already be utilised. You may import Python modules from there, use the binaries from stage/bin, etc..