Development recommendations

GIT

iota2 development uses GIT as source control tool.

GIT-Flow

There are many ways of using GIT. In iota2 git-flow has been chosen to maintain the project (without release branches). You can find more about it here.

To roughly sum-up, for the development of a new feature, the developer must :

from develop, create a feature branch

commit often

rebase (see below) new features comming from the develop branch into the feature branch

when the feature is ready, create a merge request into the develop branch.

Commits must be prefixed with one of the following label :

Label	Purpose
BUG FIX	Fix for runtime crash or incorrect result
INSTALL	Installation
DOC	Documentation update
ENH	Enhancement of an algorithm
NEW FEATURE	New functionality
PERF	Performance improvement
STYLE	No logic impact (indentation, comments)
WIP	Work In Progress not ready for merge

Branch naming conventions

branch purpose	naming convention	Example
develop a new feature	feature-*	feature-Classsifications_fusion
respond to an issue	issue-*	issue-#53_Improve_Classifications
fix a bug	fix-*	fix-#27_Bug

the #27 in fix-#27_Bug should refer to an issue reported on iota2 GIT repository.

Rebase your branch with develop

To have a clear git tree a good practice is to use rebase instead of merge. This way all the work on the branch will appear after the last develop commit.

It is important to understand that, for a simple rebase experience, you should avoid using standard merge into the feature branch. Once you have used merge, a rebase operation may fail.

The standard procedure is therefore:

git checkout develop
git branch feature-branch
git checkout feature-branch
do your work and commit often
git rebase develop

Here is a procedure to rebase a branch using emacs and magit. Develop and custom branch are supposed to be clean and ready to merge (pulled, all changes commited…). Emacs is opened on the custom branch to rebase.

    # run magit
M-x magit
# enable rebase
    r
    # and choose elsewhere
    e
    # choose the branch to rebase with
    develop
    # then rebase start

If there is no conflict the rebase is done automatically and your recent commits should look like:

e30ee6eb custom_branch WIP: start to write rebase baseline
    ...
74ef6ea5 develop origin/develop Merge branch 'solve_test_error' into 'develop'

If the custom branch is already pushed to the online repo, you must use force-push as you rewrite the history of the branch.

In case of conflict

If a file is modified both in develop and in the custom branch, that probably leads to a conflict. In this case, the conflicting file(s) appear in the Unstaged files section in magit. By pressing e on a file, an ediff cession opens. Using n to move forward all conflicting lines, you can choose a variant by typing a or b. You can also manually solve the conflict by writing in the ediff buffer. Then quit ediff with q and save files as asked.

Warning

Never commit a file until the end of rebase

Now you can restart the rebase operation with r and r to continue.

Once ended, your commit history will be clean, and all develop commits will appear aftre the custom branch commits.

Note

In order to avoid conflicts and to lighten the merge operations, it is important to make many small commits. It is then possible, once the rebase with develop is finished, to clean up the history of the branch by merging the commits and submit the merge request for integration in develop.

FRAMAGIT

iota2 is hosted on FramaGit . Anybody can create an account for free and submit merge requests. To monitor the project, issues are mainly used. Developments are often started by a reported issue describing a bug, a new interesting feature or a research idea.

Issues

If a developer wants to contribute to iota2, here is the recommended workflow:

If an issue about the new feature does not exist, create a dedicated one.

Assign the issue to herself.

When the contribution is done, close the issue.

Theses simple rules will avoid duplicate work between developers.

Note

The Board view is very useful to see which features are in development, need to be developed, or are in the backlog.

CODE

Warning

Developers must use the pre-commit hook. It can be setup by the script enable_hooks.sh, which just makes a symlink into the .git/hooks directory. The pre-commit hook will format the code with yapf and run pylint and the pre-push hook will launch unit tests.

mypy

To improve the readability and maintainability of the code, we have chosen to use mypy with the following options : “–disallow-untyped-defs –follow-imports=skip –ignore-missing-imports”. Before it can be integrated into iota2 via the merge request mechanism, the code must must comply as far as possible with mypy recommendations. If a recommendation cannot be applied, a discussion can be held on the code concerned.

doc-string

Every usefull functions must constains a doc-string.

It’s format must be in NumPy style, but using python typing in function signature.

To this end, Returns blocks must not be used. If information about return type is provided, prefer use Notes blocks.

def func(arg1: int, arg2: str):
    """Summary line.

    Extended description of function.

    Parameters
    ----------
    arg1 :
        Description of arg1
    arg2 :
        Description of arg2

    Notes
    -----
        Description of return value:
                    This function always return True, which is a boolean

    """
    return True

Citing code

Source code

To reference source code in documentation you must use sphinx API link.

The following rst code:

# First declare the current module with
.. currentmodule:: iota2.Common.customNumpyFeatures

Then create the link to a class with :class:`data_container`
or a function :meth:`compute_custom_features`

Produce the following documentation:

Then create the link to a class with data_container or a function compute_custom_features()

Example code

Creating and citing simple examples could be useful to explain algorithmic choices or function use. For this kind of citing, put the source file in iota2/doc/source/examples.

then you can cite the code:

Cite an entire class

.. literalinclude:: examples/example.py
       :pyobject: example_class

an entire function

.. literalinclude:: examples/example.py
       :pyobject: example_method

or text between two lines

.. literalinclude:: examples/example.py
       :lines: 30-44

Produce the following documentation

Cite an entire class

an entire function

def example_method(arg1: int, arg2: int) -> int:
    """
    This method sum the two args

    Parameters
    ----------

    arg1:
        a integer value
    arg2:
        a integer value

    Notes
    -----
        The output value is the sum of the two inputs
    """
    return arg1 + arg2

or text between two lines

    This method sum the two args

    Parameters
    ----------

    arg1:
        a integer value
    arg2:
        a integer value

    Notes
    -----
        The output value is the sum of the two inputs
    """
    return arg1 + arg2

TESTS

Pytest

iota2 is mainly developed in Python, and the pytest library has been chosen to implement our test framework. It is highly recommended adding tests to each new feature. Currently, unit tests are placed in the /iota2/tests directory.

Baselines

A set of baseline data is already present in /iota2/data/references. Please use them as references to your tests. If new baseline must be created, add it in the directory previously quoted after discussion and consensus among the development team.

Warning

Baselines must be as small as possible.

Code coverage

For the merge request to be accepted, the code coverage of the tests should be at least as good as the current code coverage of the branch develop. For this, a code coverage analysis tool was created. First, run pytest --cov iota2 --cov-report json:code_coverage.json on your code. As we sue sub-processes, you need to follow these instructions: https://coverage.readthedocs.io/en/latest/subprocess.html. Otherwise, the measured coverage will be inaccurate and much lower than expected. The current coverage is provided here: iota/data/current_code_coverage.json.

After running tests with the coverage option on your code, you can use the provided tool (iota2/common/tools/code_coverage_analysis.py) to compare the difference of code coverage of the tests between what is already done on develop and the new branch. Two outputs are possible: a table printed in the terminal and an HTML file containing a table. This table contains the name of .py files, the current code coverage on develop, the code coverage of the new branch and the difference between them. For example:

File	Difference	Develop coverage	Incoming merge coverage
file1.py	-5%	80%	75%
file2.py	6%	70%	76%
file3.py	40%	new file	50%
file4.py	95%	new file	95%

Four cases are possible:

The file already exists in develop:
- The coverage is worse than what’s already done in develop: file1 (not acceptable)
- The coverage is the same or better: file is “acceptable”: file2 (acceptable)
The file doesn’t exists yet in develop:
- The coverage doesn’t meet the requirements (a threshold can be given before the analysis, 75% by default): file3 (not acceptable)
- The coverage meets the requirements: file4 (acceptable)

The user can also provide an “ignore threshold”: some small changes in a file (adding/removing one line …) may induce small changes in code coverage, which may represent a real downgrade of the coverage (0.2% by default).

The inputs of the tool are:

develop_coverage: Path to the develop’s code coverage json file
merge_coverage: Path to the branch’s code coverage json file
new_files_threshold: Threshold for new files’ code coverage
ignore_threshold: Threshold for ignoring small changes in files’ coverage
html_file_path: Path to the output HTML file (if not set, the file is not produced)