Contributing

We welcome all contributions to decomon.

You can help by:

• fixing bugs (see issues with label “bug”),

• improving the documentation,

• adding and improving educational notebooks in tutorials/.

This is not exhaustive.

The project is hosted on https://github.com/airbus/decomon. Contributions to the repository are made by submitting pull requests.

This guide is organized as follows:

Setting up your development environment

Prerequisites

Python 3.9+ environment

The use of a virtual environment is recommended, and you will need to ensure that the environment use a Python version greater than 3.9. This can be achieved for instance either by using conda or by using pyenv (or pyenv-win on windows) and venv module.

The following examples show how to create a virtual environment with Python version 3.10.13 with the mentioned methods.

With conda (all platforms)

conda create -n do-env python=3.10.13
conda activate do-env

With pyenv + venv (Linux/MacOS)

pyenv install 3.10.13
pyenv shell 3.10.13
python -m venv do-venv
source do-venv/bin/activate

With pyenv-win + venv (Windows)

pyenv install 3.10.13
pyenv shell 3.10.13
python -m venv do-venv
do-venv\Scripts\activate

Keras 3+

Decomon relies on Keras 3 which allows the use of several backends: Tensorflow, PyTorch, and JAX.

To use it, you need to install at least one of the backends (refer to the documentation of each backend). You can find the version needed to be compatible with keras 3 in the Keras 3 compatibility matrix

You choose the backend used at runtime by setting the environment variable KERAS_BACKEND. Read more about it in Keras documentation.

Warning when using backend Tensorflow 2.15

When installing Tensorflow 2.15, the version 2.15 of Keras will be installed automatically. Be sure to get back keras 3 by fully uninstalling Keras then reinstalling Keras 3:

pip uninstall keras
pip install "keras>=3"

The problem will not occur starting from tensorflow 2.16.

Installing from source in developer mode

We use the option --editable (or -e) of pip install.

Note: you need a version of pip >=21.3 to be able to install in editable mode a project using a pyproject.toml file.

git clone https://github.com/airbus/decomon.git
cd decomon
pip install -U pip
pip install -e .

Building the docs locally

The documentation is using Sphinx to generate the html pages, and in particular the autogenerated API doc from in-code docstrings.

Install the documentation dependencies

See above to install decomon in developer mode, and then

pip install -r docs/requirements.txt

Define environment variables for notebook github links

In order to define appropriate github and colab links for notebooks, we need several environment variables:

• AUTODOC_NOTEBOOKS_REPO_URL: url of the content repository for the notebooks

• AUTODOC_NOTEBOOKS_BRANCH: branch containing the notebooks

For instance:

current_repo_url_withdotgit=$(git remote get-url origin)
export AUTODOC_NOTEBOOKS_REPO_URL=${current_repo_url_withdotgit/.git/}
export AUTODOC_NOTEBOOKS_BRANCH=$(git branch --show-current)

Build the docs

On Linux or Mac, or with git-bash on windows, make the documentation with

cd docs
# generate api doc source files
rm source/api/decomon*.rst
sphinx-apidoc -o source/api -f -T ../src/decomon
# generate available notebooks list
python generate_nb_index.py
# remove previous build
rm -rf build
# build doc html pages
sphinx-build -M html source build

The index of the built documentation is then available at build/html/index.html from the docs/ repository. You can for instance browse the documentation by running

python -m http.server -d build/html

and go to http://localhost:8000/. Doing this, rather than just opening index.html directly in you browser, make javascript work properly.

On windows with standard command prompt, the same lines work except that you have to use rm equivalent command to remove files and directories.

Notes

• sphinx-apidoc is used to generate the source files for api doc. It is necessary to launch it each time a new subpackage/module is added to the code.

• The line before is useful to avoid having source file corresponding to previous subpackage/module that have been removed.

• generate_nb_index.py update the list of notebooks in the generated file notebooks.md that is the source file of notebooks page. The command has to be launched when a new notebook is added.

• The last command actually build the html outputs and is required each time a doc page has a change (either beacause a source file has changed or the docstrings in the code have been modified).

• To update the doc faster when testing changes, launch only the last command without the previous rm [...].

Running unit tests

The unit tests are gathered in tests/ folder and can can be launched with tox.

Tox manage the creation of its own virtual environments with different versions of python, so do not use the virtual environment where you installed the library.

Install tox (for instance in a separate virtual environment):

python -m venv tox_venv
. tox_venv/bin/activate
pip install -U pip
pip install tox

Launch the test suite:

tox

If all python versions are not available or if you want to test only on a given version, specify the tox environment you want to test on. For instance:

tox l  # list all environments available
tox run -e py39-torch  # pytorch backend
tox run -e py39-tf  # tensorflow backend

You can also run a specific test, let say the function test_get_adv_box_1d from file “tests/test_wrapper.py” like that:

tox run -e py39-torch -- tests/test_wrapper.py::test_get_adv_box_1d

Under the hood, tox is actually calling pytest to perform the tests. You could also directly call it (after installing it in your development environment) with:

pytest tests -v

Running notebooks as tests

One can test programmatically that notebooks are not broken thanks to nbmake extension for pytest.

pip install nbmake
pytest --nbmake tutorials -v

Guidelines to follow when preparing a contribution

Coding style and code linting

To help maintaining the same coding style across the project, some code linters are used via pre-commit.

It is used by CI to run checks at each push, but can also be used locally.

Once installed, you can run it on all files with

pre-commit run --all-files

Beware that doing so, you are actually modifying the files.

You can also use it when committing:

• stage your changes: git add your_files,

• run pre-commit on the staged files: pre-commit run,

• check the changes made,

• accept them by adding modified files: git add -u,

• commit: git commit.

This can also be done automatically at each commit if you add pre-commit to git hooks with pre-commit install. Beware that when doing so,

• the changes will be refused if pre-commit actually modifies the files,

• you can then accept the modifications with git add -u,

• you can still force a commit that violates pre-commit checks with git commit -n or git commit --no-verify.

If you prefer run pre-commit manually, you can remove the hooks with pre-commit uninstall.

Annotations

The code is now fully annotated, and mypy is used to statically type checking it during CI workflow. So when you write new code, you should add the appropriate annotations to your functions and methods.

You can then test it directly with mypy

pip install mypy
mypy

This check is also part of the tox test suite, via the tox environment “type”:

tox -e type

See the section about unit tests for more information on tox.

Notebooks

We try to give some introductory examples via notebooks available in the corresponding tutorials/ directory.

Title and description extraction

The list of these notebooks is automatically inserted in the documentation with a title and a description. These are actually extracted from the first cell. To enable that, each notebook should

• start with a markdown cell,

• its first line being the title starting with one number sign (”# “),

• the remaining lines being used as the description.

• you can add lines that will be seen in notebook but not extracted in the description by wrapping them into <decomonlinks></decomonlinks> (see first cells of existing tutorials)

For instance:

# Great notebook title

A quick description of the main features of the notebook.
Can be on several lines.

<decomonlinks>
- <a href="https://airbus.github.io/decomon"> Documentation </a>
- <a href="https://github.com/airbus/decomon"> Github </a>
- <a href="https://airbus.github.io/decomon/main/tutorials.html "> Tutorials </a>
</decomonlinks>

Can include a nice thumbnail.
![Notebook_thumbnail](https://upload.wikimedia.org/wikipedia/commons/2/27/MnistExamples.png)

Colab specificities

In order to make it work on colab, you should also adding a cell that, after testing that the notebook is run on colab, install decomon and other necessary dependencies:

# On Colab: install the library
on_colab = "google.colab" in str(get_ipython())
if on_colab:
    import sys  # noqa: avoid having this import removed by pycln

    # install dev version for dev doc, or release version for release doc
    !{sys.executable} -m pip install -U pip
    !{sys.executable} -m pip install git+https://github.com/airbus/decomon@main#egg=decomon

When deployed in the release doc, the pip spec for decomon will be automatically updated to “decomon==x.y.z”, “x.y.z” being the appropriate version.

Adding unit tests

• Whenever adding some code, think to add some tests to the tests/ folder.

• Whenever fixing a bug, think to add a test that crashes before fixing the bug and does not afterwards.

Follow above instructions to run them with pytest.

Writing the documentation

In-code docstrings

The API is self-documented thanks to in-code docstrings and annotations. Whenever adding/fixing some code, you should add/update doctrings and annotations.

In order to generate properly the API doc, some guidelines should be followed:

• Docstrings should follow Google style docstrings (see this example), parsed thanks to napoleon extension.

• As we use type annotations in the code, types hints should not be added to the docstrings in order to avoid duplicates, and potentially inconsistencies.

• You should use annotations to explicit the types of public variables and of the inputs, outputs of public methods/functions.

• Docstrings should also follow PEP-257 which can be checked thanks to pydocstyle tool.

  pip install pydocstyle
  pydocstyle decomon
  

Doc pages

Natively, sphinx is meant to parse reStructuredText files.

To be able to directly reuse materials already written in markdown format, we use here the myST extension, that allows to write makdown files.

In markdown files, we can still write sphinx directives and roles as explained in myST documentation. For instance a sphinx table of contents tree can be inserted in a markodown file with a code block like:

```{toctree}
---
maxdepth: 2
caption: Contents
---
install
getting_started
notebooks
api/modules
contribute
```

Submitting pull requests

When you think you are ready to merge your modifications into the main repository, you will have to open a pull request (PR). We can summarize the process as follows:

• Fork the repository on github.

• Clone your fork on your computer.

• Make your changes and push them to your fork.

• Do the necessary checks (see below).

• Reorganize your commits (see below).

• Submit your pull request (see github documentation).

• See if all CI checks passed on your PR.

• Wait for a review.

• Take the comments and required changes into account.

Note that a PR needs at least one review by a core developer to be merged.

You may want to add a reference to the main repository to fetch from it and (re)base your changes on it:

git remote add upstream https://github.com/airbus/decomon

This post points out good practices to follow to submit great pull requests and review them efficiently.

Prior checks

Before submitting your pull request, think to

• run the unit tests,

• check the documentation locally if you modified it,

• check you respect the coding styles by running linters,

• check that mypy is happy with your annotations.

If you do not, you will still be able to see the status of your PR as CI will do these checks for you.

Reorganizing commits

On your way to implement your contribution, you will probably have lots of commits, some modifying other ones from the same PR, or only modifying the code style.

At the end of your work, consider reorganizing them by

• squashing them into one or only a few logical commits,

• having a separate commit to reformat previous existing code if necessary,

• rewritting commit messages so that it explains the changes made and why, the “how” part being explained by the code itself (see this post about what a commit message should and should not contain),

• rebasing on upstream repository main branch if it diverged too much by the time you finished.

You can use git rebase -i to do that, as explained in git documentation.