4. Developer Environment

This section is a repository of information that might be potentially useful to developers. Note that information regarding intermediate files/caches that are created automatically, which might cause issues during development and testing, is split across sections.

4.1. Eigen

Eigen is a header-only C++ template library for linear algebra. Being header-only means there is no compiled library to link against, it is used purely by including its headers directly into source files.

4.1.1. Installation

The OpenBT Meson build system satisfies the Eigen dependence automatically. First, Meson uses different techniques to search for an existing Eigen installation. If found, that installation is used for the build. If not found, Meson falls back to the subprojects/eigen.wrap file, which instructs it to download a pinned Eigen version automatically from Eigen’s repository and use it internally for that build. As a result, Eigen is always available to the build regardless of whether it is preinstalled on the system.

Developers using macOS who need to test the build system or who prefer to have a system-wide installation can install Eigen via Homebrew:

$ brew install eigen

4.2. Meson Build

The OpenBT Python package uses the Meson build system together with its ninja backend to compile the C++ command line tools during installation. Please refer to the relevant installation instructions to determine if manual installation of these tools is required for a particular task.

Please refer to the documentation in tools/build_openbt_clt.sh for information about using that tool, for an example of how to configure and use the Meson build system, and for potential build difficulties (e.g., due to intermediate and cached files).

4.2.1. Build Process with Python

The Meson build is not invoked directly by developers working on or testing the Python package. The build is triggered automatically when the OpenBT Python package is installed via

$ cd /path/to/OpenBT/openbt_pypkg
$ python -m pip install .

or installed in editable mode via

$ python -m pip install -e .

It is also invoked automatically to build wheels. We generally refer to this automated process as a “package build.”

Internally, setup.py defines a custom build_clt command that wipes and rebuilds the Meson build directory openbt_pypkg/cpp/builddir from scratch on every package build, forcing Meson to re-detect the compiler, MPI, and Eigen installations rather than reusing stale detection results. Developers who need the exact Meson invocation can inspect build_clt in setup.py directly.

A successful package build creates the following files and directories:

• openbt_pypkg/cpp/builddir/ — Meson’s working build directory. Build output including object files are stored here. Since this directory is wiped and recreated on every package build, it can be deleted safely at any time.

• openbt_pypkg/src/openbt/_version.py — Written by setuptools_scm from the current git tag, not by Meson.

Note that while openbt_pypkg/cpp officially contains the package’s C++ source code and Meson build system, its contents simply alias the actual code and build system defined at the root of the repository. Therefore, for example, all intermediate and cached issues associated with the base folder also exist for package builds.

Editable Python package installations install build products, such as the command line tools, directly in a developer’s clone rather than inside the Python execution environment (e.g., within the site-packages folder of a virtual environment). These cached files, which can occasionally cause issues, are

• openbt_pypkg/src/openbt/{bin,include,lib}/ — The install destination populated by meson install. This is the most problematic caching layer: meson install overlays new files onto these directories but never removes stale ones. If a binary is renamed, a tool is removed from the build, or Eigen headers change, the old files persist silently. Consider deleting these if the build produces unexpected behaviour. Note that, of these contents, only a subset of the command line tools in bin is included in a package build. See meson.build for the current list of built tools. See setup.py to determine which of these are included in the Python package.

• openbt_pypkg/src/openbt/include/eigen3/ — Eigen headers installed under the package prefix as a side effect of Eigen’s own Meson install step, regardless of whether Eigen came from the system or the bundled subprojects/eigen.wrap. These files are unimportant once the command line tools are built and are not included in package distributions.

• openbt_pypkg/src/openbt/lib/pkgconfig/eigen3.pc — A pkg-config file for the installed Eigen, with its prefix pointing into src/openbt/, that is installed as a side effect. This file is unimportant and is not included in package distributions.

4.3. Tox

Developers are free to setup whatever environment that they may need to facilitate their work with the Python package. However, the package includes a tox setup, which developers can also use to automatically setup and manage dedicated virtual environments for different predefined development tasks. Some tasks are more broadly useful at the level of the whole repository since they can, for instance, build the User Guides for all OpenBT tools.

4.3.1. Development with tox

The following is a rough guide to help install tox as a command line tool in a dedicated, minimal virtual environment. tox is made available with no need to manually activate its virtual environment.

Note

Developers that would like to use tox should, at the very least, learn enough about it that they understand the difference between running tox and tox -r. Some potential issues are highlighted below.

$ cd $HOME/local/venv
$ deactivate
$ /path/to/desired/python --version
$ /path/to/desired/python -m venv $HOME/local/venv/.toxbase
$ ./.toxbase/bin/python -m pip list
$ ./.toxbase/bin/python -m pip install --upgrade pip setuptools
$ ./.toxbase/bin/python -m pip install tox
$ ./.toxbase/bin/python -m pip list
$ ./.toxbase/bin/tox --version

To avoid having to activate .toxbase every time we would like to work with tox, we setup tox in PATH. Note that developers can use this single tox installation for multiple projects. Please replace .bash_profile with the appropriate shell configuration file and tailor the following to your needs.

$ mkdir -p $HOME/local/bin
$ ln -s $HOME/local/venv/.toxbase/bin/tox $HOME/local/bin/tox
$ vi $HOME/.bash_profile (add $HOME/local/bin to PATH)
$ . $HOME/.bash_profile
$ which tox
$ tox --version

No work will be carried out by default with the calls tox and tox -r.

Run the following from the directory hierarchy that contains the OpenBT tox configuration file /path/to/OpenBT/openbt_pypkg/tox.ini to see the full list of available environments and what each one does:

$ tox list -v

Two or more tasks can be executed in a single invocation, (e.g., tox -r -e report,coverage). Users needing pdf should note that tox does not install make or a LaTeX distribution; those must be installed separately.

The tox tool caches all of its virtual environments in openbt_pypkg/.tox/. Running tox -r -e <task> forces a clean environment rebuild including installation of (potentially more modern) dependencies and a full package build from scratch. Happily, developers can activate and work directly in tox’s cached virtual environments.

4.3.2. Direct use of tox virtual environments

Many of the tox tasks will build the OpenBT binary automatically each time they are run, which can significantly slow development work. In such cases, developer productivity can benefit from creating a clean virtual environment for their task using tox -r -e <task> and subsequently loading and working in that virtual environment directly.

Developers can inspect tox.ini to see what commands are run by their task and adapt these for their work.

The following example shows how to run only a single test case using the coverage virtual environment setup by tox.

$ cd /path/to/OpenBT/openbt_pypkg
$ tox -r -e coverage
$ . ./.tox/coverage/bin/activate
$ which python
$ python --version
$ python -m pip list
$ python -m pytest --pyargs openbt.tests.test_mixing

Note that using the coverage virtual environment directly can be particularly useful since the package is installed in editable mode and therefore facilitates interactive development and testing of the Python code.

The html environment can be activated directly in the same way to rebuild documentation iteratively without paying the cost of a full package rebuild each time:

$ cd /path/to/OpenBT/openbt_pypkg
$ tox -r -e html
$ . ./.tox/html/bin/activate
$ which sphinx-build
$ sphinx-build -W -E -b html ../docs ../docs/build_html

4.3.3. Caching

As noted above, some tox tasks build the OpenBT package in editable mode. They, therefore, can suffer from the potential caching issues mentioned above for direct editable installations of the package.