Local development of packages

Very often, developers are confronted with the need to clone package repositories locally and develop installation/build and runtime code. It is recommended to create isolated environments using conda to develop new projects. Tools implemented in bob.devtools help automate this process for Bob packages. In the following we talk about how to checkout and build one or several packages from their git source and build proper isolated environments to develop them. Then we will describe how to create a new bob package from scratch and develop existing bob packages along side it.

TL;DR

Suppose you want to checkout the package bob.io.base from source and start developing it locally. We will use the tools implemented in bob.devtools to create a proper developing environment to build and develop bob.io.base. We assume you have bob.devtools installed on a conda environment named bdt (Refer to Installation for detailed information.)

  • Checkout the source of the package from git:

$ bdt dev checkout --use-ssh bob.io.base

  • Create a proper conda environment:

$ cd bob.io.base
$ bdt dev create -vv dev
$ conda activate dev

Note

bdt might try to install the cuda version of deep learning packages. If you don’t have cuda drivers installed and face errors such as nothing provides __cuda, you might need to run: export CONDA_OVERRIDE_CUDA=11.6 where instead of 11.6 you should put the latest version of cuda.

  • Build the package using pip:

$ bdt dev install -n dev . # calls pip with correct arguments
$ python

for example:

>>> import bob.io.base
>>> bob.io.base # should print from '.../bob.io.base/bob/blitz/...'
<module 'bob.io.base' from '.../bob.io.base/bob/blitz/__init__.py'>
>>> print(bob.io.base.get_config())
bob.io.base: 2.0.20b0 [api=0x0202] (.../bob.io.base)
* C/C++ dependencies:
  - Blitz++: 0.10
  - Boost: 1.67.0
  - Compiler: {'name': 'gcc', 'version': '7.3.0'}
  - NumPy: {'abi': '0x01000009', 'api': '0x0000000D'}
  - Python: 3.6.9
* Python dependencies:
  - bob.extension: 3.2.1b0 (.../envs/dev/lib/python3.6/site-packages)
  - click: 7.0 (.../envs/dev/lib/python3.6/site-packages)
  - click-plugins: 1.1.1 (.../envs/dev/lib/python3.6/site-packages)
  - numpy: 1.16.4 (.../envs/dev/lib/python3.6/site-packages)
  - setuptools: 41.0.1 (.../envs/dev/lib/python3.6/site-packages)

  • You can optionally run the test suit to check your installation:

$ pytest -sv ...

  • Some packages may come with a pre-commit config file (.pre-commit-config.yaml). Make sure to install pre-commit if the config file exists:

$ # check if the configuration file exists:
$ ls .pre-commit-config.yaml
$ pip install pre-commit
$ pre-commit install

Local development of existing packages

To develop existing Bob packages you need to checkout their source code and install them in your environment.

Checking out package sources

Bob packages are developed through gitlab. Various packages exist in Bob’s gitlab instance. In the following we assume you want to install and build locally the bob.io.base package. In order to checkout a package, just use git:

$ bdt dev checkout --use-ssh bob.io.base

Create an isolated conda environment

Now that we have the package checked out we need an isolated environment with proper configuration to develop the package. bob.devtools provides a tool that automatically creates such environment. Before proceeding, you need to make sure that you already have a conda environment with bob.devtools installed in it (Refer to Installation for more information). let’s assume that you have a conda environment named bdt with installed bob.devtools.

$ cd bob.io.base
$ bdt dev create -vv dev
$ conda activate dev

Now you have an isolated conda environment named dev with proper channels set. For more information about conda channels refer to conda channel documentation.

The bdt dev create command assumes a directory named conda, exists on the current directory. The conda directory contains a file named meta.yaml, that is the recipe required to create a development environment for the package you want to develop.

Note

When developing and testing new features, one often wishes to work against the very latest, bleeding edge, available set of changes on dependent packages.

bdt dev create command adds Bob beta channels to highest priority which creates an environment with the latest dependencies instead of the latest stable versions of each package.

If you want to create your environment using stable channels, you can use this command instead:

$ bdt dev create --stable -vv dev

To see which channels you are using run:

$ conda config --get channels

Note

We recommend creating a new conda environment for every project or task that you work on. This way you can have several isolated development environments which can be very different form each other.

Installing the package

The last step is to install the package:

$ cd bob.io.base #if that is not the case
$ conda activate dev #if that is not the case
$ bdt dev install -n dev .

To run the test suite:

$ pytest -sv ...

or build the documentation:

$ sphinx-build -aEn doc sphinx  # make sure it finishes without warnings.
$ xdg-open sphinx/index.html  # view the docs.

You can see what is installed in your environment:

$ conda list

And you can install new packages using mamba:

$ mamba install <package>

Note

If you want to debug a package regarding an issues showing on the ci you can use bob.devtools. Make sure the conda environment containing bob.devtools is activated (typically, base).

$ cd <package>
$ bdt local build

One important advantage of using conda is that it does not require administrator privileges for setting up any of the above. Furthermore, you will be able to create distributable environments for each project you have. This is a great way to release code for laboratory exercises or for a particular publication that depends on Bob.

Developing multiple existing packages simultaneously

It so happens that you want to develop several packages against each other for your project. Let’s assume you want to develop bob.io.base and bob.extension simultaneously. bob.io.base is dependent on bob.extension. First we checkout package bob.io.base and build an isolated conda environment as explained in the previous section. Then checkout and install bob.extension as following:

$ bdt dev checkout --use-ssh --subfolder src bob.extension
$ bdt dev install -n dev src/bob.extension

Local development of a new package

In this section we explain how to create a new bob package from scratch and start developing it. Once again bob.devtools is here to help you. You need to activate your conda environment with bob.devtools installed in it.

$ bdt dev new -vv bob/bob.project.awesome author_name author_email

This command will create a new bob package named “bob.project.awesome” that includes the correct anatomy of a package. For more information about the functionality of each file check Anatomy of a new package.

Now you have all the necessary tools available and you can make a development environment using bdt dev create command, run bdt dev install in it and start developing your package.

$ cd bob.project.awesome
$ bdt dev create --stable -vv awesome-project  #here we used the stable channels to make the conda environment.
$ conda activate awesome-project
$ bdt dev install -n awesome-project .

Developing existing bob packages along with your new package

Let’s assume you need to develop two packages, bob.extension and bob.io.base, as part of developing your new bob.project.awesome package.

You need to checkout and install these packages:

$ bdt dev checkout --use-ssh --subfolder src bob.extension bob.io.base
$ bdt dev install -n awesome-project src/bob.extension src/bob.io.base  # the order of installing dependencies matters!

When you build your new package, it is customary to checkout the dependent packages (in this example bob.extension and bob.io.base) in the src folder in the root of your project.

As usual, first create an isolated conda environment using bdt dev create command. Some of bob packages need dependencies that might not be installed on your environment. You can find these dependencies by checking conda/meta.yaml of each package. Install the required packages and then run bdt dev install. For our example you need to do the following:

$ mamba install gcc_linux-64 gxx_linux-64 libblitz
$ bdt dev install -n awesome-project src/bob.extension src/bob.io.base  # the order of installing dependencies matters!