Writing unit tests is an important asset on code that needs to run in different platforms and a great way to make sure all is OK. Test units are run with nose. To run the test units on your package call:
$ ./bin/nosetests -v bob.example.library.test.test_reverse ... ok ---------------------------------------------------------------------- Ran 1 test in 0.253s OK
This example shows the results of the tests in the
bob.example.project package. Ideally, you should
write test units for each function of your package ...
You should put additional packages needed for testing (e.g.
Continuous integration (CI)¶
This is valid for people at Idiap (or external bob contributors with access to Idiap’s gitlab)
Before going into CI, you should make sure that your pacakge has a gitlab repository. If not, do the following in your package root folder:
$ git init $ git remote add origin firstname.lastname@example.org:bob/`basename $(pwd)` $ git add bob/ buildout.cfg COPYING doc/ MANIFEST.IN README.rst requirements.txt setup.py version.txt $ git commit -m '[Initial commit]' $ git push -u origin master
Copy the appropriate yml template for the CI builds:
# for pure python $ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-python-only.yml > .gitlab-ci.yml # for c/c++ extensions $ curl -k --silent https://gitlab.idiap.ch/bob/bob.admin/raw/master/templates/ci-for-cxx-extensions.yml | tr -d '\r' > .gitlab-ci.yml
Add the file to git:
$ git add .gitlab-ci.yml
The ci file should work out of the box. It is long-ish, but generic to any package in the system.
You also need to enable the following options - through gitlab - on your project:
- In the project “Settings” page, make sure builds are enabled
- If you have a private project, check the package settings and make sure that the “Deploy Keys” for our builders (all conda-* related servers) are enabled
- Visit the “Runners” section of your package settings and enable all conda runners, for linux and macosx variants
- Go into the “Variables” section of your package setup and add common variables corresponding to the usernames and passwords for uploading wheels and documentation tar balls to our (web DAV) server, as well as PyPI packages. You can copy required values from [the “Variables” section of bob.admin](https://gitlab.idiap.ch/bob/bob.admin/variables). N.B.: You must be logged into gitlab to access that page.
- Make sure to disable the service “Build e-mails” (those are very annoying)
- Setup the coverage regular expression under “CI/CD pipelines” to have the value ^TOTAL.*s+(d+%)$, which is adequate for figuring out the output of coverage report
Python package namespace¶
We like to make use of namespaces to define combined sets of functionality that go well together.
Python package namespaces are explained in details here together with implementation details.
For bob packages, we usually use the
bob namespace, using several sub-namespaces such as
bob.db or (like here)
The scripts you created should also somehow contain the namespace of the package. In our example,
the script is named
bob_example_project_version.py, reflecting the namespace
Distributing your work¶
To distribute a package, we recommend you use PyPI. Python Packaging User Guide contains details and good examples on how to achieve this. Moreover, you can provide a conda package for your PyPI package for easier installation. In order to create a conda package, you need to create a conda recipe for that package.
To ease up your life, we also provide a script to run all steps to publish
your package. Please read the following sections to understand the steps
./bin/bob_new_version.py script that will be explained at the
end of these section.
If you want to create a new recipe for a package, first:
- Learn about conda.
- Read the official conda build guide.
- Read conda-forge’s documentation.
- Read the readme of anaconda-recipes.
If you are at Idiap, every newly developed Bob package must have a conda recipe in bob.conda before a new version is tagged. Since our conda recipes pull your package source from PyPI and you only get to have a PyPI source after you tag, there is a little chicken and egg problem. Do not worry. To avoid this the first time you create a conda recipe for a new Bob package, make sure the builds are skipped for your recipe:
build: skip: true
Later when you tag a package for the first time, the CI process will
automatically update your recipe in bob.conda to the tagged version.
Then, you can remove the line that says
skip: true to actually start
building your package in bob.conda. This is only relevant for the
packages that are being tagged for the first time. In future, you just tag
and your package will be uploaded to PyPI first and then its recipe in
bob.conda will be updated through a merge request.
Folder structure in
There are several folder inside this repository and they are organized like this:
dependenciesA folder to keep the recipes for packages that are not part of Bob.
recipesA folder to keep recipes for Bob packages which contain C/C++ code.
skeletonA folder to keep recipes for Bob packages that contain Pyhon 2/3 compatible and platform-independent code. Usually this is the place that you want to place your recipe.
scriptsContains some useful scripts.
To add a new recipe, you can start by copying one of the recipes available
bob.conda. Below is some detailed information about our channel.
A few remarks¶
- Make sure you follow the pinning policies already available inside other packages recipes. You need to do a little searching here.
- Clearly mark your name (gitlab id) as maintainer of the recipe
- If the package has some C/C++ code, add following build and run
requirements: build: - toolchain 2.3.2 - gcc 4.8.5 # [linux] - libgcc 4.8.5 # [linux] run: - libgcc 4.8.5 # [linux]
In order to test the recipe, you’ll need to install
your conda root environment. Make sure you have the latest available
version on your system.
You must install conda on your local machine. You cannot use a central conda instllation (like the one we provide at Idiap) for this building purpose.
You should have only these two channels on the top of your channel list:
- https://www.idiap.ch/software/bob/conda - defaults
This can be achieved with the following commands:
$ conda config --add channels defaults $ conda config --add channels http://www.idiap.ch/software/bob/conda
Once that is done, you can try to build your new recipe with:
$ CONDA_NPY=112 conda build <package>
If that works, upload your changes on this package on a branch and create a merge request. Wait for the tests to pass on the MR and make sure everything completes well, by inspecting the log files. If all is good, assign the branch for merge to one of the package maintainers.
You should only use the packages from the
defaults channel and our
channel. The packages from our channel are not co-installable with the
packages from conda-forge. If you need to use a package from
conda-forge, you need to first port that package to bob.conda first.
Version numbering scheme¶
We recommend you follow Bob‘s version numbering scheme using a 3-tier string:
The value of
M is a number starting at 1.
This number is changed in case of a major release that brings new APIs and concepts to the table.
The value of
m is a number starting at 0.
Every time a new API is available (but no conceptual modifications are done to the platform)
that number is increased.
Finally, the value of p represents the patch level, starting at 0.
Every time we need to post a new version of Bob that does not bring incompatible API modifications, that number is increased.
For example, version 1.0.0 is the first release of Bob.
Version 1.0.1 would be the first patch release.
The numbering scheme for your package and Bob‘s may look the same, but should be totally independent of each other.
Bob may be on version 3.4.2 while your package, still compatible with that release could be on 1.4.5.
You should state on your
setup.py file which version of Bob your package is compatible with, using the standard notation defined for setuptools installation requirements for packages.
You may use version number extenders for alpha, beta, and candidate releases with the above scheme, by appending
cN to the version number.
The value of
N should be an integer starting at zero.
Python’s setuptools package will correctly classifier package versions following this simple scheme.
For more information on package numbers, consult Python’s PEP 386.
Here are lists of valid Python version numbers following this scheme:
0.0.1 0.1.0a35 1.2.3b44 2.4.99c32
Release methodology for packages¶
Here is a set of steps we recommend you follow when releasing a new version of your package:
First decide on the new version number your package will get. If you are making a minor, API preserving, modification on an existing stable package (already published on PyPI), just increment the last digit on the version. Bigger changes may require that you signal them to users by changing the first digits of the package. Alpha, beta or candidate releases don’t need to have their main components of the version changed, just bump-up the last digit. For example
In case you are making an API modification to your package, you should think if you would like to branch your repository at this position. You don’t have to care about this detail with new packages, naturally.
If required, branching will allow you to still make modifications (patches) on the old version of the code and develop on the
masterbranch for the new release, in parallel. It is important to branch when you break functionality on existing code - for example to reach compatibility with an upcoming version of Bob. After a few major releases, your repository should look somewhat like this:
----> time initial commit o---------------o---------o-----o-----------------------> master | | | | | | v2.0.0 | | +---x----------> 2.0 | | | | v1.1.0 v1.1.1 | +-x-------x------> 1.1 | | v1.0.0 v1.0.1a0 +---x-------x-------> 1.0
o‘s mark the points in which you decided to branch your project. The
x‘s mark places where you decided to release a new version of your satellite package on PyPI. The
-‘s mark commits on your repository. Time flies from left to right.
In this fictitious representation, the
masterbranch continue under development, but one can see older branches don’t receive much attention anymore.
Here is an example for creating a branch at gitlab (many of our packages are hosted there). Let’s create a branch called
$ git branch 1.1 $ git checkout 1.1 $ git push origin 1.1
When you decide to release something publicly, we recommend you tag the version of the package on your repository, so you have a marker to what code you actually published on PyPI. Tagging on gitlab would go like this:
$ git tag v1.1.0 $ git push && git push --tags
Notice use prefix tag names with
Finally, after branching and tagging, it is time for you to publish your new package on PyPI. When the package is ready and you have tested it, just do the following:
$ ./bin/python setup.py register #if you modified your setup.py or README.rst $ ./bin/python setup.py sdist --formats zip upload
You can also check the .zip file that will be uploaded to PyPI before actually uploading it. Just call:
$ ./bin/python setup.py sdist --formats zip
and check what was put into the
To be able to upload a package to PyPI you have to register at the web page using a user name and password.
Announce the update on the relevant channels.
Change the version of your package¶
In total, 5 steps need to be performed, in the right order. These steps are:
- Change the value in version.txt
- Change links in README.rst so documentation and build tags point to the right version instead of master
- Commit, tag and push
- Change the value in version.txt to the next possible version tag (beta)
All these steps are combined in the
This script needs to be run from within the root directory of your package.
By default, it will make an elaborate guess on the version that you want to upload.
$ ./bin/bob_new_version.py --help
to see a list of options.
Detailed information of what the script is doing, you can get when using the
--dry-run option (a step that you always should consider before actually executing the script):
$ ./bin/bob_new_version.py -vv --dry-run