Python API¶
Logging utilities. |
|
Tools to help CI-based builds and artifact deployment. |
|
Constants used for building and more. |
|
Utilities to needed to release packages. |
|
Utilities for retrieving, parsing and auto-generating changelogs. |
|
Bootstraps a new miniconda installation and prepares it for development. |
|
Tools for self-building and other utilities. |
|
Mirroring functionality for conda channels |
|
Deployment utilities for conda packages and documentation via webDAV. |
|
Utilities for calculating package dependencies and drawing graphs |
Detailed Information¶
Logging utilities.
- bob.devtools.log.COLORMAP = {'critical': {'attrs': ['bold'], 'color': 'red'}, 'debug': {}, 'error': {'color': 'red'}, 'exception': {'attrs': ['bold'], 'color': 'red'}, 'info': {'attrs': ['bold']}, 'warn': {'attrs': ['bold'], 'color': 'yellow'}, 'warning': {'attrs': ['bold'], 'color': 'yellow'}}¶
Default color map for homogenized color display
- bob.devtools.log.echo_normal(text)[source]¶
Color preset for normal text output for
click.echo()
- bob.devtools.log.echo_info(text)[source]¶
Color preset for normal text output for
click.echo()
- bob.devtools.log.echo_warning(text)[source]¶
Color preset for normal warning output for
click.echo()
- bob.devtools.log.setup(logger_name, format='%(levelname)s:%(name)s@%(asctime)s: %(message)s')[source]¶
This function returns a logger object that is set up to perform logging using Bob loggers.
- Parameters
logger_name (str) – The name of the module to generate logs for
format (
str
, optional) – The format of the logs, seelogging.LogRecord
for more details. By default, the log contains the logger name, the log time, the log level and the massage.
- Returns
logger – The logger configured for logging. The same logger can be retrieved using the
logging.getLogger()
function.- Return type
- bob.devtools.log.set_verbosity_level(logger, level)[source]¶
Sets the log level for the given logger.
- Parameters
logger (
logging.Logger
or str) – The logger to generate logs for, or the name of the module to generate logs for.level (int) – Possible log levels are: 0: Error; 1: Warning; 2: Info; 3: Debug.
- Raises
ValueError – If the level is not in range(0, 4).
- bob.devtools.log.verbosity_option(**kwargs)[source]¶
Adds a -v/–verbose option to a click command.
- Parameters
**kwargs – All kwargs are passed to click.option.
- Returns
A decorator to be used for adding this option.
- Return type
callable
Tools to help CI-based builds and artifact deployment.
- bob.devtools.ci.is_master(refname, tag, repodir)[source]¶
Tells if we’re on the master branch via ref_name or tag.
This function checks if the name of the branch being built is “master”. If a tag is set, then it checks if the tag is on the master branch. If so, then also returns
True
, otherwise,False
.- Parameters
refname – The value of the environment variable
CI_COMMIT_REF_NAME
tag – The value of the environment variable
CI_COMMIT_TAG
- (may beNone
)
Returns: a boolean, indicating we’re building the master branch or that the tag being built was issued on the master branch.
- bob.devtools.ci.is_private(baseurl, package)[source]¶
Tells if a given package (with a namespace) is public or private
This function checks if a fully qualified package in the format
<namespace>/<name>
is publicly accessible. It does this by trying to accessinfo/refs?service=git-upload-pack
from the package in question.This method does not rely on the fact the user has access to Gitlab.
Warning
This method only works for fully qualified package names (i.e., containing at least one forward-slash
/
).- Parameters
baseurl – The base URL for the gitlab service to consult
package – Fully qualified (i.e., with a namespace) package name. For example:
bob/bob.extension
.
Returns: a boolean, indicating if the package is private (
True
) or not (False
).
- bob.devtools.ci.is_stable(package, refname, tag, repodir)[source]¶
Determines if the package being published is stable.
This is done by checking if a tag was set for the package. If that is the case, we still cross-check the tag is on the “master” branch. If everything checks out, we return
True
. Else,False
.- Parameters
package – Package name in the format “group/name”
refname – The current value of the environment
CI_COMMIT_REF_NAME
tag – The current value of the enviroment
CI_COMMIT_TAG
(may beNone
)repodir – The directory that contains the clone of the git repository
Returns: a boolean, indicating if the current build is for a stable release
- bob.devtools.ci.read_packages(filename)[source]¶
Return a python list of tuples (repository, branch), given a file containing one package (and branch) per line.
Comments are excluded
- bob.devtools.ci.select_build_file(basename, paths, branch)[source]¶
Selects the file to use for a build.
This method will return the name of the most adequate build-accessory file (conda_build_config.yaml, recipe_append.yaml) for a given build, in this order of priority:
The first file found is returned
We first search for a specific file if
branch
is setIf that file does not exist, returns the unbranded filename if that exists in one of the paths
If no candidates exists, returns
None
The candidate filename is built using
os.path.splitext(os.path.basename(basename))[0]
.- Parameters
basename – Name of the file to use for the search
paths (list) – A list of paths leading to the location of the variants file to use. Priority is given to paths that come first
branch (str) – Optional key to be set when searching for the variants file to use. This is typically the git-branch name of the current branch of the repo being built.
- Returns
A string containing the full, resolved path of the file to use. Returns
None
, if no candidate is found- Return type
- bob.devtools.ci.select_conda_build_config(paths, branch)[source]¶
Selects the default conda_build_config.yaml.
See
select_build_file()
for implementation details. If no build config file is found byselect_build_file()
, then returns the defaultconda_build_config.yaml
shipped with this package.
- bob.devtools.ci.select_conda_recipe_append(paths, branch)[source]¶
Selects the default recipe_append.yaml.
See
select_build_file()
for implementation details. If no recipe append file is found byselect_build_file()
, then returns the defaultrecipe_append.yaml
shipped with this package.
- bob.devtools.ci.select_user_condarc(paths, branch)[source]¶
Selects the user condarc file to read (if any)
See
select_build_file()
for implementation details. If no recipe condarc is found byselect_build_file()
, then returnsNone
.
- bob.devtools.ci.cleanup(dry_run, username, password, includes)[source]¶
Cleans-up WebDAV resources. Executes if
dry_run==False
only.- Parameters
dry_run (bool) – If set, then does not execute any action, just print what it would do instead.
username (str) – The user to use for interacting with the WebDAV service
password (str) – Password the the above user
includes (re.SRE_Pattern) – A regular expression that matches the names of packages that should be considered for clean-up. For example: for Bob and BATL packages, you may use
^(bob|batl|gridtk).*
For BEAT packages you may use^beat.*
- bob.devtools.ci.temporary_cwd(path)[source]¶
Temporarily changes the working directory to a given path
A context manager that temporarily changes the path to the given directory. The working directory changes the working directory back to the starting directory when it exits.
- Parameters
path – The directory to temporarily change to
Constants used for building and more.
- bob.devtools.constants.BASE_CONDARC = "add_pip_as_python_dependency: false #!final\nalways_yes: true #!final\nanaconda_upload: false #!final\nchannel_priority: strict #!final\nconda_build: #!final\n pkg_format: '2'\ndefault_channels: #!final\n - https://repo.anaconda.com/pkgs/main\nchannels:\n - conda-forge\nchannel_alias: https://conda.anaconda.org #!final\nquiet: true #!final\nremote_connect_timeout_secs: 120.0 #!final\nremote_max_retries: 50 #!final\nremote_read_timeout_secs: 180.0 #!final\nshow_channel_urls: true #!final\nssl_verify: false #!final\n"¶
Default setup for conda builds
- bob.devtools.constants.CONDA_BUILD_CONFIG = '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.9/site-packages/bob/devtools/data/conda_build_config.yaml'¶
Configuration variants we like building
- bob.devtools.constants.CONDA_RECIPE_APPEND = '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.9/site-packages/bob/devtools/data/recipe_append.yaml'¶
Extra information to be appended to every recipe upon building
- bob.devtools.constants.WEBDAV_PATHS = {False: {False: {'conda': '/conda/label/beta', 'docs': '/docs', 'root': '/private-upload'}, True: {'conda': '/conda/label/beta', 'docs': '/docs', 'root': '/public-upload'}}, True: {False: {'conda': '/conda', 'docs': '/docs', 'root': '/private-upload'}, True: {'conda': '/conda', 'docs': '/docs', 'root': '/public-upload'}}}¶
Default locations of our webdav upload paths
- bob.devtools.constants.CACERT_URL = 'https://curl.haxx.se/ca/cacert.pem'¶
Location of the most up-to-date CA certificate bundle
- bob.devtools.constants.CACERT = '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.9/site-packages/bob/devtools/data/cacert.pem'¶
We keep a copy of the CA certificates we trust here
To update this file use:
curl --remote-name --time-cond cacert.pem https://curl.haxx.se/ca/cacert.pem
More information here: https://curl.haxx.se/docs/caextract.html
- bob.devtools.constants.MATPLOTLIB_RCDIR = '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.9/site-packages/bob/devtools/data'¶
Base directory where the file matplotlibrc lives
It is required for certain builds that use matplotlib functionality.
- bob.devtools.constants.BOBRC_PATH = '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.9/site-packages/bob/devtools/data/bobrc'¶
The path to custom Bob configuration file to be used during the CI
Utilities to needed to release packages.
- bob.devtools.release.download_path(package, path, output=None, ref='master')[source]¶
Downloads paths from gitlab, with an optional recurse.
This method will download an archive of the repository from chosen reference, and then it will search insize the zip blob for the path to be copied into output. It uses
zipfile.ZipFile
to do this search. This method will not be very efficient for larger repository references, but works recursively by default.- Parameters
package – the gitlab package object to use (should be pre-fetched)
path – the path on the project to download
output – where to place the path to be downloaded - if not provided, use the basename of
path
as storage point with respect to the current directoryref – the name of the git reference (branch, tag or commit hash) to use
- bob.devtools.release.get_gitlab_instance()[source]¶
Returns an instance of the gitlab object for remote operations.
- bob.devtools.release.get_latest_tag_name(gitpkg)[source]¶
Find the name of the latest tag for a given package in the format ‘#.#.#’.
- Parameters
gitpkg – gitlab package object
Returns: The name of the latest tag in format ‘#.#.#’. None if no tags for the package were found.
- bob.devtools.release.get_parsed_tag(gitpkg, tag)[source]¶
An older tag is formatted as ‘v2.1.3 (Sep 22, 2017 10:37)’, from which we need only v2.1.3.
The latest tag is either patch, minor, major, or none
- bob.devtools.release.update_tag_comments(gitpkg, tag_name, tag_comments_list, dry_run=False)[source]¶
Write annotations inside the provided tag of a given package.
- Parameters
gitpkg – gitlab package object
tag_name – The name of the tag to update
tag_comments_list – New annotations for this tag in a form of list
dry_run – If True, nothing will be committed or pushed to GitLab
Returns: The gitlab object for the tag that was updated
- bob.devtools.release.update_files_with_mr(gitpkg, files_dict, message, branch, automerge, dry_run, user_id)[source]¶
Update (via a commit) files of a given gitlab package, through an MR.
This function can update a file in a gitlab package, but will do this through a formal merge request. You can auto-merge this request
- Parameters
gitpkg – gitlab package object
files_dict – Dictionary of file names and their contents (as text)
message – Commit message
branch – The branch name to use for the merge request
automerge – If we should set the “merge if build suceeds” flag on the created MR
dry_run – If True, nothing will be pushed to gitlab
user_id – The integer which numbers the user to attribute this MR to
- bob.devtools.release.update_files_at_master(gitpkg, files_dict, message, dry_run)[source]¶
Update (via a commit) files of a given gitlab package, directly on the master branch.
- Parameters
gitpkg – gitlab package object
files_dict – Dictionary of file names and their contents (as text)
message – Commit message
dry_run – If True, nothing will be committed or pushed to GitLab
- bob.devtools.release.get_last_pipeline(gitpkg)[source]¶
Returns the last pipeline of the project.
- Parameters
gitpkg – gitlab package object
Returns: The gtilab object of the pipeline
- bob.devtools.release.just_build_package(gitpkg, dry_run=False)[source]¶
Creates the pipeline with the latest tag and starts it.
- Parameters
gitpkg – gitlab package object
dry_run – If True, the pipeline will not be created on GitLab
Returns:
- bob.devtools.release.wait_for_pipeline_to_finish(gitpkg, pipeline_id, dry_run=False)[source]¶
Using sleep function, wait for the latest pipeline to finish building.
This function pauses the script until pipeline completes either successfully or with error.
- Parameters
gitpkg – gitlab package object
pipeline_id – id of the pipeline for which we are waiting to finish
dry_run – If True, outputs log message and exit. There wil be no waiting.
- bob.devtools.release.cancel_last_pipeline(gitpkg)[source]¶
Cancel the last started pipeline of a package.
- Parameters
gitpkg – gitlab package object
- bob.devtools.release.release_package(gitpkg, tag_name, tag_comments_list, dry_run=False)[source]¶
Release package.
The provided tag will be annotated with a given list of comments. README.rst and version.txt files will also be updated according to the release procedures.
- Parameters
gitpkg – gitlab package object
tag_name – The name of the release tag
tag_comments_list – New annotations for this tag in a form of list
dry_run – If True, nothing will be committed or pushed to GitLab
- bob.devtools.release.parse_and_process_package_changelog(gl, gitpkg, package_changelog, dry_run)[source]¶
Process the changelog of a single package.
Parse the log following specific format. Update annotations of the provided older tags and release the package by following the last tag description.
- Parameters
gl – Gitlab API object
gitpkg – gitlab package object
package_changelog – the changelog corresponding to the provided package
dry_run – If True, nothing will be committed or pushed to GitLab
Returns: the name of the latest tag, and tag’s comments
Utilities for retrieving, parsing and auto-generating changelogs.
- bob.devtools.changelog.get_file_from_gitlab(gitpkg, path, ref='master')[source]¶
Retrieves a file from a Gitlab repository, returns a (StringIO) file.
- bob.devtools.changelog.get_last_tag(package)[source]¶
Returns the last (gitlab object) tag for the given package.
- Parameters
package – The gitlab project object from where to fetch the last release date information
Returns: a tag object
- bob.devtools.changelog.get_last_tag_date(package)[source]¶
Returns the last release date for the given package.
Falls back to the first commit date if the package has not yet been tagged
- Parameters
package – The gitlab project object from where to fetch the last release date information
- Returns: a datetime object that refers to the last date the package was
released. If the package was never released, then returns the date just before the first commit.
- bob.devtools.changelog.get_changes_since(gitpkg, since)[source]¶
Gets the list of MRs, tags, and commits since the provided date
- bob.devtools.changelog.write_tags_with_commits(f, gitpkg, since, mode)[source]¶
Writes all tags and commits of a given package to the output file.
- Parameters
f – A
File
ready to be written atgitpkg – A pointer to the gitlab package object
since – Starting date (as a datetime object)
mode – One of mrs (merge-requests), commits or tags indicating how to list entries in the changelog for this package
- bob.devtools.changelog.write_tags(f, gitpkg, since)[source]¶
Writes all tags of a given package to the output file.
- Parameters
f – A
File
ready to be written atgitpkg – A pointer to the gitlab package object
since – Starting date as a datetime object
Bootstraps a new miniconda installation and prepares it for development.
- bob.devtools.bootstrap.set_environment(name, value, env=environ({'GITLAB_CHECKOUT_STRATEGY': 'https://gitlab-ci-token:QCGiwuWNiCQK4YQzAnmz@gitlab.idiap.ch/', 'CI_COMMIT_SHORT_SHA': '4b3f40f0', 'CI_API_V4_URL': 'https://gitlab.idiap.ch/api/v4', 'CI_PROJECT_NAME': 'docs', 'NOSE_EVAL_ATTR': '', 'DIRTY': '', 'CI_COMMIT_REF_PROTECTED': 'true', 'CI_DEPENDENCY_PROXY_USER': 'gitlab-ci-token', 'supkg': 'su-exec', 'BOOTSTRAP': 'https://gitlab.idiap.ch/bob/bob.devtools/raw/master/bob/devtools/bootstrap.py', 'CI_REGISTRY': 'docker.idiap.ch', 'CPU_COUNT': '40', 'CI_COMMIT_BRANCH': 'master', 'CI_DEPENDENCY_PROXY_PASSWORD': 'QCGiwuWNiCQK4YQzAnmz', 'CI_BUILD_TOKEN': 'QCGiwuWNiCQK4YQzAnmz', 'HOSTNAME': 'runner-h1nkcwbs-project-2426-concurrent-0', 'CI_PIPELINE_CREATED_AT': '2022-06-27T04:03:29Z', 'CI_PROJECT_URL': 'https://gitlab.idiap.ch/bob/docs', 'SSL_CERT_FILE': '/scratch/builds/bob/docs/miniconda/lib/python3.9/site-packages/bob/devtools/data/cacert.pem', 'CI_PROJECT_VISIBILITY': 'public', 'CI_JOB_URL': 'https://gitlab.idiap.ch/bob/docs/-/jobs/273275', 'FF_USE_NEW_SHELL_ESCAPE': 'false', 'FF_NETWORK_PER_BUILD': 'false', 'HISTSIZE': '1000', 'PYTHONNOUSERSITE': '1', 'CONDA_PERL': '5.26', 'CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX': 'gitlab.idiap.ch:443/bob/dependency_proxy/containers', 'CI_REGISTRY_USER': 'gitlab-ci-token', 'CONDA_SHLVL': '3', 'CI_SERVER_PROTOCOL': 'https', 'CONDA_PROMPT_MODIFIER': '(/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place) ', 'PYTHONUNBUFFERED': '1', 'CI_BUILD_BEFORE_SHA': '0000000000000000000000000000000000000000', 'FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY': 'false', 'PYPIPASS': 'pwd4Biometrics', 'GITLAB_USER_LOGIN': 'flavio.tarsetti', 'CI_SERVER_VERSION': '14.10.4', 'CI_BUILD_ID': '273275', 'GSETTINGS_SCHEMA_DIR_CONDA_BACKUP': '', 'GITLAB_USER_EMAIL': 'flavio.tarsetti@idiap.ch', 'BOB_BUILD_NUMBER': '56', 'DISTRO_NAME': 'centos', 'CI_PROJECT_ROOT_NAMESPACE': 'bob', 'FF_SCRIPT_SECTIONS': 'false', 'CONDA_OVERRIDE_CUDA': '11.6', 'CI_COMMIT_AUTHOR': 'Amir MOHAMMADI <amir.mohammadi@idiap.ch>', 'CI_RUNNER_EXECUTABLE_ARCH': 'linux/amd64', 'CI_DISPOSABLE_ENVIRONMENT': 'true', 'CI_COMMIT_TITLE': "Merge branch 'update' into 'master'", 'LC_ALL': 'en_US.UTF-8', 'CI_COMMIT_REF_NAME': 'master', 'CI_SERVER_VERSION_PATCH': '4', 'CI_SERVER_TLS_CA_FILE': '/scratch/builds/bob/docs.tmp/CI_SERVER_TLS_CA_FILE', 'USER': 'conda', 'CI_SERVER_VERSION_MINOR': '10', 'PERL_VER': '5.26', 'CI_PROJECT_ID': '2426', 'CI_JOB_TOKEN': 'QCGiwuWNiCQK4YQzAnmz', 'CI_JOB_IMAGE': 'quay.io/condaforge/linux-anvil-cos7-x86_64', 'sphinx': '4.4.0', 'DISTRO_VER': '7', 'CI_RUNNER_REVISION': 'febb2a09', 'CI_RUNNER_ID': '81', 'PY_VER': '3.9', 'CMAKE_GENERATOR': 'Unix Makefiles', 'CONDA_EXE': '/scratch/builds/bob/docs/miniconda/bin/conda', 'FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR': 'false', 'REQUESTS_CA_BUNDLE': '/scratch/builds/bob/docs/miniconda/lib/python3.9/site-packages/bob/devtools/data/cacert.pem', 'CI_SERVER_VERSION_MAJOR': '14', 'NPY_VER': '1.16', 'CONDA_R': '3.5', 'CONDA_BUILD_STATE': 'TEST', 'CI_COMMIT_DESCRIPTION': '\nUpdate installation instructions\n\nSee merge request bob/docs!28', 'FF_ENABLE_BASH_EXIT_CODE_CHECK': 'false', 'graphviz': '2.50.0', 'CI_COMMIT_MESSAGE': "Merge branch 'update' into 'master'\n\nUpdate installation instructions\n\nSee merge request bob/docs!28", 'CI_SERVER_URL': 'https://gitlab.idiap.ch', 'PYPIUSER': 'biometrics', 'CI_DEPENDENCY_PROXY_SERVER': 'gitlab.idiap.ch:443', 'CI_PROJECT_REPOSITORY_LANGUAGES': '', 'CI_PROJECT_TITLE': 'docs', 'PKG_BUILDNUM': '56', 'font_ttf_dejavu_sans_mono': '2.37', 'MKL_INTERFACE_LAYER': 'LP64,GNU', '_CE_CONDA': '', 'PYTHON_VERSION': '3.9', 'CI_BUILD_REF_NAME': 'master', 'CI_BUILD_REF': '4b3f40f0cdf2737736dc7f54db0b0767511e1396', 'CI_PIPELINE_ID': '62411', 'CONDA_PREFIX_1': '/opt/conda', 'CONDA_ROOT': '/scratch/builds/bob/docs/miniconda', 'CI_COMMIT_REF_SLUG': 'master', 'CI_PAGES_DOMAIN': 'example.com', 'CONDA_BUILD': '1', 'MAIL': '/var/spool/mail/conda', 'GITLAB_USER_NAME': 'Flavio TARSETTI', 'CI_REPOSITORY_URL': 'https://gitlab-ci-token:QCGiwuWNiCQK4YQzAnmz@gitlab.idiap.ch/bob/docs.git', 'PATH': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place:/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin:/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin:/opt/conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/conda/bin', 'CI_BUILD_STAGE': 'build', 'CI_REGISTRY_PASSWORD': 'QCGiwuWNiCQK4YQzAnmz', 'CI_PROJECT_DIR': '/scratch/builds/bob/docs', 'SRC_DIR': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/test_tmp', 'GSETTINGS_SCHEMA_DIR': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/share/glib-2.0/schemas', 'CI_RUNNER_TAGS': 'bob, linux, docker, intel', 'FF_USE_DIRECT_DOWNLOAD': 'true', 'CONDA_PREFIX_2': '/scratch/builds/bob/docs/miniconda', 'CONDA_MKL_INTERFACE_LAYER_BACKUP': '', 'CONDA_PREFIX': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place', 'CI_NODE_TOTAL': '1', 'LD_RUN_PATH': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib', 'BUILD': 'x86_64-conda_cos6-linux-gnu', 'PWD': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/test_tmp', 'nose': '1.3.7', 'CI_PIPELINE_SOURCE': 'schedule', 'CI_PAGES_URL': 'http://bob.example.com/docs', 'CI_SERVER_HOST': 'gitlab.idiap.ch', 'CI_JOB_STAGE': 'build', 'FF_POSIXLY_CORRECT_ESCAPES': 'false', 'PY3K': '1', 'PKG_VERSION': '2018.02.21b0', 'target_platform': 'linux-64', 'LANG': 'en_US.UTF-8', 'CI_PROJECT_PATH': 'bob/docs', 'CI_SERVER_NAME': 'GitLab', 'STDLIB_DIR': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.9', 'LUA_VER': '5', 'DOCPASS': 'zXmol5PPQz', 'CI_COMMIT_BEFORE_SHA': '0000000000000000000000000000000000000000', 'GITLAB_FEATURES': '', 'CONDA_PY': '39', 'CI_PIPELINE_IID': '1710', 'SHLIB_EXT': '.so', 'CI_PIPELINE_URL': 'https://gitlab.idiap.ch/bob/docs/-/pipelines/62411', 'FF_ENABLE_JOB_CLEANUP': 'false', 'CI_RUNNER_VERSION': '15.0.0', 'CI_SERVER_PORT': '443', 'GITLAB_CI': 'true', 'ROOT': '/scratch/builds/bob/docs/miniconda', 'BUILD_EGG': 'true', 'CI_SERVER_REVISION': '3297a73eca3', 'HTTPS_PROXY': '', '_CE_M': '', 'HISTCONTROL': 'ignoredups', 'CI_COMMIT_SHA': '4b3f40f0cdf2737736dc7f54db0b0767511e1396', 'PYTEST_ADDOPTS': "-m ''", 'BUILD_PREFIX': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_', 'TORCH_HOME': '/scratch/builds/bob/docs/torch', 'CI_BUILDS_DIR': '/scratch/builds', 'SHLVL': '3', 'HOME': '/home/conda', 'CI_CONFIG_PATH': '.gitlab-ci.yml', 'CI_BUILD_NAME': 'build', 'FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION': 'false', 'PKG_BUILD_STRING': 'hca615a1_56', 'CI_PROJECT_PATH_SLUG': 'bob-docs', 'CI_SERVER': 'yes', 'FF_SKIP_NOOP_BUILD_STAGES': 'true', 'build_platform': 'linux-64', 'FF_USE_POWERSHELL_PATH_RESOLVER': 'false', 'sphinxcontrib_programoutput': '0.16', 'CI': 'true', 'CI_RUNNER_SHORT_TOKEN': 'h1nkcWBs', 'FF_USE_IMPROVED_URL_MASKING': 'false', 'CI_DEFAULT_BRANCH': 'master', 'CI_JOB_STATUS': 'running', 'CONDA_ALLOW_SOFTLINKS': 'false', 'HTTP_PROXY': '', 'SP_DIR': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.9/site-packages', 'CI_PROJECT_NAMESPACE': 'bob', 'SYS_PREFIX': '/scratch/builds/bob/docs/miniconda', 'CONDA_PYTHON_EXE': '/scratch/builds/bob/docs/miniconda/bin/python', 'LOGNAME': 'conda', 'SUBDIR': 'linux-64', 'CONDA_NPY': '116', 'sphinx_rtd_theme': '1.0.0', 'CI_CONCURRENT_ID': '1', 'PREFIX': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place', 'NPY_DISTUTILS_APPEND_FLAGS': '1', 'R_VER': '3.5', 'CI_COMMIT_TIMESTAMP': '2022-06-03T15:13:13+00:00', 'CI_PROJECT_CLASSIFICATION_LABEL': '', 'CI_JOB_STARTED_AT': '2022-06-27T04:03:33Z', 'CONDA_LUA': '5', 'RECIPE_DIR': '/tmp/tmpza9djb4q/info/recipe', 'CONDA_DEFAULT_ENV': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place', 'CI_BUILD_REF_SLUG': 'master', 'FF_USE_NEW_BASH_EVAL_STRATEGY': 'false', 'PKG_CONFIG_PATH': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/pkgconfig', 'BOBRC': '/scratch/builds/bob/docs/miniconda/lib/python3.9/site-packages/bob/devtools/data/bobrc', 'BOB_DOCUMENTATION_SERVER': 'http://www.idiap.ch/software/bob/docs/bob/%(name)s/master/', 'ARCH': '64', 'MATPLOTLIBRC': '/scratch/builds/bob/docs/miniconda/lib/python3.9/site-packages/bob/devtools/data', 'BOB_PACKAGE_VERSION': '2018.02.21b0', 'CI_RUNNER_DESCRIPTION': 'linux-docker', 'CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX': 'gitlab.idiap.ch:443/bob/dependency_proxy/containers', 'FF_USE_FASTZIP': 'false', 'DOCUSER': 'uploader', 'GITLAB_USER_ID': '13', 'FF_KUBERNETES_HONOR_ENTRYPOINT': 'false', 'CI_JOB_ID': '273275', 'SYS_PYTHON': '/scratch/builds/bob/docs/miniconda/bin/python', 'PKG_NAME': 'bob.docs', 'PKG_HASH': 'hca615a1', 'CI_JOB_NAME': 'build', 'CI_CONCURRENT_PROJECT_ID': '0', 'FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY': 'true', 'coverage': '6.3.2', 'FF_USE_DYNAMIC_TRACE_FORCE_SEND_INTERVAL': 'false', '_': '/scratch/builds/bob/docs/miniconda/conda-bld/bob.docs_1656302849688/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin/sphinx-build', 'DOCUTILSCONFIG': '/scratch/builds/bob/docs/doc/docutils.conf', 'KMP_DUPLICATE_LIB_OK': 'True', 'KMP_INIT_AT_FORK': 'FALSE'}))[source]¶
Function to setup the environment variable and print debug message.
- Parameters
name – The name of the environment variable to set
value – The value to set the environment variable to
env – Optional environment (dictionary) where to set the variable at
- bob.devtools.bootstrap.human_time(seconds, granularity=2)[source]¶
Returns a human readable time string like “1 day, 2 hours”.
- bob.devtools.bootstrap.run_cmdline(cmd, env=None, **kwargs)[source]¶
Runs a command on a environment, logs output and reports status.
- Parameters
cmd (list) – The command to run, with parameters separated on a list
env (dict, Optional) – Environment to use for running the program on. If not set, use
os.environ
.
- bob.devtools.bootstrap.touch(path)[source]¶
Python-implementation of the “touch” command-line application.
- bob.devtools.bootstrap.merge_conda_cache(cache, prefix, name)[source]¶
Merges conda pkg caches and conda-bld folders.
- Parameters
cache – The cached directory (from previous builds)
prefix – The current prefix (root of conda installation)
name – The name of the current package
- bob.devtools.bootstrap.ensure_miniconda_sh()[source]¶
Retrieves the miniconda3 installer for the current system.
Checks the hash of the miniconda3 installer against the expected version, if that does not match, erase existing installer and re-downloads new installer.
- bob.devtools.bootstrap.install_miniconda(prefix, name)[source]¶
Creates a new miniconda installation.
- Parameters
prefix – The path leading to the (new) root of the miniconda installation
name – The name of this package
- bob.devtools.bootstrap.get_channels(public, stable, server, intranet, group, add_dependent_channels=False)[source]¶
Returns the relevant conda channels to consider if building project.
The subset of channels to be returned depends on the visibility and stability of the package being built. Here are the rules:
public and stable: returns the public stable channel
public and not stable: returns the public beta channel
not public and stable: returns both public and private stable channels
not public and not stable: returns both public and private beta channels
Public channels have priority over private channles, if turned.
- Parameters
public – Boolean indicating if we’re supposed to include only public channels
stable – Boolean indicating if we’re supposed to include stable channels
server – The base address of the server containing our conda channels
intranet – Boolean indicating if we should add “private”/”public” prefixes on the conda paths
group – The group of packages (gitlab namespace) the package we’re compiling is part of. Values should match URL namespaces currently available on our internal webserver. Currently, only “bob” or “beat” will work.
add_dependent_channels – If True, will add the conda-forge channel to the list
Returns: a list of channels that need to be considered.
- bob.devtools.bootstrap.setup_logger(logger, level)[source]¶
Sets-up the logging for this command at level
INFO
Tools for self-building and other utilities.
- bob.devtools.build.root_logger_protection()[source]¶
Protects the root logger against spurious (conda) manipulation
- bob.devtools.build.comment_cleanup(lines)[source]¶
Cleans-up comments and empty lines from textual data read from files.
- bob.devtools.build.load_order_file(path)[source]¶
Loads an order.txt style file, removes empty lines and comments.
- bob.devtools.build.conda_arch()[source]¶
Returns the current OS name and architecture as recognized by conda.
- bob.devtools.build.should_skip_build(metadata_tuples)[source]¶
Takes the output of render_recipe as input and evaluates if this recipe’s build should be skipped.
- bob.devtools.build.next_build_number(channel_url, basename)[source]¶
Calculates the next build number of a package given the channel.
This function returns the next build number (integer) for a package given its resulting tarball base filename (can be obtained with
get_output_path()
).- Parameters
channel_url – The URL where to look for packages clashes (normally a beta channel)
basename – The tarball basename to check on the channel
Returns: The next build number with the current configuration. Zero (0) is returned if no match is found. Also returns the URLs of the packages it finds with matches on the name, version and python-version, ordered by (reversed) build-number.
- bob.devtools.build.make_conda_config(config, python, append_file, condarc_options)[source]¶
Creates a conda configuration for a build merging various sources.
This function will use the conda-build API to construct a configuration by merging different sources of information.
- Parameters
config – Path leading to the
conda_build_config.yaml
to usepython – The version of python to use for the build as
x.y
(e.g.3.6
)append_file – Path leading to the
recipe_append.yaml
file to usecondarc_options – A dictionary (typically read from a condarc YAML file) that contains build and channel options
Returns: A dictionary containing the merged configuration, as produced by conda-build API’s
get_or_merge_config()
function.
- bob.devtools.build.get_output_path(metadata, config)[source]¶
Renders the recipe and returns the name of the output file.
- bob.devtools.build.use_mambabuild()[source]¶
Will inject mamba solver to conda build API to speed up resolves
- bob.devtools.build.get_rendered_metadata(recipe_dir, config)[source]¶
Renders the recipe and returns the interpreted YAML file.
- bob.devtools.build.get_parsed_recipe(metadata)[source]¶
Renders the recipe and returns the interpreted YAML file.
- bob.devtools.build.exists_on_channel(channel_url, basename)[source]¶
Checks on the given channel if a package with the specs exist.
This procedure always ignores the package hash code, if one is set. It differentiates between .conda and .tar.bz2 packages.
- Parameters
channel_url – The URL where to look for packages clashes (normally a beta channel)
basename – The basename of the tarball to search for
Returns: A complete package url, if the package already exists in the channel or
None
otherwise.
- bob.devtools.build.get_env_directory(conda, name)[source]¶
Get the directory of a particular conda environment or fail silently.
- bob.devtools.build.conda_create(conda, name, overwrite, condarc, packages, dry_run, use_local)[source]¶
Creates a new conda environment following package specifications.
This command can create a new conda environment following the list of input packages. It will overwrite an existing environment if indicated.
- Parameters
conda – path to the main conda executable of the installation
name – the name of the environment to create or overwrite
overwrite – if set to
`True
, overwrite potentially existing environments with the same namecondarc – a dictionary of options for conda, including channel urls
packages – the package list specification
dry_run – if set, then don’t execute anything, just print stuff
use_local – include the local conda-bld directory as a possible installation channel (useful for testing multiple interdependent recipes that are built locally)
- bob.devtools.build.get_docserver_setup(public, stable, server, intranet, group)[source]¶
Returns a setup for BOB_DOCUMENTATION_SERVER.
BOB_DOCUMENTATION_SERVER is used in bob.extension.utils.link_documentation to find the links of the documentation of Bob packages. The initial servers in the output servers list have higher priority.
What is available to build the documentation depends on the setup of
public
andstable
:public and stable: only returns the public stable folder
public and not stable: only returns the public beta folder
not public and stable: returns public and private stable folders
not public and not stable: returns public and private beta folders
If returned, Priavte channels have lower priority over public channles because, private packages become public eventually.
- Parameters
public – Boolean indicating if we’re supposed to include only public channels
stable – Boolean indicating if we’re supposed to include only stable channels
server – The base address of the server containing our conda channels
intranet – Boolean indicating if we should add “private”/”public” prefixes on the returned paths
group – The group of packages (gitlab namespace) the package we’re
compiling – is part of. Values should match URL namespaces currently available on our internal webserver. Currently, only “bob” or “beat” will work.
Returns: a string to be used by bob.extension to find dependent documentation projects.
- bob.devtools.build.check_version(workdir, envtag)[source]¶
Checks if the version being built and the value reported match.
This method will read the contents of the file
version.txt
and compare it to the potentially setenvtag
(may beNone
). If the value ofenvtag
is different thanNone
, ensure it matches the value inversion.txt
or raises an exception.- Parameters
workdir – The work directory where the repo of the package being built was checked-out
envtag – (optional) tag provided by the environment
Returns: A tuple with the version of the package that we’re currently building and a boolean flag indicating if the version number represents a pre-release or a stable release.
- bob.devtools.build.git_clean_build(runner, verbose)[source]¶
Runs git-clean to clean-up build products.
- Parameters
runner – A pointer to the
run_cmdline()
functionverbose – A boolean flag indicating if the git command should report erased files or not
- bob.devtools.build.base_build(bootstrap, server, intranet, group, recipe_dir, conda_build_config, condarc_options)[source]¶
Builds a non-beat/non-bob software dependence that doesn’t exist on conda-forge.
This function will build a software dependence that is required for our software stack, but does not (yet) exist on the conda-forge channels. It first check if the build should run for the current architecture, checks if the package is not already built on our public channel and, if that is true, then proceeds with the build of the dependence.
- Parameters
bootstrap – Module that should be pre-loaded so this function can be used in a pre-bdt build
server – The base address of the server containing our conda channels
intranet – Boolean indicating if we should add “private”/”public” prefixes on the returned paths
group – The group of packages (gitlab namespace) the package we’re compiling is part of. Values should match URL namespaces currently available on our internal webserver. Currently, only “bob” or “beat” will work.
recipe_dir – The directory containing the recipe’s
meta.yaml
fileconda_build_config – Path to the
conda_build_config.yaml
file to usecondarc_options – Pre-parsed condarc options loaded from the respective YAML file
- Returns
The list of built packages, as returned by
conda_build.api.build()
- Return type
- bob.devtools.build.load_packages_from_conda_build_config(conda_build_config, condarc_options)[source]¶
- bob.devtools.build.bob_devel(bootstrap, server, intranet, group, conda_build_config, condarc_options, work_dir)[source]¶
Tests that all packages listed in bob/devtools/data/conda_build_config.yaml can be installed in one environment.
Mirroring functionality for conda channels
Some constructs are bluntly copied from https://github.com/valassis-digital-media/conda-mirror
- bob.devtools.mirror.get_json(channel, platform, name)[source]¶
Get a JSON file for a channel/platform combo on conda channel
- Parameters
- Returns
repodata – contents of repodata.json
- Return type
- Raises
RuntimeError : – If the URL cannot be reached
- bob.devtools.mirror.get_local_contents(path, arch)[source]¶
Returns the local package contents as a set
- bob.devtools.mirror.load_glob_list(path)[source]¶
Loads a list of globs from a configuration file
Excludes comments and empty lines
- bob.devtools.mirror.blacklist_filter(packages, globs)[source]¶
Filters out the input package set with the glob list
- bob.devtools.mirror.whitelist_filter(packages, globs)[source]¶
Filters in the input package set with the glob list
- bob.devtools.mirror.download_packages(packages, repodata, channel_url, dest_dir, arch, dry_run)[source]¶
Downloads remote packages to a download directory
Packages are downloaded first to a temporary directory, then validated according to the expected sha256/md5 sum and then moved, one by one, to the destination directory. An error is raised if the package cannot be correctly downloaded.
- Parameters
packages (list of str) – List of packages to download from the remote channel
repodata (dict) – A dictionary containing the remote repodata.json contents
channel_url (str) – The complete channel URL
dest_dir (str) – The local directory where the channel is being mirrored
arch (str) – The current architecture which we are mirroring
dry_run (bool) – A boolean flag indicating if this is just a dry-run (simulation), flagging so we don’t really do anything (set to
True
).
- bob.devtools.mirror.remove_packages(packages, dest_dir, arch, dry_run)[source]¶
Removes local packages that no longer matter
- bob.devtools.mirror.copy_and_clean_json(url, dest_dir, arch, name, dry_run)[source]¶
Copies and cleans conda JSON file
- bob.devtools.mirror.copy_and_clean_patch(url, dest_dir, arch, name, dry_run)[source]¶
Copies and cleans conda patch_instructions JSON file
- bob.devtools.mirror.checksum_packages(repodata, dest_dir, arch, packages)[source]¶
Checksums packages on the local mirror and compare to remote repository
- Parameters
- Returns
issues – List of matching errors
- Return type
Deployment utilities for conda packages and documentation via webDAV.
- bob.devtools.deploy.deploy_conda_package(package, arch, stable, public, username, password, overwrite, dry_run)[source]¶
Deploys a single conda package on the appropriate path.
- Parameters
package (str) – Path leading to the conda package to be deployed
arch (str) – The conda architecture to deploy to (
linux-64
,linux-aarch64
,osx-64
,osx-arm64
,noarch
, orNone
- in which case the architecture is going to be guessed from the directory where the package sits)stable (bool) – Indicates if the package should be deployed on a stable (
True
) or beta (False
) channelpublic (bool) – Indicates if the package is supposed to be distributed publicly or privatly (within Idiap network)
username (str) – The name of the user on the webDAV server to use for uploading the package
password (str) – The password of the user on the webDAV server to use for uploading the package
overwrite (bool) – If we should overwrite a package with equal name existing on the destination directory. Otherwise, an exception is raised.
dry_run (bool) – If we’re supposed to really do the actions, or just log messages.
- bob.devtools.deploy.deploy_documentation(path, package, stable, latest, public, branch, tag, username, password, dry_run)[source]¶
Deploys sphinx documentation to the appropriate webdav locations.
- Parameters
path (str) – Path leading to the root of the documentation to be deployed
package (str) – Full name (with namespace) of the package being treated
stable (bool) – Indicates if the documentation corresponds to the latest stable build
latest (bool) – Indicates if the documentation being deployed correspond to the latest stable for the package or not. In case the documentation comes from a patch release which is not on the master branch, please set this flag to
False
, which will make us avoid deployment of the documentation tomaster
andstable
sub-directories.public (bool) – Indicates if the documentation is supposed to be distributed publicly or privatly (within Idiap network)
branch (str) – The name of the branch for the current build
tag (str) – The name of the tag currently built (may be
None
)username (str) – The name of the user on the webDAV server to use for uploading the package
password (str) – The password of the user on the webDAV server to use for uploading the package
dry_run (bool) – If we’re supposed to really do the actions, or just log messages.
Utilities for calculating package dependencies and drawing graphs
- bob.devtools.graph.compute_adjencence_matrix(gl, package, conda_config, main_channel, recurse_regexp='^(bob|beat|batl|gridtk)(\\.)?(?!-).*$', current={}, ref='master', deptypes=[])[source]¶
Given a target package, returns an adjacence matrix with its dependencies returned via the conda-build API
- Parameters
gl (object) – Pre-instantiated instance of the gitlab server API to use, of type
gitlab.Gitlab
.package (str) – Name of the package, including its group in the format
group/package
conda_config (dict) – Dictionary of conda configuration options loaded from command-line and read from defaults available.
main_channel (str) – Main channel to consider when looking for the next build number of the target package
recurse_regexp (str) – Regular expression to use, for determining where to recurse for resolving dependencies. Typically, this should be set to a list of packages which exists in gitlab. If it includes more than that, then we may not be able to reach the package repository and an error will be raised. The default expression avoids recursing over bob/beat-devel packages.
current (dict) – Current list of packages already inspected - useful for recurrent calls to this function, to avoid potential cyclic dependencies. Corresponds to the current return value of this function.
ref (str) – Name of the git reference (branch, tag or commit hash) to use
deptypes (list) – A list of dependence types to preserve when building the graph. If empty, then preserve all. You may set values “build”, “host”, “run” and “test”, in any combination
- Returns
adjacence_matrix – A dictionary that contains the dependencies of all packages considered in the recursion. The keys are the name of the packages, the values, correspond to the dependencies (host, build, run and test) as a list of strings.
- Return type
- bob.devtools.graph.generate_graph(adjacence_matrix, deptypes, whitelist)[source]¶
Computes a graphviz/dot representation of the build graph
- Parameters
adjacence_matrix (dict) – A dictionary containing the adjacence matrix, that states the dependencies for each package in the build, to other packages
deptypes (list) – A list of dependence types to preserve when building the graph. If empty, then preserve all. You may set values “build”, “host”, “run” and “test”, in any combination
whitelist (str) – Regular expression for matching strings to preserve while building the graph
- Returns
graph – The generated graph
- Return type
graphviz.Digraph