Python API

bob.devtools.log	Logging utilities
bob.devtools.ci	Tools to help CI-based builds and artifact deployment
bob.devtools.constants	Constants used for building and more
bob.devtools.release	Utilities to needed to release packages
bob.devtools.changelog	Utilities for retrieving, parsing and auto-generating changelogs
bob.devtools.bootstrap	Bootstraps a new miniconda installation and prepares it for development
bob.devtools.build	Tools for self-building and other utilities
bob.devtools.webdav3.client

Detailed Information

Logging utilities

bob.devtools.log.setup(logger_name, format='%(levelname)s:%(name)s@%(asctime)s: %(message)s')

This function returns a logger object that is set up to perform logging using Bob loggers.

Parameters:

• logger_name (str) – The name of the module to generate logs for
• format (str, optional) – The format of the logs, see logging.LogRecord for more details. By default, the log contains the logger name, the log time, the log level and the massage.

Returns:

logger – The logger configured for logging. The same logger can be retrieved using the logging.getLogger() function.

Return type:

logging.Logger

bob.devtools.log.set_verbosity_level(logger, level)

Sets the log level for the given logger.

Parameters:

• logger (logging.Logger or str) – The logger to generate logs for, or the name of the module to generate logs for.
• level (int) – Possible log levels are: 0: Error; 1: Warning; 2: Info; 3: Debug.

Raises:

ValueError – If the level is not in range(0, 4).

bob.devtools.log.verbosity_option(**kwargs)

Adds a -v/–verbose option to a click command.

Parameters:**kwargs – All kwargs are passed to click.option. Returns:A decorator to be used for adding this option. Return type:callable

Tools to help CI-based builds and artifact deployment

bob.devtools.ci.is_master(refname, tag, repodir)

Tells if we’re on the master branch via ref_name or tag

This function checks if the name of the branch being built is “master”. If a tag is set, then it checks if the tag is on the master branch. If so, then also returns True, otherwise, False.

Parameters:

• refname – The value of the environment variable CI_COMMIT_REF_NAME
• tag – The value of the environment variable CI_COMMIT_TAG - (may be None)

Returns: a boolean, indicating we’re building the master branch or that the tag being built was issued on the master branch.

bob.devtools.ci.is_stable(package, refname, tag, repodir)

Determines if the package being published is stable

This is done by checking if a tag was set for the package. If that is the case, we still cross-check the tag is on the “master” branch. If everything checks out, we return True. Else, False.

Parameters:

• package – Package name in the format “group/name”
• refname – The current value of the environment CI_COMMIT_REF_NAME
• tag – The current value of the enviroment CI_COMMIT_TAG (may be None)
• repodir – The directory that contains the clone of the git repository

Returns: a boolean, indicating if the current build is for a stable release

Constants used for building and more

bob.devtools.constants.BASE_CONDARC = 'default_channels:\n - https://repo.anaconda.com/pkgs/main\n - https://repo.anaconda.com/pkgs/free\n - https://repo.anaconda.com/pkgs/r\n - https://repo.anaconda.com/pkgs/pro\nadd_pip_as_python_dependency: false #!final\nchangeps1: false #!final\nalways_yes: true #!final\nquiet: true #!final\nshow_channel_urls: true #!final\nanaconda_upload: false #!final\nssl_verify: false #!final\nchannels:\n - defaults\n'

Default setup for conda builds

bob.devtools.constants.CONDA_BUILD_CONFIG = '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/bob/devtools/data/conda_build_config.yaml'

Configuration variants we like building

bob.devtools.constants.CONDA_RECIPE_APPEND = '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/bob/devtools/data/recipe_append.yaml'

Extra information to be appended to every recipe upon building

bob.devtools.constants.SERVER = 'http://www.idiap.ch'

This is the default server use use to store data and build artifacts

bob.devtools.constants.WEBDAV_PATHS = {False: {False: {'conda': '/conda/label/beta', 'docs': '/docs', 'root': '/private-upload'}, True: {'conda': '/conda/label/beta', 'docs': '/docs', 'root': '/public-upload'}}, True: {False: {'conda': '/conda', 'docs': '/docs', 'root': '/private-upload'}, True: {'conda': '/conda', 'docs': '/docs', 'root': '/public-upload'}}}

Default locations of our webdav upload paths

bob.devtools.constants.CACERT_URL = 'https://curl.haxx.se/ca/cacert.pem'

Location of the most up-to-date CA certificate bundle

bob.devtools.constants.CACERT = '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/bob/devtools/data/cacert.pem'

We keep a copy of the CA certificates we trust here

To update this file use: curl --remote-name --time-cond cacert.pem https://curl.haxx.se/ca/cacert.pem

More information here: https://curl.haxx.se/docs/caextract.html

bob.devtools.constants.MATPLOTLIB_RCDIR = '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/bob/devtools/data'

Base directory where the file matplotlibrc lives

It is required for certain builds that use matplotlib functionality.

Utilities to needed to release packages

bob.devtools.release.download_path(package, path, output=None, ref='master')

Downloads paths from gitlab, with an optional recurse

This method will download an archive of the repository from chosen reference, and then it will search insize the zip blob for the path to be copied into output. It uses zipfile.ZipFile to do this search. This method will not be very efficient for larger repository references, but works recursively by default.

Parameters:

• package – the gitlab package object to use (should be pre-fetched)
• path – the path on the project to download
• output – where to place the path to be downloaded - if not provided, use the basename of path as storage point with respect to the current directory
• ref – the name of the git reference (branch, tag or commit hash) to use

bob.devtools.release.get_gitlab_instance()

Returns an instance of the gitlab object for remote operations

bob.devtools.release.get_latest_tag_name(gitpkg)

Find the name of the latest tag for a given package in the format ‘#.#.#’

Parameters:gitpkg – gitlab package object

Returns: The name of the latest tag in format ‘#.#.#’. None if no tags for the package were found.

bob.devtools.release.get_parsed_tag(gitpkg, tag)

An older tag is formatted as ‘v2.1.3 (Sep 22, 2017 10:37)’, from which we need only v2.1.3

The latest tag is either patch, minor, major, or none

bob.devtools.release.update_tag_comments(gitpkg, tag_name, tag_comments_list, dry_run=False)

Write annotations inside the provided tag of a given package.

Parameters:

• gitpkg – gitlab package object
• tag_name – The name of the tag to update
• tag_comments_list – New annotations for this tag in a form of list
• dry_run – If True, nothing will be committed or pushed to GitLab

Returns: The gitlab object for the tag that was updated

bob.devtools.release.commit_files(gitpkg, files_dict, message='Updated files', dry_run=False)

Commit files of a given GitLab package.

Parameters:

• gitpkg – gitlab package object
• files_dict – Dictionary of file names and their contents (as text)
• message – Commit message
• dry_run – If True, nothing will be committed or pushed to GitLab

bob.devtools.release.get_last_pipeline(gitpkg)

Returns the last pipeline of the project

Parameters:gitpkg – gitlab package object

Returns: The gtilab object of the pipeline

bob.devtools.release.just_build_package(gitpkg, dry_run=False)

Creates the pipeline with the latest tag and starts it

Parameters:

• gitpkg – gitlab package object
• dry_run – If True, the pipeline will not be created on GitLab

Returns:

bob.devtools.release.wait_for_pipeline_to_finish(gitpkg, pipeline_id, dry_run=False)

Using sleep function, wait for the latest pipeline to finish building.

This function pauses the script until pipeline completes either successfully or with error.

Parameters:

• gitpkg – gitlab package object
• pipeline_id – id of the pipeline for which we are waiting to finish
• dry_run – If True, outputs log message and exit. There wil be no waiting.

bob.devtools.release.cancel_last_pipeline(gitpkg)

Cancel the last started pipeline of a package

Parameters:gitpkg – gitlab package object

bob.devtools.release.release_package(gitpkg, tag_name, tag_comments_list, dry_run=False)

Release package

The provided tag will be annotated with a given list of comments. README.rst and version.txt files will also be updated according to the release procedures.

Parameters:

• gitpkg – gitlab package object
• tag_name – The name of the release tag
• tag_comments_list – New annotations for this tag in a form of list
• dry_run – If True, nothing will be committed or pushed to GitLab

bob.devtools.release.parse_and_process_package_changelog(gl, gitpkg, package_changelog, dry_run)

Process the changelog of a single package

Parse the log following specific format. Update annotations of the provided older tags and release the package by following the last tag description.

Parameters:

• gl – Gitlab API object
• gitpkg – gitlab package object
• package_changelog – the changelog corresponding to the provided package
• dry_run – If True, nothing will be committed or pushed to GitLab

Returns: the name of the latest tag, and tag’s comments

bob.devtools.release.release_bob(changelog_file)

Process the changelog and releases the bob metapackage

Utilities for retrieving, parsing and auto-generating changelogs

bob.devtools.changelog.parse_date(d)

Parses any date supported by dateutil.parser.parse()

bob.devtools.changelog.get_file_from_gitlab(gitpkg, path, ref='master')

Retrieves a file from a Gitlab repository, returns a (StringIO) file

bob.devtools.changelog.get_last_tag(package)

Returns the last (gitlab object) tag for the given package

Parameters:package – The gitlab project object from where to fetch the last release date information

Returns: a tag object

bob.devtools.changelog.get_last_tag_date(package)

Returns the last release date for the given package

Falls back to the first commit date if the package has not yet been tagged

Parameters:package – The gitlab project object from where to fetch the last release date information

Returns: a datetime object that refers to the last date the package was

released. If the package was never released, then returns the date just before the first commit.

bob.devtools.changelog.write_tags_with_commits(f, gitpkg, since, mode)

Writes all tags and commits of a given package to the output file

Parameters:

• f – A File ready to be written at
• gitpkg – A pointer to the gitlab package object
• since – Starting date (as a datetime object)
• mode – One of mrs (merge-requests), commits or tags indicating how to list entries in the changelog for this package

bob.devtools.changelog.write_tags(f, gitpkg, since)

Writes all tags of a given package to the output file

Parameters:

• f – A File ready to be written at
• gitpkg – A pointer to the gitlab package object
• since – Starting date as a datetime object

Bootstraps a new miniconda installation and prepares it for development

bob.devtools.bootstrap.set_environment(name, value, env=environ({'CI_PROJECT_NAME': 'bob.devtools', 'DIRTY': '', 'CI_REGISTRY': 'docker.idiap.ch', 'CPU_COUNT': '4', 'HOSTNAME': 'runner-fd5d3bf0-project-2853-concurrent-0', 'CI_BUILD_TOKEN': 'W596C2CQdt3FBS91ynn9', 'CI_PROJECT_URL': 'https://gitlab.idiap.ch/bob/bob.devtools', 'CI_PROJECT_VISIBILITY': 'public', 'CI_JOB_URL': 'https://gitlab.idiap.ch/bob/bob.devtools/-/jobs/155413', 'PYTHONNOUSERSITE': '1', 'CONDA_PERL': '5.26', 'CI_REGISTRY_USER': 'gitlab-ci-token', 'CONDA_SHLVL': '2', 'CONDA_PROMPT_MODIFIER': '', 'CI_BUILD_BEFORE_SHA': '0000000000000000000000000000000000000000', 'PYTHONUNBUFFERED': '1', 'CI_BUILD_ID': '155413', 'CI_SERVER_VERSION': '11.6.2', 'GITLAB_USER_LOGIN': 'andre.anjos', 'PYPIPASS': 'pwd4Biometrics', 'GITLAB_USER_EMAIL': 'andre.anjos@idiap.ch', 'BOB_BUILD_NUMBER': '0', 'CI_COMMIT_TITLE': 'Increased stable version to 0.1.1', 'CI_DISPOSABLE_ENVIRONMENT': 'true', 'CI_RUNNER_EXECUTABLE_ARCH': 'linux/amd64', 'LC_ALL': 'en_US.UTF-8', 'CI_SERVER_TLS_CA_FILE': '/local/builds/bob/bob.devtools.tmp/CI_SERVER_TLS_CA_FILE', 'CI_SERVER_VERSION_PATCH': '2', 'CI_COMMIT_REF_NAME': 'v0.1.1', 'CI_SERVER_VERSION_MINOR': '6', 'PERL_VER': '5.26', 'CI_JOB_TOKEN': 'W596C2CQdt3FBS91ynn9', 'CI_PROJECT_ID': '2853', 'CI_RUNNER_ID': '18', 'CI_RUNNER_REVISION': 'f100a208', 'DOCSERVER': 'http://www.idiap.ch', 'PY_VER': '3.7', 'CMAKE_GENERATOR': 'Unix Makefiles', 'CONDA_EXE': '/local/builds/bob/bob.devtools/miniconda/bin/conda', 'REQUESTS_CA_BUNDLE': '', 'CI_SERVER_VERSION_MAJOR': '11', 'NPY_VER': '1.11', 'CONDA_R': '3.5', 'CONDA_BUILD_STATE': 'TEST', 'CI_COMMIT_DESCRIPTION': '', 'CI_COMMIT_MESSAGE': 'Increased stable version to 0.1.1', 'PYPIUSER': 'biometrics', 'PKG_BUILDNUM': '0', 'CI_PIPELINE_ID': '26238', 'CI_BUILD_REF': '9913b0b178e743d9a4d74e6099dcdcda143069fc', 'CI_BUILD_REF_NAME': 'v0.1.1', 'PYTHON_VERSION': '3.6', 'CI_COMMIT_REF_SLUG': 'v0-1-1', 'CI_BUILD_TAG': 'v0.1.1', 'CONDA_ROOT': '/local/builds/bob/bob.devtools/miniconda', 'CONDA_BUILD': '1', 'CONDA_PREFIX_1': '/local/builds/bob/bob.devtools/miniconda', 'PATH': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin:/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho:/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin:/local/builds/bob/bob.devtools/miniconda/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'FF_K8S_USE_ENTRYPOINT_OVER_COMMAND': 'true', 'CI_REPOSITORY_URL': 'https://gitlab-ci-token:W596C2CQdt3FBS91ynn9@gitlab.idiap.ch/bob/bob.devtools.git', 'GITLAB_USER_NAME': 'André Anjos', 'CI_PROJECT_DIR': '/local/builds/bob/bob.devtools', 'CI_REGISTRY_PASSWORD': 'W596C2CQdt3FBS91ynn9', 'CI_BUILD_STAGE': 'build', 'SRC_DIR': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/test_tmp', 'CI_RUNNER_TAGS': 'docker, deployer', 'CONDA_PREFIX': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho', 'CI_NODE_TOTAL': '1', 'LD_RUN_PATH': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib', 'BUILD': 'x86_64-conda_cos6-linux-gnu', 'PWD': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/test_tmp', 'CI_JOB_STAGE': 'build', 'CI_PIPELINE_SOURCE': 'push', 'PY3K': '1', 'PKG_VERSION': '0.1.1', 'CI_SERVER_NAME': 'GitLab', 'CI_PROJECT_PATH': 'bob/bob.devtools', 'LANG': 'en_US.UTF-8', 'STDLIB_DIR': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6', 'LUA_VER': '5', 'GITLAB_FEATURES': '', 'CI_COMMIT_BEFORE_SHA': '0000000000000000000000000000000000000000', 'DOCPASS': 'zXmol5PPQz', 'CONDA_PY': '37', 'CI_PIPELINE_IID': '146', 'SHLIB_EXT': '.so', 'CI_PIPELINE_URL': 'https://gitlab.idiap.ch/bob/bob.devtools/pipelines/26238', 'GITLAB_CI': 'true', 'CI_RUNNER_VERSION': '11.6.0', 'ROOT': '/local/builds/bob/bob.devtools/miniconda', 'click_plugins': '1.0.3', 'CI_SERVER_REVISION': '4d0c547e7d3', 'BUILD_EGG': 'true', 'HTTPS_PROXY': '', 'CI_COMMIT_SHA': '9913b0b178e743d9a4d74e6099dcdcda143069fc', 'BUILD_PREFIX': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_build_env', 'CI_BUILD_NAME': 'build_linux_36', 'CI_CONFIG_PATH': '.gitlab-ci.yml', 'HOME': '/root', 'SHLVL': '3', 'PKG_BUILD_STRING': 'py36h6b6bea4_0', 'CI_SERVER': 'yes', 'CI_PROJECT_PATH_SLUG': 'bob-bob-devtools', 'LANGUAGE': 'en_US:en', 'CI': 'true', 'CONDA_ALLOW_SOFTLINKS': 'false', 'HTTP_PROXY': '', 'SP_DIR': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages', 'CI_PROJECT_NAMESPACE': 'bob', 'SYS_PREFIX': '/local/builds/bob/bob.devtools/miniconda', 'CONDA_PYTHON_EXE': '/local/builds/bob/bob.devtools/miniconda/bin/python', 'SUBDIR': 'linux-64', 'CONDA_NPY': '111', 'sphinx_rtd_theme': '0.4.1', 'PREFIX': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho', 'NPY_DISTUTILS_APPEND_FLAGS': '1', 'R_VER': '3.5', 'CONDA_LUA': '5', 'RECIPE_DIR': '/tmp/tmp_d4gwp26/info/recipe', 'CONDA_DEFAULT_ENV': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho', 'CI_BUILD_REF_SLUG': 'v0-1-1', 'PKG_CONFIG_PATH': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/pkgconfig', 'ARCH': '64', 'BOB_PACKAGE_VERSION': '0.1.1', 'CI_RUNNER_DESCRIPTION': 'docker', 'GITLAB_USER_ID': '22', 'DOCUSER': 'uploader', 'CI_JOB_ID': '155413', 'SYS_PYTHON': '/local/builds/bob/bob.devtools/miniconda/bin/python3', 'PKG_NAME': 'bob.devtools', 'setuptools': '40.2.0', 'CI_COMMIT_TAG': 'v0.1.1', 'PKG_HASH': 'h6b6bea4', 'CI_JOB_NAME': 'build_linux_36', 'PYTHON': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/python', '_': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/sphinx-build', 'DOCUTILSCONFIG': '/local/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1548152250852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/share/doc/bob.devtools/doc/docutils.conf'}))

Function to setup the environment variable and print debug message

Parameters:

• name – The name of the environment variable to set
• value – The value to set the environment variable to
• env – Optional environment (dictionary) where to set the variable at

bob.devtools.bootstrap.human_time(seconds, granularity=2)

Returns a human readable time string like “1 day, 2 hours”

bob.devtools.bootstrap.run_cmdline(cmd, env=None)

Runs a command on a environment, logs output and reports status

Parameters:

• cmd (list) – The command to run, with parameters separated on a list
• env (dict, Optional) – Environment to use for running the program on. If not set, use os.environ.

bob.devtools.bootstrap.touch(path)

Python-implementation of the “touch” command-line application

bob.devtools.bootstrap.merge_conda_cache(cache, prefix, name)

Merges conda pkg caches and conda-bld folders

Parameters:

• cache – The cached directory (from previous builds)
• prefix – The current prefix (root of conda installation)
• name – The name of the current package

bob.devtools.bootstrap.get_miniconda_sh()

Retrieves the miniconda3 installer for the current system

bob.devtools.bootstrap.install_miniconda(prefix, name)

Creates a new miniconda installation

Parameters:

• prefix – The path leading to the (new) root of the miniconda installation
• name – The name of this package

bob.devtools.bootstrap.get_channels(public, stable, server, intranet)

Returns the relevant conda channels to consider if building project

The subset of channels to be returned depends on the visibility and stability of the package being built. Here are the rules:

• public and stable: only returns the public stable channel(s)
• public and not stable: returns both public stable and beta channels
• not public and stable: returns both public and private stable channels
• not public and not stable: returns all channels

Beta channels have priority over stable channels, if returned. Private channels have priority over public channles, if turned.

Parameters:

• public – Boolean indicating if we’re supposed to include only public channels
• stable – Boolean indicating if we’re supposed to include only stable channels
• server – The base address of the server containing our conda channels
• intranet – Boolean indicating if we should add “private”/”public” prefixes on the conda paths

Returns: a list of channels that need to be considered.

bob.devtools.bootstrap.setup_logger(logger, level)

Sets-up the logging for this command at level INFO

Tools for self-building and other utilities

bob.devtools.build.conda_arch()

Returns the current OS name and architecture as recognized by conda

bob.devtools.build.should_skip_build(metadata_tuples)

Takes the output of render_recipe as input and evaluates if this recipe’s build should be skipped.

bob.devtools.build.next_build_number(channel_url, name, version, python)

Calculates the next build number of a package given the channel

This function returns the next build number (integer) for a package given its recipe, dependencies, name, version and python version. It looks on the channel URL provided and figures out if any clash would happen and what would be the highest build number available for that configuration.

Parameters:

• channel_url – The URL where to look for packages clashes (normally a beta channel)
• name – The name of the package
• version – The version of the package
• python – The version of python as 2 digits (e.g.: “2.7” or “3.6”)

Returns: The next build number with the current configuration. Zero (0) is returned if no match is found. Also returns the URLs of the packages it finds with matches on the name, version and python-version.

bob.devtools.build.make_conda_config(config, python, append_file, condarc_options)

Creates a conda configuration for a build merging various sources

This function will use the conda-build API to construct a configuration by merging different sources of information.

Parameters:

• config – Path leading to the conda_build_config.yaml to use
• python – The version of python to use for the build as x.y (e.g. 3.6)
• append_file – Path leading to the recipe_append.yaml file to use
• condarc_options – A dictionary (typically read from a condarc YAML file) that contains build and channel options

Returns: A dictionary containing the merged configuration, as produced by conda-build API’s get_or_merge_config() function.

bob.devtools.build.get_rendered_metadata(recipe_dir, config)

Renders the recipe and returns the interpreted YAML file

bob.devtools.build.get_parsed_recipe(metadata)

Renders the recipe and returns the interpreted YAML file

bob.devtools.build.remove_pins(deps)

bob.devtools.build.parse_dependencies(recipe_dir, config)

bob.devtools.build.get_env_directory(conda, name)

Get the directory of a particular conda environment or fail silently

bob.devtools.build.conda_create(conda, name, overwrite, condarc, packages, dry_run, use_local)

Creates a new conda environment following package specifications

This command can create a new conda environment following the list of input packages. It will overwrite an existing environment if indicated.

Parameters:

• conda – path to the main conda executable of the installation
• name – the name of the environment to create or overwrite
• overwrite – if set to `True, overwrite potentially existing environments with the same name
• condarc – a dictionary of options for conda, including channel urls
• packages – the package list specification
• dry_run – if set, then don’t execute anything, just print stuff
• use_local – include the local conda-bld directory as a possible installation channel (useful for testing multiple interdependent recipes that are built locally)

bob.devtools.build.get_docserver_setup(public, stable, server, intranet)

Returns a setup for BOB_DOCUMENTATION_SERVER

What is available to build the documentation depends on the setup of public and stable:

• public and stable: only returns the public stable channel(s)
• public and not stable: returns both public stable and beta channels
• not public and stable: returns both public and private stable channels
• not public and not stable: returns all channels

Beta channels have priority over stable channels, if returned. Private channels have priority over public channles, if turned.

Parameters:

• public – Boolean indicating if we’re supposed to include only public channels
• stable – Boolean indicating if we’re supposed to include only stable channels
• server – The base address of the server containing our conda channels
• intranet – Boolean indicating if we should add “private”/”public” prefixes on the returned paths

Returns: a string to be used by bob.extension to find dependent documentation projects.

bob.devtools.build.check_version(workdir, envtag)

Checks if the version being built and the value reported match

This method will read the contents of the file version.txt and compare it to the potentially set envtag (may be None). If the value of envtag is different than None, ensure it matches the value in version.txt or raises an exception.

Parameters:

• workdir – The work directory where the repo of the package being built was checked-out
• envtag – (optional) tag provided by the environment

Returns: A tuple with the version of the package that we’re currently building and a boolean flag indicating if the version number represents a pre-release or a stable release.

bob.devtools.build.git_clean_build(runner, verbose)

Runs git-clean to clean-up build products

Parameters:

• runner – A pointer to the run_cmdline() function
• verbose – A boolean flag indicating if the git command should report erased files or not

WebDAV Python Client

bob.devtools.webdav3.client.listdir(directory)

Returns list of nested files and directories for local directory by path

Parameters:directory – absolute or relative path to local directory Returns:list nested of file or directory names

bob.devtools.webdav3.client.get_options(option_type, from_options)

Extract options for specified option type from all options

Parameters:

• option_type – the object of specified type of options
• from_options – all options dictionary

Returns:

the dictionary of options for specified type, each option can be filled by value from all options dictionary or blank in case the option for specified type is not exist in all options dictionary

bob.devtools.webdav3.client.wrap_connection_error(fn)

class bob.devtools.webdav3.client.Client(options)

Bases: object

The client for WebDAV servers provides an ability to control files on remote WebDAV server.

root = '/'

large_size = 2147483648

timeout = 30

http_header = {'check': ['Accept: */*'], 'clean': ['Accept: */*', 'Connection: Keep-Alive'], 'copy': ['Accept: */*'], 'free': ['Accept: */*', 'Depth: 0', 'Content-Type: text/xml'], 'get_property': ['Accept: */*', 'Depth: 1', 'Content-Type: application/x-www-form-urlencoded'], 'info': ['Accept: */*', 'Depth: 1'], 'list': ['Accept: */*', 'Depth: 1'], 'mkdir': ['Accept: */*', 'Connection: Keep-Alive'], 'move': ['Accept: */*'], 'set_property': ['Accept: */*', 'Depth: 1', 'Content-Type: application/x-www-form-urlencoded']}

get_headers(action, headers_ext=None)

Returns HTTP headers of specified WebDAV actions.

Parameters:

• action – the identifier of action.
• headers_ext – (optional) the addition headers list witch sgould be added to basic HTTP headers for the specified action.

Returns:

the dictionary of headers for specified action.

get_url(path)

Generates url by uri path.

Parameters:path – uri path. Returns:the url string.

get_full_path(urn)

Generates full path to remote resource exclude hostname.

Parameters:urn – the URN to resource. Returns:full path to resource with root path.

execute_request(action, path, data=None, headers_ext=None)

Generate request to WebDAV server for specified action and path and execute it.

Parameters:

• action – the action for WebDAV server which should be executed.
• path – the path to resource for action
• data – (optional) Dictionary or list of tuples [(key, value)] (will be form-encoded), bytes, or file-like object to send in the body of the Request.
• headers_ext – (optional) the addition headers list witch should be added to basic HTTP headers for the specified action.

Returns:

HTTP response of request.

requests = {'check': 'HEAD', 'clean': 'DELETE', 'copy': 'COPY', 'download': 'GET', 'free': 'PROPFIND', 'get_property': 'PROPFIND', 'info': 'PROPFIND', 'list': 'PROPFIND', 'mkdir': 'MKCOL', 'move': 'MOVE', 'publish': 'PROPPATCH', 'published': 'PROPPATCH', 'set_property': 'PROPPATCH', 'unpublish': 'PROPPATCH', 'upload': 'PUT'}

meta_xmlns = {'https://webdav.yandex.ru': 'urn:yandex:disk:meta'}

valid()

Validates of WebDAV and proxy settings.

Returns:True in case settings are valid and False otherwise.

list(remote_path='/')

Returns list of nested files and directories for remote WebDAV directory by path. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND

Parameters:remote_path – path to remote directory. Returns:list of nested file or directory names.

free()

Returns an amount of free space on remote WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND

Returns:an amount of free space in bytes.

check(remote_path='/')

Checks an existence of remote resource on WebDAV server by remote path. More information you can find by link http://webdav.org/specs/rfc4918.html#rfc.section.9.4

Parameters:remote_path – (optional) path to resource on WebDAV server. Defaults is root directory of WebDAV. Returns:True if resource is exist or False otherwise

mkdir(remote_path)

Makes new directory on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_MKCOL

Parameters:remote_path – path to directory Returns:True if request executed with code 200 or 201 and False otherwise.

download_from(buff, remote_path)

Downloads file from WebDAV and writes it in buffer.

Parameters:

• buff – buffer object for writing of downloaded file content.
• remote_path – path to file on WebDAV server.

download(remote_path, local_path, progress=None)

Downloads remote resource from WebDAV and save it in local path. More information you can find by link http://webdav.org/specs/rfc4918.html#rfc.section.9.4

Parameters:

• remote_path – the path to remote resource for downloading can be file and directory.
• local_path – the path to save resource locally.
• progress – progress function. Not supported now.

download_directory(remote_path, local_path, progress=None)

Downloads directory and downloads all nested files and directories from remote WebDAV to local. If there is something on local path it deletes directories and files then creates new.

Parameters:

• remote_path – the path to directory for downloading form WebDAV server.
• local_path – the path to local directory for saving downloaded files and directories.
• progress – Progress function. Not supported now.

download_file(remote_path, local_path, progress=None)

Downloads file from WebDAV server and save it locally. More information you can find by link http://webdav.org/specs/rfc4918.html#rfc.section.9.4

Parameters:

• remote_path – the path to remote file for downloading.
• local_path – the path to save file locally.
• progress – progress function. Not supported now.

download_sync(remote_path, local_path, callback=None)

Downloads remote resources from WebDAV server synchronously.

Parameters:

• remote_path – the path to remote resource on WebDAV server. Can be file and directory.
• local_path – the path to save resource locally.
• callback – the callback which will be invoked when downloading is complete.

download_async(remote_path, local_path, callback=None)

Downloads remote resources from WebDAV server asynchronously

Parameters:

• remote_path – the path to remote resource on WebDAV server. Can be file and directory.
• local_path – the path to save resource locally.
• callback – the callback which will be invoked when downloading is complete.

upload_to(buff, remote_path)

Uploads file from buffer to remote path on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PUT

Parameters:

• buff – the buffer with content for file.
• remote_path – the path to save file remotely on WebDAV server.

upload(remote_path, local_path, progress=None)

Uploads resource to remote path on WebDAV server. In case resource is directory it will upload all nested files and directories. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PUT

Parameters:

• remote_path – the path for uploading resources on WebDAV server. Can be file and directory.
• local_path – the path to local resource for uploading.
• progress – Progress function. Not supported now.

upload_directory(remote_path, local_path, progress=None)

Uploads directory to remote path on WebDAV server. In case directory is exist on remote server it will delete it and then upload directory with nested files and directories.

Parameters:

• remote_path – the path to directory for uploading on WebDAV server.
• local_path – the path to local directory for uploading.
• progress – Progress function. Not supported now.

upload_file(remote_path, local_path, progress=None)

Uploads file to remote path on WebDAV server. File should be 2Gb or less. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PUT

Parameters:

• remote_path – the path to uploading file on WebDAV server.
• local_path – the path to local file for uploading.
• progress – Progress function. Not supported now.

upload_sync(remote_path, local_path, callback=None)

Uploads resource to remote path on WebDAV server synchronously. In case resource is directory it will upload all nested files and directories.

Parameters:

• remote_path – the path for uploading resources on WebDAV server. Can be file and directory.
• local_path – the path to local resource for uploading.
• callback – the callback which will be invoked when downloading is complete.

upload_async(remote_path, local_path, callback=None)

Uploads resource to remote path on WebDAV server asynchronously. In case resource is directory it will upload all nested files and directories.

Parameters:

• remote_path – the path for uploading resources on WebDAV server. Can be file and directory.
• local_path – the path to local resource for uploading.
• callback – the callback which will be invoked when downloading is complete.

copy(remote_path_from, remote_path_to)

Copies resource from one place to another on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_COPY

Parameters:

• remote_path_from – the path to resource which will be copied,
• remote_path_to – the path where resource will be copied.

move(remote_path_from, remote_path_to, overwrite=False)

Moves resource from one place to another on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_MOVE

Parameters:

• remote_path_from – the path to resource which will be moved,
• remote_path_to – the path where resource will be moved.
• overwrite – (optional) the flag, overwrite file if it exists. Defaults is False

clean(remote_path)

Cleans (Deletes) a remote resource on WebDAV server. The name of method is not changed for back compatibility with original library. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_DELETE

Parameters:remote_path – the remote resource whisch will be deleted.

info(remote_path)

Gets information about resource on WebDAV. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND

Parameters:remote_path – the path to remote resource. Returns:a dictionary of information attributes and them values with following keys: created: date of resource creation, name: name of resource, size: size of resource, modified: date of resource modification.

is_dir(remote_path)

Checks is the remote resource directory. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND

Parameters:remote_path – the path to remote resource. Returns:True in case the remote resource is directory and False otherwise.

get_property(remote_path, option)

Gets metadata property of remote resource on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND

Parameters:

• remote_path – the path to remote resource.
• option – the property attribute as dictionary with following keys: namespace: (optional) the namespace for XML property which will be set, name: the name of property which will be set.

Returns:

the value of property or None if property is not found.

set_property(remote_path, option)

Sets metadata property of remote resource on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPPATCH

Parameters:

• remote_path – the path to remote resource.
• option – the property attribute as dictionary with following keys: namespace: (optional) the namespace for XML property which will be set, name: the name of property which will be set, value: (optional) the value of property which will be set. Defaults is empty string.

set_property_batch(remote_path, option)

Sets batch metadata properties of remote resource on WebDAV server in batch. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPPATCH

Parameters:

• remote_path – the path to remote resource.
• option – the property attributes as list of dictionaries with following keys: namespace: (optional) the namespace for XML property which will be set, name: the name of property which will be set, value: (optional) the value of property which will be set. Defaults is empty string.

resource(remote_path)

push(remote_directory, local_directory)

pull(remote_directory, local_directory)

sync(remote_directory, local_directory)

class bob.devtools.webdav3.client.Resource(client, urn)

Bases: object

is_dir()

rename(new_name)

move(remote_path)

copy(remote_path)

info(params=None)

clean()

check()

read_from(buff)

read(local_path)

read_async(local_path, callback=None)

write_to(buff)

write(local_path)

write_async(local_path, callback=None)

publish()

unpublish()

property

class bob.devtools.webdav3.client.WebDavXmlUtils

Bases: object

static parse_get_list_response(content)

Parses of response content XML from WebDAV server and extract file and directory names.

Parameters:content – the XML content of HTTP response from WebDAV server for getting list of files by remote path. Returns:list of extracted file or directory names.

static create_free_space_request_content()

Creates an XML for requesting of free space on remote WebDAV server.

Returns:the XML string of request content.

static parse_free_space_response(content, hostname)

Parses of response content XML from WebDAV server and extract an amount of free space.

Parameters:

• content – the XML content of HTTP response from WebDAV server for getting free space.
• hostname – the server hostname.

Returns:

an amount of free space in bytes.

static parse_info_response(content, path, hostname)

Parses of response content XML from WebDAV server and extract an information about resource.

Parameters:

• content – the XML content of HTTP response from WebDAV server.
• path – the path to resource.
• hostname – the server hostname.

Returns:

a dictionary of information attributes and them values with following keys: created: date of resource creation, name: name of resource, size: size of resource, modified: date of resource modification.

static parse_is_dir_response(content, path, hostname)

Parses of response content XML from WebDAV server and extract an information about resource.

Parameters:

• content – the XML content of HTTP response from WebDAV server.
• path – the path to resource.
• hostname – the server hostname.

Returns:

True in case the remote resource is directory and False otherwise.

static create_get_property_request_content(option)

Creates an XML for requesting of getting a property value of remote WebDAV resource.

Parameters:option – the property attributes as dictionary with following keys: namespace: (optional) the namespace for XML property which will be get, name: the name of property which will be get. Returns:the XML string of request content.

static parse_get_property_response(content, name)

Parses of response content XML from WebDAV server for getting metadata property value for some resource.

Parameters:

• content – the XML content of response as string.
• name – the name of property for finding a value in response

Returns:

the value of property if it has been found or None otherwise.

static create_set_property_batch_request_content(options)

Creates an XML for requesting of setting a property values for remote WebDAV resource in batch.

Parameters:options – the property attributes as list of dictionaries with following keys: namespace: (optional) the namespace for XML property which will be set, name: the name of property which will be set, value: (optional) the value of property which will be set. Defaults is empty string. Returns:the XML string of request content.

static etree_to_string(tree)

Creates string from lxml.etree.ElementTree with XML declaration and UTF-8 encoding.

Parameters:tree – the instance of ElementTree Returns:the string of XML.

static extract_response_for_path(content, path, hostname)

Extracts single response for specified remote resource.

Parameters:

• content – raw content of response as string.
• path – the path to needed remote resource.
• hostname – the server hostname.

Returns:

XML object of response for the remote resource defined by path.