Python API

This section includes information for using the Python API of bob.extension.

Summary

Core Functionality

bob.extension.boost([requirement])

A class for capturing configuration information from boost

bob.extension.build_ext(dist, **kw)

Compile the C++ :py:class`Library`’s using CMake, and the python extensions afterwards

bob.extension.check_packages(packages)

Checks if the requirements for the given packages are satisfied.

bob.extension.CMakeListsGenerator(name, …)

Generates a CMakeLists.txt file for the given sources, include directories and libraries.

bob.extension.construct_search_paths([…])

Constructs a list of candidate paths to search for.

bob.extension.DEFAULT_PREFIXES

The list common places to search for library-related files.

bob.extension.Extension(name, sources, **kwargs)

Extension building with pkg-config packages.

bob.extension.find_executable(name[, …])

Finds an executable on the file system.

bob.extension.find_library(name[, version, …])

Finds a library file on the file system.

bob.extension.generate_self_macros(extname, …)

Generates standard macros with library, module names and prefix

bob.extension.get_bob_libraries(bob_packages)

Returns a list of include directories, libraries and library directories for the given bob libraries.

bob.extension.get_config([package, …])

Returns a string containing the configuration information for the given package name.

bob.extension.get_full_libname(name[, path, …])

Generates the name of the library from the given name, path and version.

bob.extension.Library(name, sources, version)

A class to compile a pure C++ code library used within and outside an extension using CMake.

bob.extension.load_bob_library(name, _file_)

Loads the bob Library for the given package name in the given version (if given).

bob.extension.normalize_requirements(…)

Normalizes the requirements keeping only the most tight

bob.extension.pkgconfig(name[, paths])

A class for capturing configuration information from pkg-config

bob.extension.rc

The content of the global configuration file loaded as a dictionary.

bob.extension.rc_context(dict)

A context manager for bob.extension.rc.

bob.extension.reorganize_isystem(args)

Re-organizes the -isystem includes so that more specific paths come first

bob.extension.uniq(seq[, idfun])

Very fast, order preserving uniq function

bob.extension.uniq_paths(seq)

Uniq’fy a list of paths taking into consideration their real paths

bob.extension.download.get_file(filename, urls)

Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in ~/bob_data by default. You may change the location of this folder using::.

Utilities

bob.extension.utils.egrep(filename, expression)

Runs grep for a given expression on each line of the file

bob.extension.utils.find_file(name[, …])

Finds a generic file on the file system.

bob.extension.utils.find_header(name[, …])

Finds a header file on the file system.

bob.extension.utils.find_packages([directories])

This function replaces the find_packages command from setuptools to search for packages only in the given directories.

bob.extension.utils.link_documentation([…])

Generates a list of documented packages on our documentation server for the packages read from the “requirements.txt” file and the given list of additional packages.

bob.extension.utils.load_requirements([f])

Loads the contents of requirements.txt on the given path.

bob.extension.download.get_file(filename, urls)

Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in ~/bob_data by default. You may change the location of this folder using::.

bob.extension.download.search_file(…)

Search for files either in a file structure, or in a tarball.

bob.extension.download.list_dir(base_path[, …])

Lists the files and folders inside a folder or a tarball.

Configuration

bob.extension.rc_config.ENVNAME

Name of environment variable to look for an alternative for the RC file

bob.extension.rc_config.RCFILENAME

Default name to be used for the RC file to load

bob.extension.config.load(paths[, context, …])

Loads a set of configuration files, in sequence

Scripts

bob.extension.scripts.click_helper.ConfigCommand(…)

A click.Command that can take options both form command line options and configuration files.

bob.extension.scripts.click_helper.ResourceOption([…])

An extended click.Option that automatically loads resources from config files.

bob.extension.scripts.click_helper.verbosity_option(…)

Adds a -v/–verbose option to a click command.

bob.extension.scripts.click_helper.bool_option(…)

Generic provider for boolean options

bob.extension.scripts.click_helper.list_float_option(…)

Get option to get a list of float f

bob.extension.scripts.click_helper.open_file_mode_option(…)

Get open mode file option

bob.extension.scripts.click_helper.AliasedGroup([…])

Class that handles prefix aliasing for commands

bob.extension.scripts.click_helper.log_parameters(…)

Logs the click parameters with the logging module.

bob.extension.scripts.click_helper.assert_click_runner_result(result)

Helper for asserting click runner results

Core Functionality

A custom build class for Pkg-config based extensions

class bob.extension.CMakeListsGenerator(name, sources, target_directory, version='1.0.0', include_directories=[], system_include_directories=[], libraries=[], library_directories=[], macros=[])[source]

Bases: object

Generates a CMakeLists.txt file for the given sources, include directories and libraries.

generate(source_directory, build_directory)[source]

Generates the CMakeLists.txt file in the given directory.

bob.extension.DEFAULT_PREFIXES = ['/opt/local', '/usr/local', '/usr']

The list common places to search for library-related files.

bob.extension.DistutilsExtension

alias of setuptools.extension.Extension

class bob.extension.Extension(name, sources, **kwargs)[source]

Bases: setuptools.extension.Extension

Extension building with pkg-config packages.

See the documentation for distutils.extension.Extension for more details on input parameters.

class bob.extension.Library(name, sources, version, bob_packages=[], packages=[], boost_modules=[], include_dirs=[], system_include_dirs=[], libraries=[], library_dirs=[], define_macros=[])[source]

Bases: bob.extension.Extension

A class to compile a pure C++ code library used within and outside an extension using CMake.

compile(build_directory, compiler=None, stdout=None)[source]

This function will automatically create a CMakeLists.txt file in the package_directory including the required information. Afterwards, the library is built using CMake in the given build_directory. The build type is automatically taken from the debug option in the buildout.cfg. To change the compiler, use the compiler parameter.

class bob.extension.boost(requirement='')[source]

Bases: object

A class for capturing configuration information from boost

Example usage:

>>> from bob.extension import boost
>>> pkg = boost('>= 1.35')
>>> pkg.include_directory
'...'
>>> pkg.version
'...'

You can also use this class to retrieve information about installed Boost libraries and link information:

>>> from bob.extension import boost
>>> pkg = boost('>= 1.35')
>>> pkg.libconfig(['python', 'system'])
(...)
libconfig(modules, only_static=False, templates=['boost_%(name)s-mt-%(py)s', 'boost_%(name)s-%(py)s', 'boost_%(name)s-mt', 'boost_%(name)s'])[source]

Returns a tuple containing the library configuration for requested modules.

This function respects the path location where the include files for Boost are installed.

Parameters:

modules (list of strings)

A list of string specifying the requested libraries to search for. For example, to search for libboost_mpi.so, pass only mpi.

static (bool)

A boolean, indicating if we should try only to search for static versions of the libraries. If not set, any would do.

templates (list of template strings)

A list that defines in which order to search for libraries on the default search path, defined by self.include_directory. Tune this list if you have compiled specific versions of Boost with support to multi-threading (-mt), debug (-g), STLPORT (-p) or required to insert compiler, the underlying thread API used or your own namespace.

Here are the keywords you can use:

%(name)s

resolves to the module name you are searching for

%(ver)s

resolves to the current boost version string (e.g. '1.50.0')

%(py)s

resolves to the string 'pyXY' where XY represent the major and minor versions of the current python interpreter.

Example templates:

  • 'boost_%(name)s-mt'

  • 'boost_%(name)s'

  • 'boost_%(name)s-gcc43-%(ver)s'

Returns:

directories (list of strings)

A list of directories indicating where the libraries are installed

libs (list of strings)

A list of strings indicating the names of the libraries you can use

macros()[source]

Returns package availability and version number macros

This method returns a python list with 2 macros indicating package availability and a version number, using standard GNU compatible names. Example:

>>> from bob.extension import boost
>>> pkg = boost('>= 1.34')
>>> pkg.macros()
[('HAVE_BOOST', '1')]
class bob.extension.build_ext(dist, **kw)[source]

Bases: setuptools.command.build_ext.build_ext

Compile the C++ :py:class`Library`’s using CMake, and the python extensions afterwards

See the documentation for distutils.command.build_ext for more information.

finalize_options()[source]

Set final values for all the options that this command supports. This is always called as late as possible, ie. after any option assignments from the command-line or from other commands have been done. Thus, this is the place to code option dependencies: if ‘foo’ depends on ‘bar’, then it is safe to set ‘foo’ from ‘bar’ as long as ‘foo’ still has the same value it was assigned in ‘initialize_options()’.

This method must be implemented by all command classes.

run()[source]

Iterates through the list of Extension packages and reorders them, so that the Library’s come first

build_extension(ext)[source]

Builds the given extension.

When the extension is of type Library, it compiles the library with CMake, otherwise the default compilation mechanism is used. Afterwards, it adds the according library, and the include and library directories of the Library’s, so that other Extensions can find the newly generated lib.

get_ext_filename(fullname)[source]

Returns the library path for the given name

bob.extension.check_packages(packages)[source]

Checks if the requirements for the given packages are satisfied.

:param list of str: Each representing a requirement that

must be statistfied. Package requirements can be set like this:

"pkg > VERSION"

In this case, the package version should be greater than the given version number. Comparisons are done using distutils.version.LooseVersion. You can use other comparators such as <, <=, >= or ==. If no version number is given, then we only require that the package is installed.

Raises

RuntimeError – in case requirements are not satisfied. This means either not finding a package if no version number is specified or verifying that the package version does not match the required version by the builder.

bob.extension.construct_search_paths(prefixes=None, subpaths=None, suffix=None)[source]

Constructs a list of candidate paths to search for.

The list of paths is constructed using the following order of priority:

  1. BOB_PREFIX_PATH environment variable, if set. BOB_PREFIX_PATH can contain several paths divided by os.pathsep.

  2. The paths provided with the prefixes parameter.

  3. The current python executable prefix.

  4. The CONDA_PREFIX environment variable, if set.

  5. DEFAULT_PREFIXES.

Parameters
  • prefixes ([str], optional) – The list of paths to be added to the results.

  • subpaths ([str], optional) – A list of subpaths to be appended to each path at the end. For example, if you specify ['foo', 'bar'] for this parameter, then os.path.join(paths[0], 'foo'), os.path.join(paths[0], 'bar'), and so on are added to the returned paths. Globs are accepted in this list and resolved using the function glob.glob().

  • suffix (str, optional) – suffix will be appended to all paths except prefixes.

Returns

paths – A list of unique and existing paths to be used in your search.

Return type

[str]

bob.extension.find_executable(name, subpaths=None, prefixes=None)[source]

Finds an executable on the file system. Returns all candidates.

This method will find all occurrences of a given name on the file system and will return them to the user. It uses construct_search_paths to construct the candidate folders that the executable may exist in accounting automatically for typical executable folder names.

Parameters
Returns

A list of filenames that exist on the filesystem, matching your description.

Return type

[str]

bob.extension.find_library(name, version=None, subpaths=None, prefixes=None, only_static=False)[source]

Finds a library file on the file system. Returns all candidates.

This method will find all occurrences of a given name on the file system and will return them to the user. It uses construct_search_paths to construct the candidate folders that the library may exist in accounting automatically for typical library folder names.

Parameters
  • name (str) – The name of the module to be found. If you’d like to find libz.so, for example, specify "z". For libmath.so, specify "math".

  • version (str, optional) – The version of the library we are searching for. If not specified, then look only for the default names, such as libz.so and the such.

  • subpaths (str, optional) – See construct_search_paths

  • subpaths – See construct_search_paths

  • only_static (bool, optional) – A boolean, indicating if we should try only to search for static versions of the libraries. If not set, any would do.

Returns

A list of filenames that exist on the filesystem, matching your description.

Return type

[str]

bob.extension.find_system_include_paths()[source]

Finds system include paths if the environment variable CC is set

Returns

A list of include paths defined by the compiler

Return type

list

bob.extension.generate_self_macros(extname, version)[source]

Generates standard macros with library, module names and prefix

bob.extension.get_bob_libraries(bob_packages)[source]

Returns a list of include directories, libraries and library directories for the given bob libraries.

bob.extension.get_config(package='bob.extension', externals=None, api_version=None)[source]

Returns a string containing the configuration information for the given package name. By default, it returns the configuration of this package.

This function can be reused by other packages. If these packages have external C or C++ dependencies, the externals dictionary can be specified. Also, if the package provides an API version, it can be specified.

Keyword parameters:

packagestr

The name of the package to get the configuration information from. Usually, the __name__ of the package.

externals{dep: description}

A dictionary of external C or C++ dependencies, with the dep``endent package name as key, and a free ``description as value.

api_versionint (usually in hexadecimal)

The API version of the package, if any.

bob.extension.get_full_libname(name, path=None, version=None)[source]

Generates the name of the library from the given name, path and version.

bob.extension.load_bob_library(name, _file_)[source]

Loads the bob Library for the given package name in the given version (if given). The _file_ parameter is expected to be the __file__ member of the main __init__.py of the package. It is used to determine the directory, where the library should be loaded from.

Keyword parameters

namestring

The name of the bob package to load the library from, e.g. bob.core

_file_string

The __file__ member of the __init__.py file in which the library is loaded.

bob.extension.normalize_requirements(requirements)[source]

Normalizes the requirements keeping only the most tight

class bob.extension.pkgconfig(name, paths=None)[source]

Bases: object

A class for capturing configuration information from pkg-config

Example usage:

>>> from bob.extension import pkgconfig
>>> blitz = pkgconfig('blitz')
>>> blitz.include_directories()
[...]
>>> blitz.library_directories()
[...]

If the package does not exist, a RuntimeError is raised. All calls to any methods of a pkgconfig object are translated into a subprocess call that queries for that specific information. If pkg-config fails, a RuntimeError is raised.

include_directories()[source]

Returns a pre-processed list containing include directories.

Equivalent command line version:

$ PKG_CONFIG_PATH=<paths> pkg-config --cflags-only-I <name>
cflags_other()[source]

Returns a pre-processed dictionary containing compilation options.

Equivalent command line version:

$ PKG_CONFIG_PATH=<paths> pkg-config --cflags-only-other <name>

The returned dictionary contains two entries extra_compile_args and define_macros. The define_macros entries are ready for deployment in the setup() function of your package.

libraries()[source]

Returns a pre-processed list containing libraries to link against

Equivalent command line version:

$ PKG_CONFIG_PATH=<paths> pkg-config --libs-only-l <name>
other_libraries()[source]

Returns a pre-processed list containing libraries to link against

Equivalent command line version:

$ PKG_CONFIG_PATH=<paths> pkg-config --libs-only-other <name>
library_directories()[source]

Returns a pre-processed list containing library directories.

Equivalent command line version:

$ PKG_CONFIG_PATH=<paths> pkg-config --libs-only-L <name>

Returns a pre-processed list containing extra link arguments.

Equivalent command line version:

$ PKG_CONFIG_PATH=<paths> pkg-config --libs-only-other <name>
variable_names()[source]

Returns a list with all variable names know to this package

Equivalent command line version:

$ PKG_CONFIG_PATH=<paths> pkg-config --print-variables <name>
variable(name)[source]

Returns a variable with a specific name (if it exists)

Equivalent command line version:

$ PKG_CONFIG_PATH=<paths> pkg-config --variable=<variable-name> <name>

Warning

If a variable does not exist in a package, pkg-config does not signal an error. Instead, it returns an empty string. So, do we.

package_macros()[source]

Returns package availability and version number macros

This method returns a python list with 1 macro indicating package availability, using standard GNU compatible names. For example, if the package is named blitz, this command would return:

>>> from bob.extension import pkgconfig
>>> blitz = pkgconfig('blitz')
>>> blitz.package_macros()
[('HAVE_BLITZ', '1')]
bob.extension.rc = {'bob.db.replaymobile.directory': None, 'bob.db.youtube.directory': None}

The content of the global configuration file loaded as a dictionary. The value for any non-existing key is None.

bob.extension.rc_context(dict)[source]

A context manager for bob.extension.rc. You can use this context manager to temporarily change a value in bob.extension.rc.

Example

>>> from bob.extension import rc, rc_context
>>> assert rc.get("non-existing-key") is None
>>> with rc_context({"non-existing-key": 1}):
...     a = rc.get("non-existing-key")
>>> a
1
bob.extension.reorganize_isystem(args)[source]

Re-organizes the -isystem includes so that more specific paths come first

bob.extension.resource_filename(package_or_requirement, resource_name)

Return a true filesystem path for specified resource

bob.extension.uniq(seq, idfun=None)[source]

Very fast, order preserving uniq function

bob.extension.uniq_paths(seq)[source]

Uniq’fy a list of paths taking into consideration their real paths

Utilities

General utilities for building extensions

bob.extension.utils.construct_search_paths(prefixes=None, subpaths=None, suffix=None)[source]

Constructs a list of candidate paths to search for.

The list of paths is constructed using the following order of priority:

  1. BOB_PREFIX_PATH environment variable, if set. BOB_PREFIX_PATH can contain several paths divided by os.pathsep.

  2. The paths provided with the prefixes parameter.

  3. The current python executable prefix.

  4. The CONDA_PREFIX environment variable, if set.

  5. DEFAULT_PREFIXES.

Parameters
  • prefixes ([str], optional) – The list of paths to be added to the results.

  • subpaths ([str], optional) – A list of subpaths to be appended to each path at the end. For example, if you specify ['foo', 'bar'] for this parameter, then os.path.join(paths[0], 'foo'), os.path.join(paths[0], 'bar'), and so on are added to the returned paths. Globs are accepted in this list and resolved using the function glob.glob().

  • suffix (str, optional) – suffix will be appended to all paths except prefixes.

Returns

paths – A list of unique and existing paths to be used in your search.

Return type

[str]

bob.extension.utils.find_file(name, subpaths=None, prefixes=None)[source]

Finds a generic file on the file system. Returns all occurrences.

This method will find all occurrences of a given name on the file system and will return them to the user. It uses construct_search_paths to construct the candidate folders that file may exist in.

Parameters
Returns

A list of filenames that exist on the filesystem, matching your description.

Return type

[str]

bob.extension.utils.find_header(name, subpaths=None, prefixes=None)[source]

Finds a header file on the file system. Returns all candidates.

This method will find all occurrences of a given name on the file system and will return them to the user. It uses construct_search_paths to construct the candidate folders that header may exist in accounting automatically for typical header folder names.

Parameters
Returns

A list of filenames that exist on the filesystem, matching your description.

Return type

[str]

bob.extension.utils.find_library(name, version=None, subpaths=None, prefixes=None, only_static=False)[source]

Finds a library file on the file system. Returns all candidates.

This method will find all occurrences of a given name on the file system and will return them to the user. It uses construct_search_paths to construct the candidate folders that the library may exist in accounting automatically for typical library folder names.

Parameters
  • name (str) – The name of the module to be found. If you’d like to find libz.so, for example, specify "z". For libmath.so, specify "math".

  • version (str, optional) – The version of the library we are searching for. If not specified, then look only for the default names, such as libz.so and the such.

  • subpaths (str, optional) – See construct_search_paths

  • subpaths – See construct_search_paths

  • only_static (bool, optional) – A boolean, indicating if we should try only to search for static versions of the libraries. If not set, any would do.

Returns

A list of filenames that exist on the filesystem, matching your description.

Return type

[str]

bob.extension.utils.find_executable(name, subpaths=None, prefixes=None)[source]

Finds an executable on the file system. Returns all candidates.

This method will find all occurrences of a given name on the file system and will return them to the user. It uses construct_search_paths to construct the candidate folders that the executable may exist in accounting automatically for typical executable folder names.

Parameters
Returns

A list of filenames that exist on the filesystem, matching your description.

Return type

[str]

bob.extension.utils.uniq(seq, idfun=None)[source]

Very fast, order preserving uniq function

bob.extension.utils.uniq_paths(seq)[source]

Uniq’fy a list of paths taking into consideration their real paths

bob.extension.utils.egrep(filename, expression)[source]

Runs grep for a given expression on each line of the file

Parameters:

filename, str

The name of the file to grep for the expression

expression

A regular expression, that will be initialized using re.compile().

Returns a list of re matches.

bob.extension.utils.load_requirements(f=None)[source]

Loads the contents of requirements.txt on the given path.

Defaults to “./requirements.txt”

bob.extension.utils.find_packages(directories=['bob'])[source]

This function replaces the find_packages command from setuptools to search for packages only in the given directories. Using this function will increase the building speed, especially when you have (links to) deep non-code-related directory structures inside your package directory. The given directories should be a list of top-level sub-directories of your package, where package code can be found. By default, it uses 'bob' as the only directory to search.

Generates a list of documented packages on our documentation server for the packages read from the “requirements.txt” file and the given list of additional packages.

Parameters:

additional_packages[str]

A list of additional bob packages for which the documentation urls are added. By default, ‘numpy’ is added

requirements_filestr or file-like

The file (relative to the documentation directory), where to read the requirements from. If None, it will be skipped.

serverstr or None

The url to the server which provides the documentation. If None (the default), the BOB_DOCUMENTATION_SERVER environment variable is taken if existent. If neither server is specified, nor a BOB_DOCUMENTATION_SERVER environment variable is set, the default "http://www.idiap.ch/software/bob/docs/bob/%(name)s/%(version)s/" is used.

bob.extension.download.extract_compressed_file(filename)[source]

Extracts a compressed file.

Parameters

filename (str) – Path to the .zip, .tar, .tar.*, .tgz, .tbz2, and .bz2 file

Raises

ValueError – If the extension of the file is not recognized.

bob.extension.download.download_file(url, out_file)[source]

Downloads a file from a given url

Parameters
  • url (str) – The url to download form.

  • out_file (str) – Where to save the file.

bob.extension.download.download_file_from_possible_urls(urls, out_file)[source]

Tries to download a file from a list of possible urls. The function stops as soon as one url works and raises an error when all urls fail.

Parameters
  • urls (list) – List of urls

  • out_file (str) – Path to save the file

Raises

RuntimeError – If downloading from all urls fails.

bob.extension.download.validate_file(fpath, file_hash, algorithm='auto', chunk_size=65535)[source]

Validates a file against a sha256 or md5 hash.

Parameters
  • fpath (str) – path to the file being validated

  • file_hash (str) – The expected hash string of the file. The sha256 and md5 hash algorithms are both supported.

  • algorithm (str) – Hash algorithm, one of ‘auto’, ‘sha256’, or ‘md5’. The default ‘auto’ detects the hash algorithm in use.

  • chunk_size (int) – Bytes to read at a time, important for large files.

Returns

Whether the file is valid

Return type

bool

bob.extension.download.get_file(filename, urls, cache_subdir='datasets', file_hash=None, hash_algorithm='auto', extract=False, force=False)[source]

Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in ~/bob_data by default. You may change the location of this folder using:

$ bob config set bob_data_folder /another/location/
Parameters
  • filename (str) – Name of the file

  • urls (list) – List containing the all the URLs. The function will try to download them in order and stops if it succeeds.

  • cache_subdir (str) – Subdirectory where the file is saved.

  • file_hash (str) – The expected hash string of the file after download. The sha256 and md5 hash algorithms are both supported.

  • hash_algorithm (str) – Select the hash algorithm to verify the file. options are ‘md5’, ‘sha256’, and ‘auto’. The default ‘auto’ detects the hash algorithm in use.

  • extract (bool) – If True, will extract the downloaded file.

  • force (bool) – If True, will download the file anyway if it already exists.

Returns

The path to the downloaded file.

Return type

str

Raises

ValueError – If the file_hash does not match the downloaded file

bob.extension.download.download_and_unzip(urls, filename)[source]

Download a file from a given URL list, save it somewhere and unzip/untar if necessary

Example:

download_and_unzip(
    ["https://mytesturl.co/my_file_example.tag.bz2"],
    filename="~/my_file_example.tag.bz2"
)
Parameters
  • urls (list) – List containing the all the URLs. The function will try to download them in order

  • filename (str) – File name (full path) where the downloaded file will be written and uncompressed

bob.extension.download.find_element_in_tarball(filename, target_path, open_as_stream=False)[source]

Search an element in a tarball.

Parameters
  • filename (str) – Tarball file name

  • target_path (str) – Target path to be searched inside of the tarball

  • open_as_stream (bool) – If True, will load the element from the tarball as a byte_stream. If False, will load as text

Returns

It returns an opened file

Return type

object

bob.extension.download.search_file(base_path, options)[source]

Search for files either in a file structure, or in a tarball.

Parameters
  • base_path (str) – Base path to start the search, or the tarball to be searched

  • options (list) – Files to be searched. This function will return the first occurency

Returns

It returns an opened file

Return type

object

bob.extension.download.list_dir(base_path, inner_folder='', folders=True, files=True)[source]

Lists the files and folders inside a folder or a tarball. To list an inner level folder (useful when base_path is a tarball), provide the inner_folder argument.

Parameters
  • base_path (str) – Path to a folder or a tarball

  • inner_folder (str) – Path to an inner folder inside base_path. If given, the folders inside this folder are listed.

  • folders (bool) – If False, will exclude folders from the results.

  • files (bool) – If False, will exclude files from the results.

Returns

Sorted list of file and directory names

Return type

list

Raises

ValueError – If base_path is not a folder or a tarball

Configuration

Implements a global configuration system for bob using json.

bob.extension.rc_config.ENVNAME = 'BOBRC'

Name of environment variable to look for an alternative for the RC file

bob.extension.rc_config.RCFILENAME = '~/.bobrc'

Default name to be used for the RC file to load

Functionality to implement python-based config file parsing and loading.

bob.extension.config.load(paths, context=None, entry_point_group=None, attribute_name=None)[source]

Loads a set of configuration files, in sequence

This method will load one or more configuration files. Every time a configuration file is loaded, the context (variables) loaded from the previous file is made available, so the new configuration file can override or modify this context.

Parameters
  • paths ([str]) – A list or iterable containing paths (relative or absolute) of configuration files that need to be loaded in sequence. Each configuration file is loaded by creating/modifying the context generated after each file readout.

  • context (dict, optional) – If provided, start the readout of the first configuration file with the given context. Otherwise, create a new internal context.

  • entry_point_group (str, optional) – If provided, it will treat non-existing file paths as entry point names under the entry_point_group name.

  • attribute_name (None or str) – If provided, will look for the attribute_name variable inside the loaded files. Paths ending with some_path:variable_name can override the attribute_name. The entry_point_group must provided as well attribute_name is not None.

Returns

mod – A module representing the resolved context, after loading the provided modules and resolving all variables. If attribute_name is given, the object with the attribute_name name (or the name provided by user) is returned instead of the module.

Return type

module or object

Raises
  • ImportError – If attribute_name is given but the object does not exist in the paths.

  • ValueError – If attribute_name is given but entry_point_group is not given.

bob.extension.config.mod_to_context(mod)[source]

Converts the loaded module of load to a dictionary context. This function removes all the variables that start and end with __.

Parameters

mod (object) – What is returned by load

Returns

The context that was in mod.

Return type

dict

bob.extension.config.resource_keys(entry_point_group, exclude_packages=[], strip=['dummy'])[source]

Reads and returns all resources that are registered with the given entry_point_group. Entry points from the given exclude_packages are ignored.

Parameters
  • entry_point_group (str) – The entry point group name.

  • exclude_packages (list, optional) – List of packages to exclude when finding resources.

  • strip (list, optional) – Entrypoint names that start with any value in strip will be ignored.

Returns

List of found resources.

Return type

list

Logging

Sets-up logging, centrally for Bob.

bob.extension.log.set_verbosity_level(logger, level)[source]

Sets the log level for the given logger.

Parameters
  • logger (logging.Logger or str) – The logger to generate logs for, or the name of the module to generate logs for.

  • level (int) – Possible log levels are: 0: Error; 1: Warning; 2: Info; 3: Debug.

Raises

ValueError – If the level is not in range(0, 4).

bob.extension.log.setup(logger_name, format='%(name)s@%(asctime)s -- %(levelname)s: %(message)s')[source]

This function returns a logger object that is set up to perform logging using Bob loggers.

Parameters
  • logger_name (str) – The name of the module to generate logs for

  • format (str, optional) – The format of the logs, see logging.LogRecord for more details. By default, the log contains the logger name, the log time, the log level and the massage.

Returns

logger – The logger configured for logging. The same logger can be retrieved using the logging.getLogger() function.

Return type

logging.Logger

Scripts

bob.extension.scripts.click_helper.bool_option(name, short_name, desc, dflt=False, **kwargs)[source]

Generic provider for boolean options

Parameters
  • name (str) – name of the option

  • short_name (str) – short name for the option

  • desc (str) – short description for the option

  • dflt (bool or None) – Default value

  • **kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

bob.extension.scripts.click_helper.list_float_option(name, short_name, desc, nitems=None, dflt=None, **kwargs)[source]

Get option to get a list of float f

Parameters
  • name (str) – name of the option

  • short_name (str) – short name for the option

  • desc (str) – short description for the option

  • nitems (obj:int, optional) – If given, the parsed list must contains this number of items.

  • dflt (list, optional) – List of default values for axes.

  • **kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

bob.extension.scripts.click_helper.open_file_mode_option(**kwargs)[source]

Get open mode file option

Parameters

**kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

bob.extension.scripts.click_helper.verbosity_option(**kwargs)[source]

Adds a -v/–verbose option to a click command.

Parameters

**kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

class bob.extension.scripts.click_helper.ConfigCommand(name, *args, help=None, entry_point_group=None, **kwargs)[source]

Bases: click.core.Command

A click.Command that can take options both form command line options and configuration files. In order to use this class, you have to use the ResourceOption class also.

config_argument_name

The name of the config argument.

Type

str

entry_point_group

The name of entry point that will be used to load the config files.

Type

str

dump_config(ctx, param, value)[source]

Generate configuration file from parameters and context

Parameters

ctx (object) – Click context

class bob.extension.scripts.click_helper.ResourceOption(param_decls=None, show_default=False, prompt=False, confirmation_prompt=False, hide_input=False, is_flag=None, flag_value=None, multiple=False, count=False, allow_from_autoenv=True, type=None, help=None, entry_point_group=None, required=False, string_exceptions=None, **kwargs)[source]

Bases: click.core.Option

An extended click.Option that automatically loads resources from config files.

This class comes with two different functionalities that are independent and could be combined:

  1. If used in commands that are inherited from ConfigCommand, it will lookup inside the config files (that are provided as argument to the command) to resolve its value. Values given explicitly in the command line take precedence.

  2. If entry_point_group is provided, it will treat values given to it (by any means) as resources to be loaded. Loading is done using load. See Resource Loading for more information. The final value cannot be a string.

You may use this class in three ways:

  1. Using this class (without using ConfigCommand) AND (providing entry_point_group).

  2. Using this class (with ConfigCommand) AND (providing entry_point_group).

  3. Using this class (with ConfigCommand) AND (without providing entry_point_group).

Using this class without ConfigCommand and without providing entry_point_group does nothing and is not allowed.

entry_point_group

If provided, the strings values to this option are assumed to be entry points from entry_point_group that need to be loaded.

Type

str or None

string_exceptions

If provided and entry_point_group is provided, the code will not treat strings in string_exceptions as entry points and does not try to load them.

Type

tuple or None

consume_value(ctx, opts)[source]
full_process_value(ctx, value)[source]
class bob.extension.scripts.click_helper.AliasedGroup(name=None, commands=None, **attrs)[source]

Bases: click.core.Group

Class that handles prefix aliasing for commands

Basically just implements get_command that is used by click to choose the command based on the name.

Example

To enable prefix aliasing of commands for a given group, just set cls=AliasedGroup parameter in click.group decorator.

get_command(ctx, cmd_name)[source]

get_command with prefix aliasing

bob.extension.scripts.click_helper.log_parameters(logger_handle, ignore=())[source]

Logs the click parameters with the logging module.

Parameters
  • logger_handle (object) – The logger handle to write debug information into.

  • ignore (tuple) – The keys in ignore will not be logged.

bob.extension.scripts.click_helper.assert_click_runner_result(result, exit_code=0, exception_type=None)[source]

Helper for asserting click runner results