Python API

This section includes information for using the Python API of bob.extension.

Summary

Core Functionality

bob.extension.get_config([package, ...])

Returns a string containing the configuration information for the given package name.

bob.extension.rc

The content of the global configuration file loaded as a dictionary.

bob.extension.rc_context(dict)

A context manager for bob.extension.rc.

bob.extension.download.get_file(filename, urls)

Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in ~/bob_data by default. You may change the location of this folder using::.

Utilities

bob.extension.utils.find_packages([directories])

This function replaces the find_packages command from setuptools to search for packages only in the given directories.

bob.extension.utils.link_documentation([...])

Generates a list of documented packages on our documentation server for the packages read from the "requirements.txt" file and the given list of additional packages.

bob.extension.utils.load_requirements([f])

Loads the contents of requirements.txt on the given path.

bob.extension.download.get_file(filename, urls)

Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in ~/bob_data by default. You may change the location of this folder using::.

bob.extension.download.search_file(...)

Search for files either in a file structure, or in a tarball.

bob.extension.download.list_dir(base_path[, ...])

Lists the files and folders inside a folder or a tarball.

Configuration

bob.extension.rc_config.ENVNAME

Name of environment variable to look for an alternative for the RC file

bob.extension.rc_config.RCFILENAME

Default name to be used for the RC file to load

bob.extension.config.load(paths[, context, ...])

Loads a set of configuration files, in sequence

Scripts

bob.extension.scripts.click_helper.ConfigCommand(...)

A click.Command that can take options both form command line options and configuration files.

bob.extension.scripts.click_helper.ResourceOption([...])

An extended click.Option that automatically loads resources from config files.

bob.extension.scripts.click_helper.verbosity_option(...)

Adds a -v/--verbose option to a click command.

bob.extension.scripts.click_helper.bool_option(...)

Generic provider for boolean options

bob.extension.scripts.click_helper.list_float_option(...)

Get option to get a list of float f

bob.extension.scripts.click_helper.open_file_mode_option(...)

Get open mode file option

bob.extension.scripts.click_helper.AliasedGroup([...])

Class that handles prefix aliasing for commands

bob.extension.scripts.click_helper.log_parameters(...)

Logs the click parameters with the logging module.

bob.extension.scripts.click_helper.assert_click_runner_result(result)

Helper for asserting click runner results

Core Functionality

A custom build class for Pkg-config based extensions

bob.extension.get_config(package='bob.extension', externals=None, api_version=None)[source]

Returns a string containing the configuration information for the given package name. By default, it returns the configuration of this package.

This function can be reused by other packages. If these packages have external C or C++ dependencies, the externals dictionary can be specified. Also, if the package provides an API version, it can be specified.

Keyword parameters:

packagestr

The name of the package to get the configuration information from. Usually, the __name__ of the package.

externals{dep: description}

A dictionary of external C or C++ dependencies, with the dep``endent package name as key, and a free ``description as value.

api_versionint (usually in hexadecimal)

The API version of the package, if any.

bob.extension.rc = {'bob_data_folder': './bob_data'}

The content of the global configuration file loaded as a dictionary. The value for any non-existing key is None.

bob.extension.rc_context(dict)[source]

A context manager for bob.extension.rc. You can use this context manager to temporarily change a value in bob.extension.rc.

Example

>>> from bob.extension import rc, rc_context
>>> assert rc.get("non-existing-key") is None
>>> with rc_context({"non-existing-key": 1}):
...     a = rc.get("non-existing-key")
>>> a
1

Utilities

General utilities for building extensions

bob.extension.utils.load_requirements(f=None)[source]

Loads the contents of requirements.txt on the given path.

Defaults to “./requirements.txt”

bob.extension.utils.find_packages(directories=['bob'])[source]

This function replaces the find_packages command from setuptools to search for packages only in the given directories. Using this function will increase the building speed, especially when you have (links to) deep non-code-related directory structures inside your package directory. The given directories should be a list of top-level sub-directories of your package, where package code can be found. By default, it uses 'bob' as the only directory to search.

Generates a list of documented packages on our documentation server for the packages read from the “requirements.txt” file and the given list of additional packages.

Parameters:

additional_packages[str]

A list of additional bob packages for which the documentation urls are added. By default, ‘numpy’ is added

requirements_filestr or file-like

The file (relative to the documentation directory), where to read the requirements from. If None, it will be skipped.

serverstr or None

The url to the server which provides the documentation. If None (the default), the BOB_DOCUMENTATION_SERVER environment variable is taken if existent. If neither server is specified, nor a BOB_DOCUMENTATION_SERVER environment variable is set, the default "http://www.idiap.ch/software/bob/docs/bob/%(name)s/%(version)s/" is used.

bob.extension.download.extract_compressed_file(filename)[source]

Extracts a compressed file.

Parameters

filename (str) – Path to the .zip, .tar, .tar.*, .tgz, .tbz2, and .bz2 file

Raises

ValueError – If the extension of the file is not recognized.

bob.extension.download.download_file(url, out_file)[source]

Downloads a file from a given url

Parameters
  • url (str) – The url to download form.

  • out_file (str) – Where to save the file.

bob.extension.download.download_file_from_possible_urls(urls, out_file)[source]

Tries to download a file from a list of possible urls. The function stops as soon as one url works and raises an error when all urls fail.

Parameters
  • urls (list) – List of urls

  • out_file (str) – Path to save the file

Raises

RuntimeError – If downloading from all urls fails.

bob.extension.download.validate_file(fpath, file_hash, algorithm='auto', chunk_size=65535)[source]

Validates a file against a sha256 or md5 hash.

Parameters
  • fpath (str) – path to the file being validated

  • file_hash (str) – The expected hash string of the file. The sha256 and md5 hash algorithms are both supported.

  • algorithm (str) – Hash algorithm, one of ‘auto’, ‘sha256’, or ‘md5’. The default ‘auto’ detects the hash algorithm in use.

  • chunk_size (int) – Bytes to read at a time, important for large files.

Returns

Whether the file is valid

Return type

bool

bob.extension.download.get_file(filename, urls, cache_subdir='datasets', file_hash=None, hash_algorithm='auto', extract=False, force=False)[source]

Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in ~/bob_data by default. You may change the location of this folder using:

$ bob config set bob_data_folder /another/location/
Parameters
  • filename (str) – Name of the file

  • urls (list) – List containing the all the URLs. The function will try to download them in order and stops if it succeeds.

  • cache_subdir (str) – Subdirectory where the file is saved.

  • file_hash (str) – The expected hash string of the file after download. The sha256 and md5 hash algorithms are both supported.

  • hash_algorithm (str) – Select the hash algorithm to verify the file. options are ‘md5’, ‘sha256’, and ‘auto’. The default ‘auto’ detects the hash algorithm in use.

  • extract (bool) – If True, will extract the downloaded file.

  • force (bool) – If True, will download the file anyway if it already exists.

Returns

The path to the downloaded file.

Return type

str

Raises

ValueError – If the file_hash does not match the downloaded file

bob.extension.download.download_and_unzip(urls, filename)[source]

Download a file from a given URL list, save it somewhere and unzip/untar if necessary

Example:

download_and_unzip(
    ["https://mytesturl.co/my_file_example.tag.bz2"],
    filename="~/my_file_example.tag.bz2"
)
Parameters
  • urls (list) – List containing the all the URLs. The function will try to download them in order

  • filename (str) – File name (full path) where the downloaded file will be written and uncompressed

bob.extension.download.find_element_in_tarball(filename, target_path, open_as_stream=False)[source]

Search an element in a tarball.

Parameters
  • filename (str) – Tarball file name

  • target_path (str) – Target path to be searched inside of the tarball

  • open_as_stream (bool) – If True, will load the element from the tarball as a byte_stream. If False, will load as text

Returns

It returns an opened file

Return type

object

bob.extension.download.search_file(base_path, options)[source]

Search for files either in a file structure, or in a tarball.

Parameters
  • base_path (str) – Base folder to start the search, or the tarball to be searched

  • options (list) – Files to be searched. This function will return the first occurrence. The option can be an incomplete relative path. For example, if you have a file called "/a/b/c/d.txt", and base_path is "/a/b", then options can be ["d.txt"].

Returns

It returns an opened file

Return type

object

bob.extension.download.list_dir(base_path, inner_folder='', folders=True, files=True)[source]

Lists the files and folders inside a folder or a tarball. To list an inner level folder (useful when base_path is a tarball), provide the inner_folder argument.

Parameters
  • base_path (str) – Path to a folder or a tarball

  • inner_folder (str) – Path to an inner folder inside base_path. If given, the folders inside this folder are listed.

  • folders (bool) – If False, will exclude folders from the results.

  • files (bool) – If False, will exclude files from the results.

Returns

Sorted list of file and directory names

Return type

list

Raises

ValueError – If base_path is not a folder or a tarball

Configuration

Implements a global configuration system for bob using json.

bob.extension.rc_config.ENVNAME = 'BOBRC'

Name of environment variable to look for an alternative for the RC file

bob.extension.rc_config.RCFILENAME = '~/.bobrc'

Default name to be used for the RC file to load

Functionality to implement python-based config file parsing and loading.

bob.extension.config.load(paths, context=None, entry_point_group=None, attribute_name=None)[source]

Loads a set of configuration files, in sequence

This method will load one or more configuration files. Every time a configuration file is loaded, the context (variables) loaded from the previous file is made available, so the new configuration file can override or modify this context.

Parameters
  • paths ([str]) – A list or iterable containing paths (relative or absolute) of configuration files that need to be loaded in sequence. Each configuration file is loaded by creating/modifying the context generated after each file readout.

  • context (dict, optional) – If provided, start the readout of the first configuration file with the given context. Otherwise, create a new internal context.

  • entry_point_group (str, optional) – If provided, it will treat non-existing file paths as entry point names under the entry_point_group name.

  • attribute_name (None or str) – If provided, will look for the attribute_name variable inside the loaded files. Paths ending with some_path:variable_name can override the attribute_name. The entry_point_group must provided as well attribute_name is not None.

Returns

mod – A module representing the resolved context, after loading the provided modules and resolving all variables. If attribute_name is given, the object with the attribute_name name (or the name provided by user) is returned instead of the module.

Return type

module or object

Raises
  • ImportError – If attribute_name is given but the object does not exist in the paths.

  • ValueError – If attribute_name is given but entry_point_group is not given.

bob.extension.config.mod_to_context(mod)[source]

Converts the loaded module of load to a dictionary context. This function removes all the variables that start and end with __.

Parameters

mod (object) – What is returned by load

Returns

The context that was in mod.

Return type

dict

bob.extension.config.resource_keys(entry_point_group, exclude_packages=[], strip=['dummy'], with_project_names=False)[source]

Reads and returns all resources that are registered with the given entry_point_group. Entry points from the given exclude_packages are ignored.

Parameters
  • entry_point_group (str) – The entry point group name.

  • exclude_packages (list, optional) – List of packages to exclude when finding resources.

  • strip (list, optional) – Entrypoint names that start with any value in strip will be ignored.

  • with_project_names (bool, optional) – If True, will return a list of tuples with the project name and the entry point name.

Returns

List of found entry point names. If with_project_names is True, will return a list of tuples with the project name and the entry point name.

Return type

list

Logging

Sets-up logging, centrally for Bob.

bob.extension.log.set_verbosity_level(logger, level)[source]

Sets the log level for the given logger.

Parameters
  • logger (logging.Logger or str) – The logger to generate logs for, or the name of the module to generate logs for.

  • level (int) – Possible log levels are: 0: Error; 1: Warning; 2: Info; 3: Debug.

Raises

ValueError – If the level is not in range(0, 4).

bob.extension.log.setup(logger_name, format='%(name)s@%(asctime)s -- %(levelname)s: %(message)s')[source]

This function returns a logger object that is set up to perform logging using Bob loggers.

Parameters
  • logger_name (str) – The name of the module to generate logs for

  • format (str, optional) – The format of the logs, see logging.LogRecord for more details. By default, the log contains the logger name, the log time, the log level and the massage.

Returns

logger – The logger configured for logging. The same logger can be retrieved using the logging.getLogger() function.

Return type

logging.Logger

Scripts

bob.extension.scripts.click_helper.bool_option(name, short_name, desc, dflt=False, **kwargs)[source]

Generic provider for boolean options

Parameters
  • name (str) – name of the option

  • short_name (str) – short name for the option

  • desc (str) – short description for the option

  • dflt (bool or None) – Default value

  • **kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

bob.extension.scripts.click_helper.list_float_option(name, short_name, desc, nitems=None, dflt=None, **kwargs)[source]

Get option to get a list of float f

Parameters
  • name (str) – name of the option

  • short_name (str) – short name for the option

  • desc (str) – short description for the option

  • nitems (obj:int, optional) – If given, the parsed list must contains this number of items.

  • dflt (list, optional) – List of default values for axes.

  • **kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

bob.extension.scripts.click_helper.open_file_mode_option(**kwargs)[source]

Get open mode file option

Parameters

**kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

bob.extension.scripts.click_helper.verbosity_option(**kwargs)[source]

Adds a -v/–verbose option to a click command.

Parameters

**kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

class bob.extension.scripts.click_helper.ConfigCommand(name, *args, help=None, entry_point_group=None, **kwargs)[source]

Bases: Command

A click.Command that can take options both form command line options and configuration files. In order to use this class, you have to use the ResourceOption class also.

config_argument_name

The name of the config argument.

Type

str

entry_point_group

The name of entry point that will be used to load the config files.

Type

str

dump_config(ctx, param, value)[source]

Generate configuration file from parameters and context

Parameters

ctx (object) – Click context

class bob.extension.scripts.click_helper.CustomParamType[source]

Bases: ParamType

name: str = 'custom'

the descriptive name of this type

class bob.extension.scripts.click_helper.ResourceOption(param_decls=None, show_default=False, prompt=False, confirmation_prompt=False, hide_input=False, is_flag=None, flag_value=None, multiple=False, count=False, allow_from_autoenv=True, type=None, help=None, entry_point_group=None, required=False, string_exceptions=None, **kwargs)[source]

Bases: Option

An extended click.Option that automatically loads resources from config files.

This class comes with two different functionalities that are independent and could be combined:

  1. If used in commands that are inherited from ConfigCommand, it will lookup inside the config files (that are provided as argument to the command) to resolve its value. Values given explicitly in the command line take precedence.

  2. If entry_point_group is provided, it will treat values given to it (by any means) as resources to be loaded. Loading is done using load. See Resource Loading for more information. The final value cannot be a string.

You may use this class in three ways:

  1. Using this class (without using ConfigCommand) AND (providing entry_point_group).

  2. Using this class (with ConfigCommand) AND (providing entry_point_group).

  3. Using this class (with ConfigCommand) AND (without providing entry_point_group).

Using this class without ConfigCommand and without providing entry_point_group does nothing and is not allowed.

entry_point_group

If provided, the strings values to this option are assumed to be entry points from entry_point_group that need to be loaded.

Type

str or None

string_exceptions

If provided and entry_point_group is provided, the code will not treat strings in string_exceptions as entry points and does not try to load them.

Type

tuple or None

consume_value(ctx, opts)[source]
type_cast_value(ctx, value)[source]

Convert and validate a value against the option’s type, multiple, and nargs. Furthermore, if the an entry_point_group is provided, it will load it.

class bob.extension.scripts.click_helper.AliasedGroup(name: Optional[str] = None, commands: Optional[Union[Dict[str, Command], Sequence[Command]]] = None, **attrs: Any)[source]

Bases: Group

Class that handles prefix aliasing for commands

Basically just implements get_command that is used by click to choose the command based on the name.

Example

To enable prefix aliasing of commands for a given group, just set cls=AliasedGroup parameter in click.group decorator.

get_command(ctx, cmd_name)[source]

get_command with prefix aliasing

bob.extension.scripts.click_helper.log_parameters(logger_handle, ignore=())[source]

Logs the click parameters with the logging module.

Parameters
  • logger_handle (object) – The logger handle to write debug information into.

  • ignore (tuple) – The keys in ignore will not be logged.

bob.extension.scripts.click_helper.assert_click_runner_result(result, exit_code=0, exception_type=None)[source]

Helper for asserting click runner results