Experiments

The BEAT command-line utility called beat can perform a variety of actions concerning experiments. These actions are grouped in two sets:

  • local: Local actions allow the user to act on locally installed objects.

  • remote: Actions on the remote web server.

Once you have setup your prefix directory as explained in Configuration, you’re ready to start configuring new experiments.

The commands available for the experiments are:

$ beat experiments --help
Usage: beat experiments [OPTIONS] COMMAND [ARGS]...

  experiments commands

Options:
  -h, -?, --help  Show this message and exit.

Commands:
  caches     Lists all cache files used by this experiment
  cancel     Cancel an experiment on the platform
  check      Checks a local asset for validity.
  diff       Shows changes between the local asset and the remote version...
  draw       Creates a visual representation of the experiment.
  edit       Edit local asset file Example: $ beat <asset_type> edit xxx
  fork       Forks a local asset Example: $ beat <asset_type> fork xxx yyy
  list       Lists the assets of this type available on the platform.
  monitor    Monitor a running experiment
  path       Displays local path of asset files Example: $ beat
             <asset_type>...

  plot       Plots output images of the experiment.
  pull       Downloads the specified experiments from the server.
  push       Uploads asset to the server Example: $ beat algorithms push...
  reset      Reset an experiment on the platform
  rm         Deletes a local asset (unless --remote is specified) Example:
             $...

  run        Runs an experiment locally
  runstatus  Shows the status of an experiment on the platform
  start      Start an experiment on the platform
  status     Shows (editing) status for all available items of asset type...

For instance, a list of the experiments available locally can be obtained as follows:

$ beat experiments list

A list of the experiments available on the remote platform can be obtained by running the following command:

$ beat experiments list --remote

How to run an experiment?

The command beat experiments run <name> can be used to run the experiment defined in an experiment definition file. It is the ideal way to debug an experiment, since by default beat will use the local executor, which provides a simple environment with PDB support without advanced features (multi-processing, optimizations, sandboxing, multiple environments, etc.).

Here, the --prefix option is used to tell the scripts where all our data formats, toolchains and algorithms are located. This option can be set in your configuration file (see Configuration).

This command displays for each block the files containing the data to use as input, and the files generated by the outputs of the block.

The default behavior is to not regenerate data files already present in the cache. You can force the script to not take the content of the cache into account with the --force flag.

Executors

“Executors” are modules that execute each block in an experiment. On the BEAT platform, there is only the one executor, which executes the experiment using Docker containers with advanced scheduling and security features. When using beat.cmdline locally, however, you have the option of using either the BEAT platform’s executor, behind the --docker flag (for more information about using docker executors (see here), or the “local” executor (refer to “Installation Instruction” in BEAT documentation for information about local environment). The local executor, is much simpler, aimed at providing a smooth development experience. However, there are two important tradeoffs:

  • Lower performance for non-trivial experiments, as it runs everything synchronously in one process on the CPU.

  • No multiple environments, as the Python environment that built beat.cmdline is used. This means that many BEAT experiments that rely on different/multiple environments will not work.

How to examine the content of a data file?

The beat cache collection of commands interact with the cache:

$ beat cache --help
Usage: beat cache [OPTIONS] COMMAND [ARGS]...

  Configuration manipulation and display

Options:
  --start INTEGER  If set, allows the user to print only a few bits of the
                   file

  --end INTEGER    If set, allows the user to print only a few bits of the
                   file

  -?, -h, --help   Show this message and exit.

Commands:
  clear   Deletes all available cache To clear all available cache: $...
  info    Displays information about a particular cache file To collect...
  remove  Remove content of the cache entries passed in parameters To
          remove...

  view    Displays information about a particular cache file To view a...

How to plot output images from experiments?

The command beat experiments plot <experiment name> can be used to plot output images for the various experiments.

There a two ways to plot data:

  • using remote data results:

$ beat experiments plot --remote <experiment name>

  • using data from the cache of locally ran experiments

$ beat experiments plot <experiment name>

In both cases the ‘output folder’ option can be specified to save all the plots to a specific directory. By default, if nothing was specified, the plots will be saved under the experiment path.

Take into account that some extra options are available such as ‘–show’ which will pop out the generated plots on your screen.