.. vim: set fileencoding=utf-8 : .. Copyright (c) 2019 Idiap Research Institute, http://www.idiap.ch/ .. .. Contact: beat.support@idiap.ch .. .. .. .. This file is part of the beat.backend.python module of the BEAT platform. .. .. .. .. Redistribution and use in source and binary forms, with or without .. modification, are permitted provided that the following conditions are met: .. 1. Redistributions of source code must retain the above copyright notice, this .. list of conditions and the following disclaimer. .. 2. Redistributions in binary form must reproduce the above copyright notice, .. this list of conditions and the following disclaimer in the documentation .. and/or other materials provided with the distribution. .. 3. Neither the name of the copyright holder nor the names of its contributors .. may be used to endorse or promote products derived from this software without .. specific prior written permission. .. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND .. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED .. WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE .. DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE .. FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .. DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR .. SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER .. CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, .. OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE .. OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .. _beat-cmdline-experiments: Experiments ----------- The BEAT command-line utility called ``beat`` can perform a variety of actions concerning experiments. These actions are grouped in two sets: * local: Local actions allow the user to act on locally installed objects. * remote: Actions on the remote web server. Once you have setup your prefix directory as explained in :ref:`beat-cmdline-configuration`, you're ready to start configuring new experiments. The commands available for the experiments are: .. command-output:: beat experiments --help For instance, a list of the experiments available locally can be obtained as follows: .. command-output:: beat experiments list :cwd: .. A list of the experiments available on the remote platform can be obtained by running the following command: .. code-block:: sh $ beat experiments list --remote .. _beat-cmdline-experiments-running: How to run an experiment? ......................... The command ``beat experiments run `` can be used to run the experiment defined in an experiment definition file. It is the ideal way to debug an experiment, since by default ``beat`` will use the local executor, which provides a simple environment with PDB support without advanced features (multi-processing, optimizations, sandboxing, multiple environments, etc.). Here, the ``--prefix`` option is used to tell the scripts where all our data formats, toolchains and algorithms are located. This option can be set in your configuration file (see :ref:`beat-cmdline-configuration`). This command displays for each block the files containing the data to use as input, and the files generated by the outputs of the block. The default behavior is to not regenerate data files already present in the cache. You can force the script to not take the content of the cache into account with the ``--force`` flag. Executors ========= "Executors" are modules that execute each block in an experiment. On the BEAT platform, there is only the one executor, which executes the experiment using Docker containers with advanced scheduling and security features. When using ``beat.cmdline`` locally, however, you have the option of using either the BEAT platform's executor, behind the ``--docker`` flag (for more information about using docker executors (see :ref:`here `), or the "local" executor (refer to "Installation Instruction" in `BEAT documentation`_ for information about local environment). The local executor, is much simpler, aimed at providing a smooth development experience. However, there are two important tradeoffs: - Lower performance for non-trivial experiments, as it runs everything synchronously in one process on the CPU. - No multiple environments, as the Python environment that built ``beat.cmdline`` is used. This means that many BEAT experiments that rely on different/multiple environments will not work. .. _beat-cmdline-experiments-displaydata: How to examine the content of a data file? .......................................... The ``beat cache`` collection of commands interact with the cache: .. command-output:: beat cache --help How to plot output images from experiments? ........................................... The command ``beat experiments plot `` can be used to plot output images for the various experiments. There a two ways to plot data: * using remote data results: .. code-block:: sh $ beat experiments plot --remote * using data from the cache of locally ran experiments .. code-block:: sh $ beat experiments plot In both cases the 'output folder' option can be specified to save all the plots to a specific directory. By default, if nothing was specified, the plots will be saved under the experiment path. Take into account that some extra options are available such as '--show' which will pop out the generated plots on your screen. .. include:: links.rst