.. vim: set fileencoding=utf-8 :

.. Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/          ..
.. Contact: beat.support@idiap.ch                                             ..
..                                                                            ..
.. This file is part of the beat.core module of the BEAT platform.            ..
..                                                                            ..
.. Commercial License Usage                                                   ..
.. Licensees holding valid commercial BEAT licenses may use this file in      ..
.. accordance with the terms contained in a written agreement between you     ..
.. and Idiap. For further information contact tto@idiap.ch                    ..
..                                                                            ..
.. Alternatively, this file may be used under the terms of the GNU Affero     ..
.. Public License version 3 as published by the Free Software and appearing   ..
.. in the file LICENSE.AGPL included in the packaging of this file.           ..
.. The BEAT platform is distributed in the hope that it will be useful, but   ..
.. WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ..
.. or FITNESS FOR A PARTICULAR PURPOSE.                                       ..
..                                                                            ..
.. You should have received a copy of the GNU Affero Public License along     ..
.. with the BEAT platform. If not, see http://www.gnu.org/licenses/.          ..


.. _beat-core-experiments:

============
Experiments
============

An experiment is the reunion of algorithms, datasets, a toolchain and
parameters that allow the platform to schedule and run the prescribed recipe
to produce displayable results. Defining a BEAT experiment can be seen as
configuring the processing blocks of a toolchain, such as selecting which
database, algorithms and algorithm parameters to use.

.. _beat-core-experiments-declaration:

Declaration of an experiment
----------------------------

.. note::

   One needs only to declare an experiment using those specifications when not
   using the web interface (i.e. when doing local development or using the web
   api). The web interface provides a user-friendly way to configure an
   experiment.

An experiment is declared in a JSON file, and must contain at least the following
fields:

.. code-block:: javascript

    {
        "datasets": [
        ],
        "blocks": [
        ],
        "analyzers": [
        ],
        "globals": [
        ]
    }


.. _beat-core-experiments-datasets:

Declaration of the dataset(s)
-----------------------------

The dataset inputs are defined by the toolchain. However, the toolchain does
not describe which data to plug in each dataset input.

This is the role of the field `datasets` from an experiment.
For each dataset, an experiment must specify three attributes as follows:

.. code-block:: javascript

    {
        "datasets": [
            "templates": {
                "set": "templates",
                "protocol": "idiap",
                "database": "atnt/1"
            },
            ...
        ],
        ...
    }


The key of an experiment dataset must correspond to the desired dataset name
from the toolchain. Then, three fields must be given:

* `database`: the database name and version
* `protocol`: the protocol name
* `set`: the dataset name of this database to associate to this toolchain
  dataset


.. _beat-core-experiments-blocks:

Declaration of the block(s)
---------------------------

The blocks are defined by the toolchain. However, the toolchain does not
describe which algorithm to run in each processing block, and how each of these
algorithms are parametrized.

This is the role of the field `blocks` from an experiment.
For each block, an experiment must specify four attributes as follows:

.. code-block:: javascript

    {
        "blocks": {
            "linear_machine_training": {
                "inputs": {
                    "image": "image"
                },
                "parameters": {},
                "algorithm": "tutorial/pca/1",
                "outputs": {
                    "subspace": "subspace"
                }
            },
            ...
        },
        ...
    }

The key of an experiment block must correspond to the desired block from the
toolchain. Then, four fields must be given:

* `algorithm`: the algorithm to use (author_name/algorithm_name/version)
* `inputs`: the list of inputs. The key is the algorithm input, while the
  value is the corresponding toolchain input.
* `outputs`: the list of outputs. The key is the algorithm output, while the
  value is the corresponding toolchain output.
* `parameters`: the algorithm parameters to use for this processing block


.. note::

   When setting an algorithm in a processing block, this will also set the
   dataformats of the outputs (and inputs) of this block. In particular,
   this has an impact on all the inputs of blocks connected to those outputs,
   which must have the same data formats (or be an extension of these data
   formats). The platform automatically validate that the data formats of
   consecutive blocks are compatible.


.. _beat-core-experiments-analyzers:

Declaration of the analyzer(s)
------------------------------

Analyzers are similar to algorithms, except that they run on toolchain
endpoints. There configuration is very similar to the one of regular blocks,
except that they have no `outputs`:

.. code-block:: javascript

    {
        "analyzers": {
            "analysis": {
                "inputs": {
                    "scores": "scores"
                },
                "algorithm": "tutorial/postperf/1"
            }
        },
    }


Global parameters
-----------------

Each block and analyzer may rely on its own local parameters. However, several
blocks may rely on the exact same parameters. In this case, it is more
convenient to define those globally.

For an experiment, this is achieved using the `globals` field in its JSON
declaration. For instance:

.. code-block:: javascript

    {
        "globals": {
            "queue": "Default",
            "environment": {
                "version": "0.0.3",
                "name": "Scientific Python 2.7"
            },
            "tutorial/pca/1": {
                "number-of-components": "5"
            }
        },
        ...
    }