Experiments

An experiment is the reunion of algorithms, datasets, a toolchain and parameters that allow the platform to schedule and run the prescribed recipe to produce displayable results. Defining a BEAT experiment can be seen as configuring the processing blocks of a toolchain, such as selecting which database, algorithms and algorithm parameters to use.

Declaration of an experiment

Note

One needs only to declare an experiment using those specifications when not using the web interface (i.e. when doing local development or using the web api). The web interface provides a user-friendly way to configure an experiment.

An experiment is declared in a JSON file, and must contain at least the following fields:

{
    "datasets": [
    ],
    "blocks": [
    ],
    "analyzers": [
    ],
    "globals": [
    ]
}

Declaration of the dataset(s)

The dataset inputs are defined by the toolchain. However, the toolchain does not describe which data to plug in each dataset input.

This is the role of the field datasets from an experiment. For each dataset, an experiment must specify three attributes as follows:

{
    "datasets": [
        "templates": {
            "set": "templates",
            "protocol": "idiap",
            "database": "atnt/1"
        },
        ...
    ],
    ...
}

The key of an experiment dataset must correspond to the desired dataset name from the toolchain. Then, three fields must be given:

• database: the database name and version
• protocol: the protocol name
• set: the dataset name of this database to associate to this toolchain dataset

Declaration of the block(s)

The blocks are defined by the toolchain. However, the toolchain does not describe which algorithm to run in each processing block, and how each of these algorithms are parametrized.

This is the role of the field blocks from an experiment. For each block, an experiment must specify four attributes as follows:

{
    "blocks": {
        "linear_machine_training": {
            "inputs": {
                "image": "image"
            },
            "parameters": {},
            "algorithm": "tutorial/pca/1",
            "outputs": {
                "subspace": "subspace"
            }
        },
        ...
    },
    ...
}

The key of an experiment block must correspond to the desired block from the toolchain. Then, four fields must be given:

• algorithm: the algorithm to use (author_name/algorithm_name/version)
• inputs: the list of inputs. The key is the algorithm input, while the value is the corresponding toolchain input.
• outputs: the list of outputs. The key is the algorithm output, while the value is the corresponding toolchain output.
• parameters: the algorithm parameters to use for this processing block

Note

When setting an algorithm in a processing block, this will also set the dataformats of the outputs (and inputs) of this block. In particular, this has an impact on all the inputs of blocks connected to those outputs, which must have the same data formats (or be an extension of these data formats). The platform automatically validate that the data formats of consecutive blocks are compatible.

Declaration of the analyzer(s)

Analyzers are similar to algorithms, except that they run on toolchain endpoints. There configuration is very similar to the one of regular blocks, except that they have no outputs:

{
    "analyzers": {
        "analysis": {
            "inputs": {
                "scores": "scores"
            },
            "algorithm": "tutorial/postperf/1"
        }
    },
}

Global parameters

Each block and analyzer may rely on its own local parameters. However, several blocks may rely on the exact same parameters. In this case, it is more convenient to define those globally.

For an experiment, this is achieved using the globals field in its JSON declaration. For instance:

{
    "globals": {
        "queue": "Default",
        "environment": {
            "version": "0.0.3",
            "name": "Scientific Python 2.7"
        },
        "tutorial/pca/1": {
            "number-of-components": "5"
        }
    },
    ...
}