Data formats

Data formats specify the transmitted data between the blocks of a toolchain. They describe the format of the data blocks that circulate between algorithms and formalize the interaction between algorithms and data sets, so they can communicate in an orderly manner. Inputs and outputs of the algorithms and datasets must be formally declared. Two algorithms that communicate directly must produce and consume the same type of data objects.

A data format specifies a list of typed fields. An algorithm or data set generating a block of data (via one of its outputs) must fill all the fields declared in that data format. An algorithm consuming a block of data (via one of its inputs) must not expect the presence of any other field than the ones defined by the data format.

The BEAT framework provides a number of pre-defined formats to facilitate experiments. They are implemented in an extensible way. This allows users to define their own formats, based on existing ones, while keeping some level of compatibility with other existing algorithms.

This section contains information on the definition of dataformats and its programmatic use on Python-based language bindings.

Note

Naming Convention

Data formats are named using three values joined by a / (slash) operator:

  • username: indicates the author of the dataformat

  • name: an identifier for the object

  • version: an integer (starting from one), indicating the version of the object.

Each tuple of these three components defines a unique data format name inside the framework. Here are examples of data format names:

  • user/my_format/1

  • johndoe/integers/37

  • mary_mary/rectangle/2

Definition

A data format is declared as a JSON object with several fields. For example, the following declaration could represent the coordinates of a rectangular region in an image:

{
    "x": "int32",
    "y": "int32",
    "width": "int32",
    "height": "int32"
}

Note

We have chosen to define objects inside the BEAT framework using JSON declarations as JSON files can be easily validated, transferred through web-based APIs and provide an easy to read format for local inspection.

Each field must be named according to typical programming rules for variable names. For example, these are valid names:

  • my_field

  • _my_field

  • number1

These are invalid field names:

  • 1number

  • my field

The following regular expression is used to validate field names: ^[a-zA-Z_][a-zA-Z0-9_-]*$. In short, a field name has to start with a letter or an underscore character and can contain, immediately after, any number of alpha-numerical characters or underscores.

By convention, fields prefixed and suffixed with a double underscore (__) are reserved and should be avoided.

The special field #description can be used to store a short description of the declared data format:

{
    "#description": "A rectangle in an image",
    "x": "int32",
    "y": "int32",
    "width": "int32",
    "height": "int32"
}

The #description field is ignored in practice and only used for informational purposes.

Each field in a declaration has a well-defined type, which can be one of the following:

A data format can also extend to another one, as explained further down (see Extensions).

Simple types

The following primitive data types are available in the BEAT frame work:

  • Integers: int8, int16, int32, int64

  • Unsigned integers: uint8, uint16, uint32, uint64

  • Floating-point numbers: float32, float64

  • Complex numbers: complex64, complex128

  • bool

  • string

Note

All primitive types are implemented using their numpy counterparts.

When determining if a block of data corresponds to a data format, the system will check that the value of each field can safely (without loss of precision) be converted to the type declared by the data format. An error is generated if you fail to follow these requirements.

For example, an int8 can be converted, without a precision loss, to an int16, but a float32 cannot be losslessly converted to an int32. In case of doubt, you can manually test for NumPy safe-casting rules yourself in order to understand imposed restrictions. If you wish to allow for a precision loss on your code, you must do it explicitly (Zen of Python).

Complex types

A data format can be composed of complex objects formed by nesting other types. The coordinates of a rectangular region in an image can be represented like this:

{
    "coords": {
        "x": "int32",
        "y": "int32"
    },
    "size": {
        "width": "int32",
        "height": "int32"
    }
}

Aggregation

A field can use the declaration of another data format instead of specifying its own declaration. Consider the following data formats, on their first version, for user user:

Listing 1 : Two dimensional coordinates (user/coordinates/1)
 {
     "x": "int32",
     "y": "int32"
 }
Listing 2 : Two dimensional size (user/size/1)
 {
     "width": "int32",
     "height": "int32"
 }

Now let’s aggregate both previous formats in order to declare a new data format for describing a rectangle:

Listing 3 : The definition of a rectangle (user/rectangle/1)
 {
     "coords": "user/coordinates/1",
     "size": "user/size/1"
 }

Arrays

A field can be a multi-dimensional array of any other type. For instance, consider the following example:

{
    "field1": [10, "int32"],
    "field2": [10, 5, "bool"]
}

Here we declare that field1 is a one-dimensional array of 10 32-bit signed integers (int32), and field2 is a two-dimensional array with 10 rows and 5 columns of booleans.

Note

In the Python language representation of data formats, multi-dimensional arrays are implemented using numpy.ndarray.

An array can have up to 32 dimensions. This number might be lower depending on the underlying programming language and methods used for producing such arrays.

An array can also contain objects (either declared inline, or using another data format):

{
    "inline": [10, {
        "x": "int32",
        "y": "int32"
    }],
    "imported": [10, "beat/coordinates/1"]
}

It is also possible to declare an array without specifying the number of elements in some of its dimensions, by using a size of 0 (zero):

{
    "field1": [0, "int32"],
    "field2": [0, 0, "bool"],
    "field3": [10, 0, "float32"]
}

Here, field1 is a one-dimensional array of 32-bit signed integers (int32), field2 is a two-dimensional array of booleans, and field3 is a two-dimensional array of floating-point numbers (float32) whose the first dimension is fixed to 10 (number of rows).

Because of the way the BEAT framework stores data, not all combinations of unspecified extents will work for arrays. As a rule of thumb, only the last dimensions may remain unspecified you can’t fix a dimension if the preceding one isn’t fixed too). These are valid:

{
    "value1": [0, "float64"],
    "value2": [3, 0, "float64"],
    "value3": [3, 2, 0, "float64"],
    "value4": [3, 0, 0, "float64"],
    "value5": [0, 0, 0, "float64"]
}

Whereas this would be invalid declarations for arrays:

{
    "value": [0, 3, "float64"],
    "value": [4, 0, 3, "float64"]
}

Note

When determining if that a block of data corresponds to a data format containing an array, the system automatically checks that:

  • the number of dimensions is correct.

  • the size of each declared dimension that isn’t 0 is correct.

  • the type of each value in the array is correct.

Extensions

Besides aggregation, it is possible to extend data formats through inheritance. In practice, inheriting from a data format is the same as pasting its declaration right on the top of the new format.

For example, one might implement a face detector algorithm and may want to create a data format containing all the informations about a face (say its position, its size and the position of each eye). This could be done by extending the type user/rectangular_area/1 defined earlier:

{
    "#extends": "user/rectangular_area/1",
    "left_eye": "coordinates",
    "right_eye": "coordinates"
}

Python API

Data formats are useful descriptions of data blocks that are consumed by algorithmic code inside the framework. In BEAT, the user never instantiates data formats directly. Instead, when a new object representing a data format needs to be created, the user may just create a dictionary in which the keys are the format field names, whereas the values are instances of the type defined for such a field. If the type is a reference to another format, the user may nest dictionaries so as to build objects of any complexity. When the dictionary representing a data format is written to an algorithm output, the data is properly validated.

This concept will become clearer when you’ll read about algorithms and the way they receive and produce data. Here is just a simple illustrative example:

# suppose, for this example, `output' is provided to your algorithm
output.write({
  "x": numpy.int32(10),
  "y": numpy.int32(20),
  "width": numpy.int32(100),
  "height": numpy.int32(100),
  })