Data formats¶
Data formats formalize the interaction between algorithms and data sets, so they can communicate data in an orderly manner. All data formats produced or consumed by these objects must be formally declared. Two algorithms which must directly communicate data must produce and consume the same type of data objects.
A data format specifies a list of typed fields. An algorithm or data set generating a block of data (via one of its outputs) must fill all the fields declared in that data format. An algorithm consuming a block of data (via one of its inputs) must not expect the presence of any other field than the ones defined by the data format.
This section contains information on the definition of dataformats, its programmatic use on Python-based language bindings.
Definition¶
A data format is declared as a JSON object with several fields. For example, the following declaration could represent the coordinates of a rectangular region in an image:
{
"x": "int32",
"y": "int32",
"width": "int32",
"height": "int32"
}
Note
We have chosen to define objects inside the BEAT platform using JSON declarations as JSON files can be easily validated, transferred through web-based APIs and provide and easy to read format for local inspection.
Each field must be named according to typical programming rules for variable names. For example, these are valid names:
my_field
_my_field
number1
These are invalid field names:
1number
my field
The following regular expression is used to validate field names:
^[a-zA-Z_][a-zA-Z0-9_-]*$
. In short, a field name has to start with a
letter or an underscore character and can contain, immediately after, any
number of alpha-numerical characters or underscores.
By convention, fields prefixed and suffixed with a double underscore (__
)
are reserved and should be avoided.
The special field #description
can be used to store a short description of
the declared data format and also ignored:
{
"#description": "A rectangle in an pixeled image",
"x": "int32",
"y": "int32",
"width": "int32",
"height": "int32"
}
The #description
field is ignored in practice and only used for
informational purposes.
Each field in a declaration has a well-defined type, which can be one of:
- a primitive, simple type (see Simple types)
- a directly nested object (see Complex types)
- another data format (see Aggregation)
- an array (see Arrays)
A data format can also extend another one, as explained further down (see ref:beat-core-dataformats-extension).
Simple types¶
The following primitive data types are available in the BEAT platform:
- Integers:
int8
,int16
,int32
,int64
- Unsigned integers:
uint8
,uint16
,uint32
,uint64
- Floating-point numbers:
float32
,float64
- Complex numbers:
complex64
,complex128
bool
string
Note
All primitive types are implemented using their numpy
counterparts.
When determining if a block of data corresponds to a data format, the platform will check that the value of each field can safely (without loss of precision) be converted to the type declared by the data format. An error is generated if you fail to follow these requirements.
For example, an int8
can be converted, without a precision loss, to an
int16
, but a float32
cannot be losslessly converted to an
int32
. In case of doubt, you can manually test for NumPy safe-casting
rules yourself in order to understand imposed restrictions. If you wish to
allow for a precision loss on your code, you must do it explicitly (Zen of
Python).
Complex types¶
A data format can be composed of complex objects formed by nesting other types. The coordinates of a rectangular region in an image be represented like this:
{
"coords": {
"x": "int32",
"y": "int32"
},
"size": {
"width": "int32",
"height": "int32"
}
}
Aggregation¶
Note
Data formats are named using 3 values joined by a /
(slash) separator:
the username who is the author of the dataformat, an identifier and the
object version (integer starting from 1). Here are examples of data format
names:
user/my_format/1
johndoe/integers/37
mary_mary/rectangle/2
A field can use the declaration of another data format instead of specifying
its own declaration. Consider the following data formats, on their first
version, for user user
:
{
"x": "int32",
"y": "int32"
}
{
"width": "int32",
"height": "int32"
}
Now let’s aggregate both previous formats in order to declare a new data format for describing a rectangle:
{
"coords": "user/coordinates/1",
"size": "user/size/1"
}
Arrays¶
A field can be a multi-dimensional array of any other type. For instance, consider the following example:
{
"field1": [10, "int32"],
"field2": [10, 5, "bool"]
}
Here we declare that field1
is a one-dimensional array of 10 32-bit signed
integers (int32
), and field2
is a two-dimensional array with 10 rows
and 5 columns of booleans.
Note
In the Python language representation of data formats, multi-dimensional
arrays are implemented using numpy.ndarray
’s.
An array can have as many dimensions as you want. It can also contain objects (either declared inline, or using another data format):
{
"inline": [10, {
"x": "int32",
"y": "int32"
}],
"imported": [10, "beat/coordinates/1"]
}
It is also possible to declare an array without specifying the number of elements in some of its dimensions, by using a size of 0 (zero):
{
"field1": [0, "int32"],
"field2": [0, 0, "bool"],
"field3": [10, 0, "float32"]
}
Here, field1
is a one-dimensional array of 32-bit signed integers
(int32
), field2
is a two-dimensional array of booleans, and field3
is a two-dimensional array of floating-point numbers (float32
) whose the
first dimension is fixed to 10 (number of rows).
Note that the following declaration isn’t valid (you can’t fix a dimension if the preceding one isn’t fixed too):
{
"error": [0, 10, "int32"]
}
Note
When determining if that a block of data corresponds to a data format containing an array, the platform automatically checks that:
- the number of dimensions is correct
- the size of each declared dimension that isn’t 0 is correct
- the type of each value in the array is correct
Extensions¶
Besides aggregation, it is possible to extend data formats through inheritance. In practice, inheriting from a data format is the same as pasting its declaration right on the top of the new format.
For example, one might implement a face detector algorithm and may want to
create a data format containing all the informations about a face (say its
position, its size and the position of each eye). This could be done by
extending the type user/rectangular_area/1
defined earlier:
{
"#extends": "user/rectangular_area/1",
"left_eye": "coordinates",
"right_eye": "coordinates"
}
Python API¶
Data formats are useful descriptions of data blocks that are consumed by algorithmic code inside the platform. In BEAT, the user never instantiates data formats directly. Instead, when a new object representing a data format needs to be created, the user may just create a dictionary in which the keys are the format field names, whereas the values are instances of the type defined for such a field. If the type is a reference to another format, the user may nest dictionaries so as to build objects of any complexity. When the dictionary representing a data format is written to an algorithm output, the data is properly validated.
This concept will become clearer when you’ll read about algorithms and the way they receive and produce data. Here is just a simple illustrative example:
# suppose, for this example, `output' is provided to your algorithm
output.write({
"x": numpy.int32(10),
"y": numpy.int32(20),
"width": numpy.int32(100),
"height": numpy.int32(100),
})