.. vim: set fileencoding=utf-8 : .. Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/ .. .. Contact: beat.support@idiap.ch .. .. .. .. This file is part of the beat.docs module of the BEAT platform. .. .. .. .. Commercial License Usage .. .. Licensees holding valid commercial BEAT licenses may use this file in .. .. accordance with the terms contained in a written agreement between you .. .. and Idiap. For further information contact tto@idiap.ch .. .. .. .. Alternatively, this file may be used under the terms of the GNU Affero .. .. Public License version 3 as published by the Free Software and appearing .. .. in the file LICENSE.AGPL included in the packaging of this file. .. .. The BEAT platform is distributed in the hope that it will be useful, but .. .. WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY .. .. or FITNESS FOR A PARTICULAR PURPOSE. .. .. .. .. You should have received a copy of the GNU Affero Public License along .. .. with the BEAT platform. If not, see http://www.gnu.org/licenses/. .. .. _beat-system-dataformats: ============== Data formats ============== Data formats specify the transmitted data between the blocks of a toolchain. They describe the format of the data blocks that circulate between algorithms and formalize the interaction between algorithms and data sets, so they can communicate in an orderly manner. Inputs and outputs of the algorithms and datasets **must** be formally declared. Two algorithms that communicate directly must produce and consume the **same** type of data objects. A data format specifies a list of typed fields. An algorithm or data set generating a block of data (via one of its outputs) **must** fill all the fields declared in that data format. An algorithm consuming a block of data (via one of its inputs) **must not** expect the presence of any other field than the ones defined by the data format. The BEAT framework provides a number of pre-defined formats to facilitate experiments. They are implemented in an extensible way. This allows users to define their own formats, based on existing ones, while keeping some level of compatibility with other existing algorithms. This section contains information on the definition of dataformats and its programmatic use on Python-based language bindings. .. note:: **Naming Convention** Data formats are named using three values joined by a ``/`` (slash) operator: * **username**: indicates the author of the dataformat * **name**: an identifier for the object * **version**: an integer (starting from one), indicating the version of the object. Each tuple of these three components defines a *unique* data format name inside the framework. Here are examples of data format names: * ``user/my_format/1`` * ``johndoe/integers/37`` * ``mary_mary/rectangle/2`` Definition ---------- A data format is declared as a `JSON`_ object with several fields. For example, the following declaration could represent the coordinates of a rectangular region in an image: .. code-block:: json { "x": "int32", "y": "int32", "width": "int32", "height": "int32" } .. note:: We have chosen to define objects inside the BEAT framework using JSON declarations as JSON files can be easily validated, transferred through web-based APIs and provide an easy to read format for local inspection. Each field must be named according to typical programming rules for variable names. For example, these are valid names: * ``my_field`` * ``_my_field`` * ``number1`` These are invalid field names: * ``1number`` * ``my field`` The following regular expression is used to validate field names: ``^[a-zA-Z_][a-zA-Z0-9_-]*$``. In short, a field name has to start with a letter or an underscore character and can contain, immediately after, any number of alpha-numerical characters or underscores. By convention, fields prefixed and suffixed with a double underscore (``__``) are reserved and should be avoided. The special field ``#description`` can be used to store a short description of the declared data format: .. code-block:: json { "#description": "A rectangle in an image", "x": "int32", "y": "int32", "width": "int32", "height": "int32" } The ``#description`` field is ignored in practice and only used for informational purposes. Each field in a declaration has a well-defined type, which can be one of the following: * a primitive, simple type (see :ref:`beat-system-dataformats-simple`) * a directly nested object (see :ref:`beat-system-dataformats-complex`) * another data format (see :ref:`beat-system-dataformats-aggregation`) * an array (see :ref:`beat-system-dataformats-array`) A data format can also extend to another one, as explained further down (see :ref:`beat-system-dataformats-extension`). .. _beat-system-dataformats-simple: Simple types ------------ The following primitive data types are available in the BEAT frame work: * Integers: ``int8``, ``int16``, ``int32``, ``int64`` * Unsigned integers: ``uint8``, ``uint16``, ``uint32``, ``uint64`` * Floating-point numbers: ``float32``, ``float64`` * Complex numbers: ``complex64``, ``complex128`` * ``bool`` * ``string`` .. note:: All primitive types are implemented using their :py:mod:`numpy` counterparts. When determining if a block of data corresponds to a data format, the system will check that the value of each field can safely (without loss of precision) be converted to the type declared by the data format. An error is generated if you fail to follow these requirements. For example, an ``int8`` **can** be converted, without a precision loss, to an ``int16``, but a ``float32`` **cannot** be losslessly converted to an ``int32``. In case of doubt, you can manually test for `NumPy safe-casting rules`_ yourself in order to understand imposed restrictions. If you wish to allow for a precision loss on your code, you must do it explicitly (`Zen of Python`_). .. _beat-system-dataformats-complex: Complex types ------------- A data format can be composed of complex objects formed by nesting other types. The coordinates of a rectangular region in an image can be represented like this: .. code-block:: json { "coords": { "x": "int32", "y": "int32" }, "size": { "width": "int32", "height": "int32" } } .. _beat-system-dataformats-aggregation: Aggregation ----------- A field can use the declaration of another data format instead of specifying its own declaration. Consider the following data formats, on their first version, for user ``user``: .. code-block:: json :caption: : Two dimensional coordinates (``user/coordinates/1``) { "x": "int32", "y": "int32" } .. code-block:: json :caption: : Two dimensional size (``user/size/1``) { "width": "int32", "height": "int32" } Now let's aggregate both previous formats in order to declare a new data format for describing a rectangle: .. code-block:: json :caption: : The definition of a rectangle (``user/rectangle/1``) { "coords": "user/coordinates/1", "size": "user/size/1" } .. _beat-system-dataformats-array: Arrays ------ A field can be a multi-dimensional array of any other type. For instance, consider the following example: .. code-block:: json { "field1": [10, "int32"], "field2": [10, 5, "bool"] } Here we declare that ``field1`` is a one-dimensional array of 10 32-bit signed integers (``int32``), and ``field2`` is a two-dimensional array with 10 rows and 5 columns of booleans. .. note:: In the Python language representation of data formats, multi-dimensional arrays are implemented using :py:class:`numpy.ndarray`. An array can have up to 32 dimensions. This number might be lower depending on the underlying programming language and methods used for producing such arrays. An array can also contain objects (either declared inline, or using another data format): .. code-block:: json { "inline": [10, { "x": "int32", "y": "int32" }], "imported": [10, "beat/coordinates/1"] } It is also possible to declare an array without specifying the number of elements in some of its dimensions, by using a size of 0 (zero): .. code-block:: json { "field1": [0, "int32"], "field2": [0, 0, "bool"], "field3": [10, 0, "float32"] } Here, ``field1`` is a one-dimensional array of 32-bit signed integers (``int32``), ``field2`` is a two-dimensional array of booleans, and ``field3`` is a two-dimensional array of floating-point numbers (``float32``) whose the first dimension is fixed to 10 (number of rows). Because of the way the BEAT framework stores data, not all combinations of unspecified extents will work for arrays. As a rule of thumb, only the last dimensions may remain unspecified you can't fix a dimension if the preceding one isn't fixed too). These are valid: .. code-block:: javascript { "value1": [0, "float64"], "value2": [3, 0, "float64"], "value3": [3, 2, 0, "float64"], "value4": [3, 0, 0, "float64"], "value5": [0, 0, 0, "float64"] } Whereas this would be invalid declarations for arrays: .. code-block:: javascript { "value": [0, 3, "float64"], "value": [4, 0, 3, "float64"] } .. note:: When determining if that a block of data corresponds to a data format containing an array, the system automatically checks that: * the number of dimensions is correct. * the size of each declared dimension that isn't 0 is correct. * the type of each value in the array is correct. .. _beat-system-dataformats-extension: Extensions ---------- Besides aggregation, it is possible to extend data formats through inheritance. In practice, inheriting from a data format is the same as pasting its declaration right on the top of the new format. For example, one might implement a face detector algorithm and may want to create a data format containing all the informations about a face (say its position, its size and the position of each eye). This could be done by extending the type ``user/rectangular_area/1`` defined earlier: .. code-block:: json { "#extends": "user/rectangular_area/1", "left_eye": "coordinates", "right_eye": "coordinates" } .. _beat-system-dataformats-usage: Python API ---------- Data formats are useful descriptions of data blocks that are consumed by algorithmic code inside the framework. In BEAT, the user never instantiates data formats directly. Instead, when a new object representing a data format needs to be created, the user may just create a dictionary in which the keys are the format field names, whereas the values are instances of the type defined for such a field. If the type is a reference to another format, the user may nest dictionaries so as to build objects of any complexity. When the dictionary representing a data format is written to an algorithm output, the data is properly validated. This concept will become clearer when you'll read about algorithms and the way they receive and produce data. Here is just a simple illustrative example: .. testsetup:: test-output-write import numpy from beat.core.dataformat import DataFormat from beat.core.test.mocks import MockDataSink from beat.core.outputs import Output dataformat = DataFormat('/not/needed', { "x": "int32", "y": "int32", "width": "int32", "height": "int32" }) assert dataformat.valid data_sink = MockDataSink(dataformat) output = Output('test', data_sink) .. testcode:: test-output-write # suppose, for this example, `output' is provided to your algorithm output.write({ "x": numpy.int32(10), "y": numpy.int32(20), "width": numpy.int32(100), "height": numpy.int32(100), }) .. include:: links.rst