.. vim: set fileencoding=utf-8 : .. Copyright (c) 2016 Idiap Research Institute, http://www.idiap.ch/ .. .. Contact: beat.support@idiap.ch .. .. .. .. This file is part of the beat.backend.python module of the BEAT platform. .. .. .. .. Commercial License Usage .. .. Licensees holding valid commercial BEAT licenses may use this file in .. .. accordance with the terms contained in a written agreement between you .. .. and Idiap. For further information contact tto@idiap.ch .. .. .. .. Alternatively, this file may be used under the terms of the GNU Affero .. .. Public License version 3 as published by the Free Software and appearing .. .. in the file LICENSE.AGPL included in the packaging of this file. .. .. The BEAT platform is distributed in the hope that it will be useful, but .. .. WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY .. .. or FITNESS FOR A PARTICULAR PURPOSE. .. .. .. .. You should have received a copy of the GNU Affero Public License along .. .. with the BEAT platform. If not, see http://www.gnu.org/licenses/. .. .. _beat_backend_python: ========================= Python Backend for BEAT ========================= The BEAT platform is a web-based system for certifying results for software-based data-driven workflows that can be sub-divided functionally (into processing blocks). The platform takes all burden of hosting data and software away from users by providing a capable computing farm that handles both aspects graciously. Data is kept sequestered inside the platform. The user provides the description of data formats, algorithms, data flows (also known as toolchains) and experimental details (parameters), which are mashed inside the platform to produce beautiful results, easily exportable into computer graphics or tables for scientific reports. This package defines a backend to execute algorithms written in the Python programming language. Creating new database views in beat =================================== To implement a view, one needs to write a class that inherits from ``beat.backend.python.database.View``, and implement two methods: ``index()`` and ``get()``. Here are the `documentation `_ of those methods: The ``index()`` function: .. code-block:: python def index(self, root_folder, parameters): """Returns a list of (named) tuples describing the data provided by the view. The ordering of values inside the tuples is free, but it is expected that the list is ordered in a consistent manner (ie. all train images of person A, then all train images of person B, ...). For instance, assuming a view providing that kind of data: ----------- ----------- ----------- ----------- ----------- ----------- | image | | image | | image | | image | | image | | image | ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- | file_id | | file_id | | file_id | | file_id | | file_id | | file_id | ----------- ----------- ----------- ----------- ----------- ----------- ----------------------------------- ----------------------------------- | client_id | | client_id | ----------------------------------- ----------------------------------- a list like the following should be generated: [ (client_id=1, file_id=1, image=filename1), (client_id=1, file_id=2, image=filename2), (client_id=1, file_id=3, image=filename3), (client_id=2, file_id=4, image=filename4), (client_id=2, file_id=5, image=filename5), (client_id=2, file_id=6, image=filename6), ... ] DO NOT store images, sound files or data loadable from a file in the list! Store the path of the file to load instead. """ The ``get()`` function: .. code-block:: python def get(self, output, index): """Returns the data of the provided output at the provided index in the list of (named) tuples describing the data provided by the view (accessible at self.objs)""" So if we take as an example the ``atnt/5 database``, the view named ``“Train”`` is implemented like this way (note that each view comes with a documentation describing the way the different outputs are synchronised together): .. code-block:: python class Train(View): """Outputs: - image: "{{ system_user.username }}/array_2d_uint8/1" - file_id: "{{ system_user.username }}/uint64/1" - client_id: "{{ system_user.username }}/uint64/1" One "file_id" is associated with a given "image". Several "image" are associated with a given "client_id". --------------- --------------- --------------- --------------- --------------- --------------- | image | | image | | image | | image | | image | | image | --------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- --------------- | file_id | | file_id | | file_id | | file_id | | file_id | | file_id | --------------- --------------- --------------- --------------- --------------- --------------- ----------------------------------------------- ----------------------------------------------- | client_id | | client_id | ----------------------------------------------- ----------------------------------------------- """ def index(self, root_folder, parameters): Entry = namedtuple('Entry', ['client_id', 'file_id', 'image']) # Open the database and load the objects to provide via the outputs db = bob.db.atnt.Database() objs = sorted(db.objects(groups='world', purposes=None), key=lambda x: (x.client_id, x.id)) return [ Entry(x.client_id, x.id, x.make_path(root_folder, '.pgm')) for x in objs ] def get(self, output, index): obj = self.objs[index] if output == 'client_id': return { 'value': np.uint64(obj.client_id) } elif output == 'file_id': return { 'value': np.uint64(obj.file_id) } elif output == 'image': return { 'value': bob.io.base.load(obj.image) } Note that: 1) This view exactly matches the example from the documentation of the View class. In particular, ``index()`` returns a list looking like: .. code-block:: python [ (client_id=1, file_id=1, image=filename1), (client_id=1, file_id=2, image=filename2), (client_id=1, file_id=3, image=filename3), (client_id=2, file_id=4, image=filename4), (client_id=2, file_id=5, image=filename5), (client_id=2, file_id=6, image=filename6), ... (client_id=100, file_id=10000, image=filename10000), ] If there are 10000 images in the dataset, there will be 10000 entries in that list. The platform will use this information to efficiently split the jobs on several machines during the experiment. It is expected that the list is ordered in a logical order (here: entries are grouped by ``client_id``). 2) For each entry in the dataset (represented as a named tuple), all the necessary data is provided by ``index()``. For performance reasons, it is expected that we don’t need to instantiate ``bob.db.atnt.Database()`` anymore in the ``get()`` method. 3) You’re free to put any info in the index, with the names you want for the field (here for simplicity, we have one field in the tuple per output of the view, with the same name). The platform doesn’t care. 4) Some data from the database can be stored directly in the ``index`` (here: ``client_id`` and ``file_id``). For others, that require opening a file, put the filename in the ``index`` and process the file later in ``get()`` 5) The implementation of ``get()`` is straightforward: the full index is available as ``“self.objs”``, just return the data corresponding to the provided output at the given index. As for the JSON file describing the database, the format hasn’t changed. For an example of the usage of the parameters defined in the JSON file and given to ``index()``, you can look at ``mnist/4``. Once the view is written, you must index the database with the command-line tool, something like this: .. code-block:: sh ./bin/beat —prefix=… db index mydatabase/1/myview .. toctree:: api Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search`