Getting started with silx.io ============================ This tutorial explains how to read data files using the :meth:`silx.io.open` function. The target audience are developers without knowledge of the *h5py* library. If you are already familiar with *h5py*, you just need to know that the :meth:`silx.io.open` function returns objects that mimic *h5py* file objects, and that the main supported file formats are: - HDF5 - all formats supported by the *FabIO* library - SPEC data files Knowledge about the python *dictionary* type and the numpy *ndarray* type are prerequisites for this tutorial. Background ---------- In the past, it was necessary to learn how to use multiple libraries to read multiple data formats. The library *FabIO* was designed to read images in many formats, but not to read more heterogeneous formats, such as *HDF5* or *SPEC*. To read *SPEC* data files in Python, a common solution was to use the *PyMca* module :mod:`PyMca5.PyMcaIO.specfilewrapper`. Regarding HDF5 files, the de-facto standard for reading them in Python is to use the *h5py* library. *silx* tries to address this situation by providing a unified way to read all data formats supported at the ESRF. Today, HDF5 is the preffered format to store data for many scientific institutions, including most synchrotrons. So it was decided to provide tools for reading data that mimic the *h5py* library's API. Definitions ----------- HDF5 ++++ The *HDF5* format is a *hierarchical data format*, designed to store and organize large amounts of data. A HDF5 file contains a number of *datasets*, which are multidimensional arrays of a homogeneous type. These datasets are stored in container structures called *groups*. Groups can also be stored in other groups, allowing to define a hierarchical tree structure. Both datasets and groups may have *attributes* attached to them. Attributes are used to document the object. They are similar to datasets in several ways (data container of homogeneous type), but they are typically much smaller. It is a common analogy to compare a HDF5 file to a filesystem. Groups are analogous to directories, while datasets are analogous to files, and attributes are analogous to file metadata (creation date, last modification...). .. image:: img/silx_view_edf.png :width: 400px h5py ++++ The *h5py* library is a Pythonic interface to the `HDF5`_ binary data format. It exposes an HDF5 group as a python object that resembles a python dictionary, and an HDF5 dataset or attribute as an object that resembles a numpy array. API description --------------- All main objects, File, Group and Dataset, share the following attributes: - :attr:`attrs`: Attributes, as a dictionary of metadata for the group or dataset. - :attr:`basename`: String giving the basename of this group or dataset. - :attr:`name`: String giving the full path to this group or dataset, relative to the root group (file). - :attr:`file`: File object at the root of the tree structure containing this group or dataset. - :attr:`parent`: Group object containing this group or dataset. File object +++++++++++ The API of the file objects returned by the :meth:`silx.io.open` function tries to be as close as possible to the API of the :class:`h5py.File` objects used to read HDF5 data. A h5py file is a group with just a few extra attributes and methods. The objects defined in `silx.io` implement a subset of these attributes and methods: - :attr:`filename`: Name of the file on disk. - :attr:`mode`: String indicating if the file is open in read mode ("r") or write mode ("w"). :meth:`silx.io.open` always returns objects in read mode. - :meth:`close`: Close this file. All open objects will become invalid. The :attr:`parent` of a file is `None`, and its :attr:`name` is an empty string. Group object ++++++++++++ Group objects behave like python dictionaries. You can iterate over a group's :meth:`keys`, which are the names of the objects encapsulated by the group (datasets and sub-groups). The :meth:`values` method returns an iterator over the encapsulated objects. The :meth:`items` method returns an iterator over `(name, value)` pairs. Groups provide a :meth:`get` method that retrieves an item, or information about an item. Like standard python dictionaries, a `default` parameter can be used to specify a value to be returned if the given name is not a member of the group. Two methods are provided to visit recursively all members of a group, :meth:`visit` and :meth:`visititems`. The former takes as argument a *callable* with the signature ``callable(name) -> None or return value``. The latter takes as argument a *callable* with the signature ``callable(name, object) -> None or return value`` (``object`` being a a group or dataset instance.) Example ------- Accessing data ++++++++++++++ In this first example, we open a Spec data file and we print some of its information. .. code-block:: python >>> import silx.io >>> sf = silx.io.open("data/CuZnO_2.spec") >>> print(sf.keys()) ['1.1', '2.1', '3.1', '4.1', '5.1', '6.1', '7.1', ...] >>> print(sf["1.1"]) We just opened a file, keeping a reference to the file object as ``sf``. We then printed all items contained in this root group. We can see that all these items are groups. Lets looks at what is inside these groups, and find datasets: .. code-block:: python >>> grp = sf["2.1"] ... for name in grp: ... item = grp[name] ... print("Found item " + name) ... if silx.io.is_dataset(item): ... print("'%s' is a dataset.\n" % name) ... elif silx.io.is_group(item): ... print("'%s' is a group.\n" % name) ... Found item title title is a dataset. Found item start_time start_time is a dataset. Found item instrument instrument is a group. Found item measurement measurement is a group. Found item sample sample is a group. We could have replaced the first three lines with this single line, by iterating over the iterator returned by the group method :meth:`items`: .. code-block:: python >>> for name, item in sf["2.1"].items(): ... In addition to :meth:`silx.io.is_group` and :meth:`silx.io.is_dataset`, you can also use :meth:`silx.io.is_file` and :meth:`silx.io.is_softlink`. Let's look at a dataset: .. code-block:: python >>> print(sf["2.1/title"]) As you can see, printing a dataset does not print the data itself, it only print a representation of the dataset object. The information printed tells us that the object is similar to a numpy array, with a *shape* and a *type*. In this case, we are dealing with a scalar dataset, so we can use the same syntax as in numpy to access the scalar value, ``result = dset[()]``: .. code-block:: python >>> print(sf["2.1/title"][()]) 2 ascan phi 0.61 1.61 20 1 Similarly, you need to use numpy slicing to access values in numeric array: .. code-block:: python >>> print (sf["2.1/measurement/Phi"]) >>> print (sf["2.1/measurement/Phi"][0:10]) [ 0.61000001 0.66000003 0.70999998 0.75999999 0.81 0.86000001 0.91000003 0.95999998 1.00999999 1.05999994] >>> entire_phi_array = sf["2.1/measurement/Phi"][:] Here we could read the entire array by slicing it with ``[:]``, because we know it is a 1D array. For a 2D array, the slicing argument would have been ``[:, :]``. For a dataset of unknown dimensionality (including scalar datasets), the ``Ellipsis`` object (represented by ``...``) can be used to slice the object. .. code-block:: python >>> print(sf["2.1/title"][...]) 2 ascan phi 0.61 1.61 20 1 >>> print (sf["2.1/measurement/Phi"][...]) [ 0.61000001 0.66000003 0.70999998 0.75999999 0.81 0.86000001 0.91000003 0.95999998 1.00999999 1.05999994 1.11000001 1.15999997 1.21000004 1.25999999 1.30999994 1.36000001 1.40999997 1.46000004 1.50999999 1.55999994 1.61000001] To read more about the usage of ``Ellipsis`` to slice arrays, see `Indexing numpy arrays `_ in the scipy documentation. Note that slicing a scalar dataset with ``[()]`` is not strictly equivalent to slicing with ``[...]``. The former gives you the actual scalar value in the dataset, while the latter always gives you an array object, which happens to be 0D in the case of a scalar. >>> sf["2.1/instrument/positioners/Delta"][()] 0.0 >>> sf["2.1/instrument/positioners/Delta"][...] array(0.0, dtype=float32) Closing the file ++++++++++++++++ You should always make sure to close the files that you opened. The simple way of closing a file is to call its :meth:`close` method. .. code-block:: python import silx.io sf = silx.io.open("data/CuZnO_2.spec") # read the information you need... maxPhi = sf["2.1/measurement/Phi"][...].max() sf.close() The drawback of this method is that, if an error is raised while processing the file, the program might never reach the ``sf.close()`` line. Leaving files open can cause various issues for the rest of your program, such as consuming memory, not being able to reopen the file when you need it... The best way to ensure the file is always properly closed is to use the file inside its context manager: .. code-block:: python import silx.io with silx.io.open("data/CuZnO_2.spec") as sf: # read the information you need... maxPhi = sf["2.1/measurement/Phi"][...].max() Additional resources -------------------- - `h5py documentation `_ - `Formats supported by FabIO `_ - `Spec file h5py-like structure `_ - `HDF5 format documentation `_