Library structure
The library is divided into 2 main sub-packages:
- providers: managing subproductions, jobs and theirs logs data access.
- zstd: managing Zstandard compression and decompression.
Providers
Logs data files are grouped by jobs, which are grouped by subproductions, into productions.
The providers package allows managing this system at subproduction scale, using DataProvider classes.
From these classes, job entries can be accessed through JobEntry instances.
Such jobs can then provide access to their logs data through DataEntry instances.
From which, it is possible to get python file-like objects to read/write the logs data.
This tree structure can be in readonly mode, or in read/write mode. In read/write mode, it is possible to add/delete new jobs, and to add/delete new logs data to existing jobs.
LHCbDIRAC stores subproductions' jobs as ZIP files, containing logs data files (the old format). The new format stores subproductions as a single SQLite database with Zstandard compression.
Both formats use the same structure as explained above.
The following DataProvider implementations are available:
| Implementation | Description | Format | Read/Write |
|---|---|---|---|
providers.database |
SQLite Database | New | Yes |
providers.fsspec |
ZIP files | Old | No |
providers.filesystem |
Local direcory | Both | Yes |
providers.memory |
in-memory | Both | Yes |
Implementations supporting the new format use a DictProvider to store the Zstandard compression dictionaries.
These classes are used to manage dictionaries, and to prepare these for the zstd package usage.
See the API documentation for more details.
The following UML class diagram shows the structure of the providers sub-package, for reference:
Zstandard
The zstd package provides a simple interface to compress and decompress data using the Zstandard algorithm.
Supporting processing from bytes and file-like objects.
An advanced API is also exposed, allowing the usage of thread pools to speed up the compression, decompression and dictionary training processes.
See the API documentation for more details.
The following UML class diagram shows the structure of the zstd sub-package, for reference:
