API
Modules
Types and constants
LegendHDF5IO.AbstractLH5TableLegendHDF5IO.AbstractLH5VectorLegendHDF5IO.Geant4HDF5InputLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ArrayLegendHDF5IO.LH5ChainedArrayLegendHDF5IO.LH5ChainedTableLegendHDF5IO.LH5FileChainLegendHDF5IO.LH5JoinedTableLegendHDF5IO.LH5LazyBroadcastLegendHDF5IO.LH5LazyColumnsLegendHDF5IO.LH5LazyTableLegendHDF5IO.LH5PartArrayLegendHDF5IO.LH5TypedTableLegendHDF5IO.LHDataStoreLegendHDF5IO.compressLegendHDF5IO.max_chunk_bytesLegendHDF5IO.max_compressed_chunk_bytesLegendHDF5IO.max_openLegendHDF5IO.rows_per_blockLegendHDF5IO.scatter_bulk_max_bytesLegendHDF5IO.scatter_bulk_max_wasteLegendHDF5IO.usechunks
Functions and macros
LegendDataTypes.readdataLegendDataTypes.writedataLegendHDF5IO._chain_columnsLegendHDF5IO._typed_tableLegendHDF5IO.add_entries!LegendHDF5IO.add_entries!LegendHDF5IO.delete_entry!LegendHDF5IO.lh5openLegendHDF5IO.lh5openLegendHDF5IO.lh5openLegendHDF5IO.lh5openLegendHDF5IO.register_datatype!
Documentation
LegendHDF5IO.AbstractLH5Table — Type
AbstractLH5Table{R<:NamedTuple} <: AbstractLH5Vector{R}Supertype of the lazy tables of LH5 data: LH5LazyTable, LH5JoinedTable, LH5ChainedTable and LH5TypedTable.
A lazy table is a vector of NamedTuple rows. Its column names are known up front, the element type of a column only once the column is opened, so the row type R is the abstract NamedTuple, except for typed tables. Accessing a column, with tbl.colname or Tables.getcolumn, opens that column and no other, and Tables.columns(tbl) is a dictionary of the columns (see LH5LazyColumns). A PropertyFunction broadcast over a lazy table is an LH5LazyBroadcast that reads only the columns the function uses, a pure column accessor returns the column, and selections return typed tables.
Treating the table as an array opens all columns. Indexing with a range, index vector or mask reads the selected rows into a StructArray, and iteration reads the rows in blocks, as described for AbstractLH5Vector. vcat of lazy tables chains them. Concatenation with other arrays is refused if a table is among the first two arguments. A table behind two other arrays is concatenated row by row. StructArray(tbl), TypedTables.Table(tbl) and TypedTables.FlexTable(tbl) are refused as well: they would append through the columns without the table's checks, and a StructArray of a column read eagerly would slice and append rows with a stale row type.
Lazy tables stop working when the file they read from is closed, and a handle stops working when its table is appended to or its columns are changed through another handle: its reads and appends then throw, and the table must be reopened. Its own appends keep it current. The rows read from a table do not depend on it.
Implementation
Subtypes implement _table_colnames(tbl), _column(tbl, name) and length(tbl), and may specialize _typed_table(tbl), _getindex_rows(tbl, idxs) and the block iteration, _next_block.
LegendHDF5IO.AbstractLH5Vector — Type
AbstractLH5Vector{T} <: AbstractVector{T}Supertype of the lazy vectors of LH5 data, which read their elements in blocks: the lazy tables, AbstractLH5Table, and lazy broadcasts over typed tables, LH5LazyBroadcast.
Indexing with a range, index vector or mask reads the selected elements, and v[:] reads them all. Iteration reads the elements in blocks of a few thousand (see LegendHDF5IO.rows_per_block), so loops, comprehensions and map read neither one element at a time nor everything at once. filter, searches such as findall, findfirst and findmax, and reductions go through iteration, while sort and sortperm read the vector first.
Implementation
Subtypes implement length(v) and _getindex_rows(v, idxs), which reads the elements at idxs, and may specialize the block iteration, _first_block_state(v) and _next_block(v, state).
LegendHDF5IO.Geant4HDF5Input — Type
Geant4HDF5Input <: AbstractLegendInputInput wrapper for Geant4 HDF5 files in GEARS or g4simple layout.
Open a file with open(filename, Geant4HDF5Input) or its do-block form, then read(input) returns a table of hits.
LegendHDF5IO.LH5Array — Type
LH5Array{T, N} <: AbstractArray{T, N}Array wrapper for an HDF5.Dataset following the LEGEND data format, as in ".lh5" files.
getindex reads only the requested part of the on-disk array. append! extends the on-disk array along its last dimension, which requires a chunked dataset.
LH5Array implements the DiskArrays.jl interface: views, iteration and reductions read the data chunk by chunk, broadcasts are lazy, and setindex! writes back into the dataset within its current size.
Default constructors
LH5Array{T, N}(ds::HDF5.Dataset)
LH5Array{T}(ds::HDF5.Dataset)
LH5Array(ds::Union{HDF5.Dataset, HDF5.H5DataStore})Example
julia> using HDF5
julia> f = h5open("path/to/lh5/file", "r+")
julia> lh = LH5Array(f["path/to/HDF5/Dataset"])
[...]
julia> x = lh[1:10] # load the first 10 elements of the on-disk array
[...]
julia> append!(lh, x) # append them, the dataset must be chunked
[...]LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray{<:Bool}})return a LH5Array with dimensions equal to that of ds and element type Bool. Reading from it gives an Array{Bool}.
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray{<:Enum}})return an in-memory array of Enum values.
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray{<:Tuple}})return an Array of NTuples
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray{<:StaticVector}})return an in-memory array of SVectors.
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray})return a LH5Array with dimensions equal to that of ds and element type equal to eltype(ds) * u
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:Bool})return a value with type Bool
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:Enum})return a value of the given Enum type.
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:String})return a String.
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:Symbol})return a Symbol.
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:Tuple})return an Tuple
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:RealQuantity})return a value with type RealQuantity
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.H5DataStore, ::Type{<:AbstractVector{<:AbstractVector{<:RealQuantity}}})return a VectorOfVectors object where data is an LH5Array (see VectorOfArrays)
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.H5DataStore, ::Type{<:AbstractVector{<:RDWaveform}})return an ArrayOfRDWaveforms where the field signal is either a VectorOfSimilarVectors with an LH5Array as data or VectorOfVectors with an LH5Array as data (see ArrayOfRDWaveforms and ArraysOfArrays)
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.H5DataStore, ::Type{<:AbstractEncodedArray{<:RealQuantity, 1}})return an EncodedArray
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.H5DataStore, ::Type{<:VectorOfEncodedArrays{T, 1} where {T}})return a VectorOfEncodedArrays
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.H5DataStore, ::Type{<:VectorOfEncodedSimilarArrays{T, 1} where {T}})return a VectorOfEncodedSimilarArrays
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.H5DataStore, ::Type{<:Histogram{<:RealQuantity}})return a Histogram.
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.H5DataStore, ::Type{<:AbstractArrayOfSimilarArrays{<:RealQuantity}})return an ArraysOfSimilarArrays where the field data is a LH5Array (see ArraysOfSimilarArrays)
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.Dataset, ::Type{<:NamedTuple{T}}) where Treturn a NamedTuple where each field is the output of LH5Array applied to it.
LegendHDF5IO.LH5Array — Method
LH5Array(ds::HDF5.H5DataStore, ::Type{<:StructArray{<:NamedTuple{(T)}}}) where Treturn an LH5LazyTable that opens the columns named in T on demand.
LegendHDF5IO.LH5ChainedArray — Type
LH5ChainedArray{T, N} <: DiskArrays.AbstractDiskArray{T, N}A read-only column of an LH5ChainedTable that reads through the disk arrays of its parts.
LegendHDF5IO.LH5ChainedTable — Type
LH5ChainedTable <: AbstractLH5TableTables one after the other, without reading them. vcat and reduce(vcat, ...) of lazy tables return one, as do ArraysOfArrays.vecflattened of a vector of lazy tables and indexing a file chain with the path of a table (see LH5FileChain).
Rows are read from the parts that hold them, so first(tbl, n) reads the first parts only. Accessing a column concatenates the columns of all parts lazily, which opens the column in each part. The parts must have the same columns, and rows take the column order of the first part. Chained tables are read-only, as are their columns that read from disk. Columns held in memory, such as those read eagerly, accept writes, which do not reach the files.
LegendHDF5IO.LH5FileChain — Type
LH5FileChainLH5 files of the same structure, read as one. lh5open of a vector of file names returns one.
chain[path] concatenates the table or array at path over all files, as an LH5ChainedTable for a table. Files are opened when they are read from, and at most max_open of them stay open. For an array, chain[path] needs its size in every file and so opens all of them right away, as does accessing a column of a chained table. Objects from the chain stop working when the chain is closed.
The first file defines the structure: the column names of tables, and the datatypes, units, codecs, element types and inner sizes of what they hold. Other files are checked against it when an array or column is first opened in them, so they need to agree only on what is used. Chains are read-only.
LegendHDF5IO.LH5JoinedTable — Type
LH5JoinedTable <: AbstractLH5TableThe columns of several tables side by side, without reading them. PropertyFunctions.innermerge returns one when its first or second table is lazy, and so does Tables.materializer of a lazy table when given columns that read from disk.
The parts are lazy tables or NamedTuples of columns. They must have the same length, which is compared when the length of the join is needed, so joining chained tables opens no file. A column comes from the last part that has it, and the columns are ordered by their first occurrence, as in merge. append! refuses joined tables. Their columns are those of the parts, so writing to a column writes to its part.
LegendHDF5IO.LH5LazyBroadcast — Method
LH5LazyBroadcast(f, tbl::LH5TypedTable, reads::LH5TypedTable = tbl)f broadcast over tbl, reading the rows from reads, a selection of the columns of tbl that f uses.
LegendHDF5IO.LH5LazyColumns — Type
LH5LazyColumns <: AbstractDict{Symbol, AbstractVector}The columns of a lazy table (see AbstractLH5Table), as Tables.columns returns them: an ordered, read-only dictionary from column name to column that opens a column when it is looked up. cols.energy and cols[:energy] are the same lookup. Nested tables are lazy tables themselves.
LegendHDF5IO.LH5LazyTable — Type
LH5LazyTable <: AbstractLH5TableA table stored in an LH5 file whose columns are opened on demand. Reading a table returns one. It behaves as described for AbstractLH5Table, and append! appends to the stored columns.
Extended help
Opening a column reads its "datatype" attribute, and for a vector of vectors its cumulative_length as well. The attribute lives in the dataset's object header, which HDF5 usually places next to the column's data rather than next to the other headers, so opening a wide table eagerly costs one scattered read per column plus the cumulative_length arrays. On a network file system that is the dominant cost of opening a file whose columns are mostly unused.
Neither the column names nor the column types are type parameters, so opening a table compiles no code specialized on them. Column access is therefore not inferable, which costs little in practice: the datatype attributes are only read at run time, so code that loops over rows has to pass the selected columns through a function boundary either way.
LegendHDF5IO.LH5PartArray — Type
LH5PartArray{T, N} <: DiskArrays.AbstractDiskArray{T, N}A dataset in one file of an LH5FileChain, which reopens the file when it is read.
LegendHDF5IO.LH5TypedTable — Method
LH5TypedTable(tbl::AbstractLH5Table, names, paths)The columns of tbl at paths, each a tuple of the names leading to a column, as a typed table with the columns named names.
LegendHDF5IO.LHDataStore — Type
LHDataStore <: AbstractDict{String,Any}Dictionary wrapper for an HDF5.H5DataStore (typically an HDF5.File, or an HDF5.Group) following the LEGEND data format, as in ".lh5" files.
Constructor:
LHDataStore(h5ds::HDF5.H5DataStore, usechunks::Bool, compress::Symbol = :none)To open ".lh5" files directly, without HDF5.h5open, use lh5open.
getindex(lh::LHDataStore, s) returns the output of LH5Array applied to lh.data_store[s], so arrays other than those of Enums, StaticVectors, NTuples and Measurements are wrapped lazily. setindex! creates and writes HDF5.Groups and HDF5.Datasets.
With usechunks = true, array datasets are created chunked, so append! can extend them. The first array written sets the chunk size, up to LegendHDF5IO.max_chunk_bytes. With compress set to :zstd or :deflate, array datasets are chunked and compressed, with chunks of at most LegendHDF5IO.max_compressed_chunk_bytes.
Supported value types include RealQuantity and arrays thereof, Bool, String, Symbol, Enums, (arrays of) NTuples and StaticVectors, ArrayOfSimilarArrays, VectorOfVectors, encoded arrays, NamedTuples, tables, AbstractVector{<:RDWaveform} and Histogram. For arrays, the last axis is assumed to correspond to the event number index.
Example
julia> lhf = lh5open("path/to/lh5/file", "cw")
julia> lhf["raw"]
[...]
julia> using Unitful
julia> x = rand(100) * u"ns"
julia> lhf["new"] = x
[...]LegendHDF5IO.compress — Constant
LegendHDF5IO.compress[]Default of the compress argument of lh5open, :none: the compression of newly written array datasets, :zstd or :deflate. A setting, see the Settings section of the guide, where "true" and "false" stand for :zstd and :none.
LegendHDF5IO.max_chunk_bytes — Constant
LegendHDF5IO.max_chunk_bytes[]Size in bytes up to which the first array written to a chunked, uncompressed dataset sets the chunk size, 16 MiB by default. A larger first write is split evenly into n chunks of at most this size, which leave fewer than n rows unused, and an empty one gets chunks of this size. An append then wastes at most one partially filled chunk. The default suits the block sizes of cluster file systems and does not depend on the file system, so that files written on one system read well on others. A setting, see the Settings section of the guide.
LegendHDF5IO.max_compressed_chunk_bytes — Constant
LegendHDF5IO.max_compressed_chunk_bytes[]Size in bytes up to which compressed datasets are chunked, 1 MiB by default. A compressed chunk is decompressed as a whole, so compressed chunks stay small enough that a partial read decompresses little more than it needs. A setting, see the Settings section of the guide.
LegendHDF5IO.max_open — Constant
LegendHDF5IO.max_open[]Default of the max_open argument of lh5open for file chains, 64: the number of files a chain keeps open at a time. A setting, see the Settings section of the guide.
LegendHDF5IO.rows_per_block — Constant
LegendHDF5IO.rows_per_block[]Number of rows that iteration over a lazy table reads at a time, 4096 by default. A setting, see the Settings section of the guide.
LegendHDF5IO.scatter_bulk_max_bytes — Constant
LegendHDF5IO.scatter_bulk_max_bytes[]Size up to which a scattered read reads the whole index span its events cover, instead of reading the events themselves.
Reading the span costs one read operation. Reading the events costs one scattered selection, or one read per run of adjacent events, depending on element type, rank and layout. With the default nothing the limit depends on the file system the file lives on: 1 MiB on local storage and 8 MiB on network file systems such as GPFS, NFS or Lustre, where fewer and larger reads pay off. Variable-length elements like strings are always read selectively. A setting, see the Settings section of the guide, where "auto" stands for nothing.
See also LegendHDF5IO.scatter_bulk_max_waste.
LegendHDF5IO.scatter_bulk_max_waste — Constant
LegendHDF5IO.scatter_bulk_max_waste[]Factor by which a scattered read may exceed the requested amount of data when it reads the whole index span its events cover.
Densely covered spans are read as a whole however large they are, so raising this trades reading more data for fewer read operations. With the default nothing the factor is 4 on local storage and 16 on network file systems. Spans up to LegendHDF5IO.scatter_bulk_max_bytes are read as a whole regardless. A setting, see the Settings section of the guide, where "auto" stands for nothing.
LegendHDF5IO.usechunks — Constant
LegendHDF5IO.usechunks[]Default of the usechunks argument of lh5open, false: whether newly written array datasets are chunked, so that append! can extend them. A setting, see the Settings section of the guide.
LegendDataTypes.readdata — Method
readdata(input::Union{HDF5.Dataset, HDF5.H5DataStore}, name::AbstractString)
readdata(input, name, datatype::Type)Read the value stored under name from input via LH5Array and materialize it in memory.
The "datatype" attribute determines the value type, unless datatype is given.
LegendDataTypes.writedata — Function
writedata(output::Union{HDF5.H5DataStore, LHDataStore}, name::AbstractString, x,
fulldatatype::DataType = typeof(x))Write the value x under name to output, via create_entry, with the usechunks and compress options of an LHDataStore.
Passing fulldatatype overrides the "datatype" attribute written for x, and passing Nothing omits it.
LegendHDF5IO._chain_columns — Method
_chain_columns(cols::AbstractVector)Concatenate the columns of the parts of a chained table, lazily and read-only for columns that read from disk.
LegendHDF5IO._typed_table — Method
_typed_table(tbl::AbstractLH5Table)Open every column and return them as a StructArray, which indexing and appending work on.
LegendHDF5IO.add_entries! — Method
add_entries!(lhd::LHDataStore, i::AbstractString, src::NamedTuple)Add the entries of src to the NamedTuple at lhd[i].
LegendHDF5IO.add_entries! — Method
add_entries!(lhd::LHDataStore, i::AbstractString, src::StructArray)Add the columns of src to the table at lhd[i].
LegendHDF5IO.delete_entry! — Method
delete_entry!(lhd::LHDataStore, i::AbstractString)Remove the dataset lhd[i] and adjust the datatype of the parent if necessary. Supported are entries of NamedTuples, tables and HDF5.Groups.
LegendHDF5IO.lh5open — Function
lh5open(f, filename::AbstractString, access::AbstractString = "r"; kwargs...)Open the LEGEND HDF5 file filename, apply f to the resulting LHDataStore and close the file afterwards, returning the result of f.
LegendHDF5IO.lh5open — Function
lh5open(filename::AbstractString, access::AbstractString = "r";
usechunks::Bool = LegendHDF5IO.usechunks[], compress = LegendHDF5IO.compress[])Open a LEGEND HDF5 file and return an LHDataStore object.
With usechunks = true, newly written array datasets are chunked and can be extended with append!. compress selects their compression: true or :zstd for Zstandard, :deflate (alias :gzip) for shuffle plus deflate. Compressed datasets are always chunked. Both default to the settings LegendHDF5IO.usechunks and LegendHDF5IO.compress. LEGEND HDF5 files typically use the file extension ".lh5".
LegendHDF5IO.lh5open — Method
lh5open(filenames::AbstractVector{<:AbstractString}; max_open::Integer = LegendHDF5IO.max_open[])Open the LEGEND HDF5 files filenames, which must have the same structure, as one LH5FileChain, for reading. At most max_open of them stay open at a time, by default the setting LegendHDF5IO.max_open.
LegendHDF5IO.lh5open — Method
lh5open(f, filenames::AbstractVector{<:AbstractString}; kwargs...)Open the files filenames as one LH5FileChain, apply f to it and close it afterwards, returning the result of f.
LegendHDF5IO.register_datatype! — Method
LegendHDF5IO.register_datatype!(name::AbstractString, ::Type{T})
LegendHDF5IO.register_datatype!(::Type{T})Register the LH5 datatype string name for type T, for reading and writing. Without name, the canonical datatype string of T is used (e.g. for Enum types).
Each name can only be registered for one type, and each type only under one name. Names consist of letters and underscores. Built-in datatype names like "real" or "array<1>{real}" are reserved. Conflicting registrations throw an ArgumentError, while repeating a registration has no effect.
The registry is not part of the precompilation cache, so register from the __init__ function of your module or package extension.
For a type T without built-in support, also define
LegendHDF5IO.LH5Array(ds::HDF5.Dataset, ::Type{<:T})
LegendHDF5IO.create_entry(parent::LHDataStore, name::AbstractString, x::T; kwargs...)