API

Modules

    Types and constants

    Functions and macros

    Documentation

    LegendHDF5IO.AbstractLH5Table — Type
    AbstractLH5Table{R<:NamedTuple} <: AbstractLH5Vector{R}

    Supertype of the lazy tables of LH5 data: LH5LazyTable, LH5JoinedTable, LH5ChainedTable and LH5TypedTable.

    A lazy table is a vector of NamedTuple rows. Its column names are known up front, the element type of a column only once the column is opened, so the row type R is the abstract NamedTuple, except for typed tables. Accessing a column, with tbl.colname or Tables.getcolumn, opens that column and no other, and Tables.columns(tbl) is a dictionary of the columns (see LH5LazyColumns). A PropertyFunction broadcast over a lazy table is an LH5LazyBroadcast that reads only the columns the function uses, a pure column accessor returns the column, and selections return typed tables.

    Treating the table as an array opens all columns. Indexing with a range, index vector or mask reads the selected rows into a StructArray, and iteration reads the rows in blocks, as described for AbstractLH5Vector. vcat of lazy tables chains them. Concatenation with other arrays is refused if a table is among the first two arguments. A table behind two other arrays is concatenated row by row. StructArray(tbl), TypedTables.Table(tbl) and TypedTables.FlexTable(tbl) are refused as well: they would append through the columns without the table's checks, and a StructArray of a column read eagerly would slice and append rows with a stale row type.

    Lazy tables stop working when the file they read from is closed, and a handle stops working when its table is appended to or its columns are changed through another handle: its reads and appends then throw, and the table must be reopened. Its own appends keep it current. The rows read from a table do not depend on it.

    Implementation

    Subtypes implement _table_colnames(tbl), _column(tbl, name) and length(tbl), and may specialize _typed_table(tbl), _getindex_rows(tbl, idxs) and the block iteration, _next_block.

    source
    LegendHDF5IO.AbstractLH5Vector — Type
    AbstractLH5Vector{T} <: AbstractVector{T}

    Supertype of the lazy vectors of LH5 data, which read their elements in blocks: the lazy tables, AbstractLH5Table, and lazy broadcasts over typed tables, LH5LazyBroadcast.

    Indexing with a range, index vector or mask reads the selected elements, and v[:] reads them all. Iteration reads the elements in blocks of a few thousand (see LegendHDF5IO.rows_per_block), so loops, comprehensions and map read neither one element at a time nor everything at once. filter, searches such as findall, findfirst and findmax, and reductions go through iteration, while sort and sortperm read the vector first.

    Implementation

    Subtypes implement length(v) and _getindex_rows(v, idxs), which reads the elements at idxs, and may specialize the block iteration, _first_block_state(v) and _next_block(v, state).

    source
    LegendHDF5IO.Geant4HDF5Input — Type
    Geant4HDF5Input <: AbstractLegendInput

    Input wrapper for Geant4 HDF5 files in GEARS or g4simple layout.

    Open a file with open(filename, Geant4HDF5Input) or its do-block form, then read(input) returns a table of hits.

    source
    LegendHDF5IO.LH5Array — Type
    LH5Array{T, N} <: AbstractArray{T, N}

    Array wrapper for an HDF5.Dataset following the LEGEND data format, as in ".lh5" files.

    getindex reads only the requested part of the on-disk array. append! extends the on-disk array along its last dimension, which requires a chunked dataset.

    LH5Array implements the DiskArrays.jl interface: views, iteration and reductions read the data chunk by chunk, broadcasts are lazy, and setindex! writes back into the dataset within its current size.

    Default constructors

    LH5Array{T, N}(ds::HDF5.Dataset)
    LH5Array{T}(ds::HDF5.Dataset)
    LH5Array(ds::Union{HDF5.Dataset, HDF5.H5DataStore})

    Example

    julia> using HDF5
    julia> f = h5open("path/to/lh5/file", "r+")
    julia> lh = LH5Array(f["path/to/HDF5/Dataset"])
    [...]
    julia> x = lh[1:10]     # load the first 10 elements of the on-disk array
    [...]
    julia> append!(lh, x)   # append them, the dataset must be chunked
    [...]
    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray{<:Bool}})

    return a LH5Array with dimensions equal to that of ds and element type Bool. Reading from it gives an Array{Bool}.

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray{<:Enum}})

    return an in-memory array of Enum values.

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray{<:StaticVector}})

    return an in-memory array of SVectors.

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.Dataset, ::Type{<:AbstractArray})

    return a LH5Array with dimensions equal to that of ds and element type equal to eltype(ds) * u

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.H5DataStore, ::Type{<:AbstractVector{<:AbstractVector{<:RealQuantity}}})

    return a VectorOfVectors object where data is an LH5Array (see VectorOfArrays)

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.H5DataStore, ::Type{<:AbstractVector{<:RDWaveform}})

    return an ArrayOfRDWaveforms where the field signal is either a VectorOfSimilarVectors with an LH5Array as data or VectorOfVectors with an LH5Array as data (see ArrayOfRDWaveforms and ArraysOfArrays)

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.H5DataStore, ::Type{<:AbstractEncodedArray{<:RealQuantity, 1}})

    return an EncodedArray

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.H5DataStore, ::Type{<:VectorOfEncodedArrays{T, 1} where {T}})

    return a VectorOfEncodedArrays

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.H5DataStore, ::Type{<:VectorOfEncodedSimilarArrays{T, 1} where {T}})

    return a VectorOfEncodedSimilarArrays

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.H5DataStore, ::Type{<:AbstractArrayOfSimilarArrays{<:RealQuantity}})

    return an ArraysOfSimilarArrays where the field data is a LH5Array (see ArraysOfSimilarArrays)

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.Dataset, ::Type{<:NamedTuple{T}}) where T

    return a NamedTuple where each field is the output of LH5Array applied to it.

    source
    LegendHDF5IO.LH5Array — Method
    LH5Array(ds::HDF5.H5DataStore, ::Type{<:StructArray{<:NamedTuple{(T)}}}) where T

    return an LH5LazyTable that opens the columns named in T on demand.

    source
    LegendHDF5IO.LH5ChainedTable — Type
    LH5ChainedTable <: AbstractLH5Table

    Tables one after the other, without reading them. vcat and reduce(vcat, ...) of lazy tables return one, as do ArraysOfArrays.vecflattened of a vector of lazy tables and indexing a file chain with the path of a table (see LH5FileChain).

    Rows are read from the parts that hold them, so first(tbl, n) reads the first parts only. Accessing a column concatenates the columns of all parts lazily, which opens the column in each part. The parts must have the same columns, and rows take the column order of the first part. Chained tables are read-only, as are their columns that read from disk. Columns held in memory, such as those read eagerly, accept writes, which do not reach the files.

    source
    LegendHDF5IO.LH5FileChain — Type
    LH5FileChain

    LH5 files of the same structure, read as one. lh5open of a vector of file names returns one.

    chain[path] concatenates the table or array at path over all files, as an LH5ChainedTable for a table. Files are opened when they are read from, and at most max_open of them stay open. For an array, chain[path] needs its size in every file and so opens all of them right away, as does accessing a column of a chained table. Objects from the chain stop working when the chain is closed.

    The first file defines the structure: the column names of tables, and the datatypes, units, codecs, element types and inner sizes of what they hold. Other files are checked against it when an array or column is first opened in them, so they need to agree only on what is used. Chains are read-only.

    source
    LegendHDF5IO.LH5JoinedTable — Type
    LH5JoinedTable <: AbstractLH5Table

    The columns of several tables side by side, without reading them. PropertyFunctions.innermerge returns one when its first or second table is lazy, and so does Tables.materializer of a lazy table when given columns that read from disk.

    The parts are lazy tables or NamedTuples of columns. They must have the same length, which is compared when the length of the join is needed, so joining chained tables opens no file. A column comes from the last part that has it, and the columns are ordered by their first occurrence, as in merge. append! refuses joined tables. Their columns are those of the parts, so writing to a column writes to its part.

    source
    LegendHDF5IO.LH5LazyBroadcast — Method
    LH5LazyBroadcast(f, tbl::LH5TypedTable, reads::LH5TypedTable = tbl)

    f broadcast over tbl, reading the rows from reads, a selection of the columns of tbl that f uses.

    source
    LegendHDF5IO.LH5LazyColumns — Type
    LH5LazyColumns <: AbstractDict{Symbol, AbstractVector}

    The columns of a lazy table (see AbstractLH5Table), as Tables.columns returns them: an ordered, read-only dictionary from column name to column that opens a column when it is looked up. cols.energy and cols[:energy] are the same lookup. Nested tables are lazy tables themselves.

    source
    LegendHDF5IO.LH5LazyTable — Type
    LH5LazyTable <: AbstractLH5Table

    A table stored in an LH5 file whose columns are opened on demand. Reading a table returns one. It behaves as described for AbstractLH5Table, and append! appends to the stored columns.

    Extended help

    Opening a column reads its "datatype" attribute, and for a vector of vectors its cumulative_length as well. The attribute lives in the dataset's object header, which HDF5 usually places next to the column's data rather than next to the other headers, so opening a wide table eagerly costs one scattered read per column plus the cumulative_length arrays. On a network file system that is the dominant cost of opening a file whose columns are mostly unused.

    Neither the column names nor the column types are type parameters, so opening a table compiles no code specialized on them. Column access is therefore not inferable, which costs little in practice: the datatype attributes are only read at run time, so code that loops over rows has to pass the selected columns through a function boundary either way.

    source
    LegendHDF5IO.LH5TypedTable — Method
    LH5TypedTable(tbl::AbstractLH5Table, names, paths)

    The columns of tbl at paths, each a tuple of the names leading to a column, as a typed table with the columns named names.

    source
    LegendHDF5IO.LHDataStore — Type
    LHDataStore <: AbstractDict{String,Any}

    Dictionary wrapper for an HDF5.H5DataStore (typically an HDF5.File, or an HDF5.Group) following the LEGEND data format, as in ".lh5" files.

    Constructor:

    LHDataStore(h5ds::HDF5.H5DataStore, usechunks::Bool, compress::Symbol = :none)

    To open ".lh5" files directly, without HDF5.h5open, use lh5open.

    getindex(lh::LHDataStore, s) returns the output of LH5Array applied to lh.data_store[s], so arrays other than those of Enums, StaticVectors, NTuples and Measurements are wrapped lazily. setindex! creates and writes HDF5.Groups and HDF5.Datasets.

    With usechunks = true, array datasets are created chunked, so append! can extend them. The first array written sets the chunk size, up to LegendHDF5IO.max_chunk_bytes. With compress set to :zstd or :deflate, array datasets are chunked and compressed, with chunks of at most LegendHDF5IO.max_compressed_chunk_bytes.

    Supported value types include RealQuantity and arrays thereof, Bool, String, Symbol, Enums, (arrays of) NTuples and StaticVectors, ArrayOfSimilarArrays, VectorOfVectors, encoded arrays, NamedTuples, tables, AbstractVector{<:RDWaveform} and Histogram. For arrays, the last axis is assumed to correspond to the event number index.

    Example

    julia> lhf = lh5open("path/to/lh5/file", "cw")
    julia> lhf["raw"]
    [...]
    julia> using Unitful
    julia> x = rand(100) * u"ns"
    julia> lhf["new"] = x
    [...]
    source
    LegendHDF5IO.compress — Constant
    LegendHDF5IO.compress[]

    Default of the compress argument of lh5open, :none: the compression of newly written array datasets, :zstd or :deflate. A setting, see the Settings section of the guide, where "true" and "false" stand for :zstd and :none.

    source
    LegendHDF5IO.max_chunk_bytes — Constant
    LegendHDF5IO.max_chunk_bytes[]

    Size in bytes up to which the first array written to a chunked, uncompressed dataset sets the chunk size, 16 MiB by default. A larger first write is split evenly into n chunks of at most this size, which leave fewer than n rows unused, and an empty one gets chunks of this size. An append then wastes at most one partially filled chunk. The default suits the block sizes of cluster file systems and does not depend on the file system, so that files written on one system read well on others. A setting, see the Settings section of the guide.

    source
    LegendHDF5IO.max_compressed_chunk_bytes — Constant
    LegendHDF5IO.max_compressed_chunk_bytes[]

    Size in bytes up to which compressed datasets are chunked, 1 MiB by default. A compressed chunk is decompressed as a whole, so compressed chunks stay small enough that a partial read decompresses little more than it needs. A setting, see the Settings section of the guide.

    source
    LegendHDF5IO.max_open — Constant
    LegendHDF5IO.max_open[]

    Default of the max_open argument of lh5open for file chains, 64: the number of files a chain keeps open at a time. A setting, see the Settings section of the guide.

    source
    LegendHDF5IO.rows_per_block — Constant
    LegendHDF5IO.rows_per_block[]

    Number of rows that iteration over a lazy table reads at a time, 4096 by default. A setting, see the Settings section of the guide.

    source
    LegendHDF5IO.scatter_bulk_max_bytes — Constant
    LegendHDF5IO.scatter_bulk_max_bytes[]

    Size up to which a scattered read reads the whole index span its events cover, instead of reading the events themselves.

    Reading the span costs one read operation. Reading the events costs one scattered selection, or one read per run of adjacent events, depending on element type, rank and layout. With the default nothing the limit depends on the file system the file lives on: 1 MiB on local storage and 8 MiB on network file systems such as GPFS, NFS or Lustre, where fewer and larger reads pay off. Variable-length elements like strings are always read selectively. A setting, see the Settings section of the guide, where "auto" stands for nothing.

    See also LegendHDF5IO.scatter_bulk_max_waste.

    source
    LegendHDF5IO.scatter_bulk_max_waste — Constant
    LegendHDF5IO.scatter_bulk_max_waste[]

    Factor by which a scattered read may exceed the requested amount of data when it reads the whole index span its events cover.

    Densely covered spans are read as a whole however large they are, so raising this trades reading more data for fewer read operations. With the default nothing the factor is 4 on local storage and 16 on network file systems. Spans up to LegendHDF5IO.scatter_bulk_max_bytes are read as a whole regardless. A setting, see the Settings section of the guide, where "auto" stands for nothing.

    source
    LegendHDF5IO.usechunks — Constant
    LegendHDF5IO.usechunks[]

    Default of the usechunks argument of lh5open, false: whether newly written array datasets are chunked, so that append! can extend them. A setting, see the Settings section of the guide.

    source
    LegendDataTypes.readdata — Method
    readdata(input::Union{HDF5.Dataset, HDF5.H5DataStore}, name::AbstractString)
    readdata(input, name, datatype::Type)

    Read the value stored under name from input via LH5Array and materialize it in memory.

    The "datatype" attribute determines the value type, unless datatype is given.

    source
    LegendDataTypes.writedata — Function
    writedata(output::Union{HDF5.H5DataStore, LHDataStore}, name::AbstractString, x,
        fulldatatype::DataType = typeof(x))

    Write the value x under name to output, via create_entry, with the usechunks and compress options of an LHDataStore.

    Passing fulldatatype overrides the "datatype" attribute written for x, and passing Nothing omits it.

    source
    LegendHDF5IO._chain_columns — Method
    _chain_columns(cols::AbstractVector)

    Concatenate the columns of the parts of a chained table, lazily and read-only for columns that read from disk.

    source
    LegendHDF5IO._typed_table — Method
    _typed_table(tbl::AbstractLH5Table)

    Open every column and return them as a StructArray, which indexing and appending work on.

    source
    LegendHDF5IO.add_entries! — Method
    add_entries!(lhd::LHDataStore, i::AbstractString, src::NamedTuple)

    Add the entries of src to the NamedTuple at lhd[i].

    source
    LegendHDF5IO.add_entries! — Method
    add_entries!(lhd::LHDataStore, i::AbstractString, src::StructArray)

    Add the columns of src to the table at lhd[i].

    source
    LegendHDF5IO.delete_entry! — Method
    delete_entry!(lhd::LHDataStore, i::AbstractString)

    Remove the dataset lhd[i] and adjust the datatype of the parent if necessary. Supported are entries of NamedTuples, tables and HDF5.Groups.

    source
    LegendHDF5IO.lh5open — Function
    lh5open(f, filename::AbstractString, access::AbstractString = "r"; kwargs...)

    Open the LEGEND HDF5 file filename, apply f to the resulting LHDataStore and close the file afterwards, returning the result of f.

    source
    LegendHDF5IO.lh5open — Function
    lh5open(filename::AbstractString, access::AbstractString = "r";
        usechunks::Bool = LegendHDF5IO.usechunks[], compress = LegendHDF5IO.compress[])

    Open a LEGEND HDF5 file and return an LHDataStore object.

    With usechunks = true, newly written array datasets are chunked and can be extended with append!. compress selects their compression: true or :zstd for Zstandard, :deflate (alias :gzip) for shuffle plus deflate. Compressed datasets are always chunked. Both default to the settings LegendHDF5IO.usechunks and LegendHDF5IO.compress. LEGEND HDF5 files typically use the file extension ".lh5".

    source
    LegendHDF5IO.lh5open — Method
    lh5open(filenames::AbstractVector{<:AbstractString}; max_open::Integer = LegendHDF5IO.max_open[])

    Open the LEGEND HDF5 files filenames, which must have the same structure, as one LH5FileChain, for reading. At most max_open of them stay open at a time, by default the setting LegendHDF5IO.max_open.

    source
    LegendHDF5IO.lh5open — Method
    lh5open(f, filenames::AbstractVector{<:AbstractString}; kwargs...)

    Open the files filenames as one LH5FileChain, apply f to it and close it afterwards, returning the result of f.

    source
    LegendHDF5IO.register_datatype! — Method
    LegendHDF5IO.register_datatype!(name::AbstractString, ::Type{T})
    LegendHDF5IO.register_datatype!(::Type{T})

    Register the LH5 datatype string name for type T, for reading and writing. Without name, the canonical datatype string of T is used (e.g. for Enum types).

    Each name can only be registered for one type, and each type only under one name. Names consist of letters and underscores. Built-in datatype names like "real" or "array<1>{real}" are reserved. Conflicting registrations throw an ArgumentError, while repeating a registration has no effect.

    The registry is not part of the precompilation cache, so register from the __init__ function of your module or package extension.

    For a type T without built-in support, also define

    LegendHDF5IO.LH5Array(ds::HDF5.Dataset, ::Type{<:T})
    LegendHDF5IO.create_entry(parent::LHDataStore, name::AbstractString, x::T; kwargs...)
    source