abstract type AbstractSetupData

Subtypes wrap SetupConfig for specific experiments.

struct ChannelId <: DataSelector

Represents a LEGEND data channel.

Example:

ch = ChannelId(1083204)
# ch = ChannelId(98) # with old channel numbering
ch.no == 1083204
string(ch) == "ch1083204"
ChannelId("ch1083204") == ch

struct DataCategory <: DataSelector

Represents a LEGEND data category (related to a DAQ/measuring mode) like "cal" or "phy".

Example:

category = DataCategory(:cal)
category.label == :cal
string(category) == "cal"
DataCategory("cal") == category

struct DataPartition <: DataSelector

Represents a LEGEND data-taking partition. If only a number is given, the struct uses category :cal and set :a as default.

The struct can also be constructed from strings in various formats, such as "calgroup001a", "calpartition001a", "calpart001a", or the short form "part001".

Example:

partition = DataPartition(1)
partition.cat == :cal
partition.no == 1
partition.set == :a
string(partition) == "calpartition001a"
DataPartition("calgroup001a") == partition
DataPartition("calpart001a") == partition
DataPartition("part001") == partition

struct DataPeriod <: DataSelector

Represents a LEGEND data-taking period.

Example:

period = DataPeriod(2)
period.no == 2
string(period) == "p02"
DataPeriod("p02") == period

struct DataRun <: DataSelector

Represents a LEGEND data-taking run.

Example:

r = DataRun(6)
r.no == 6
string(r) == "r006"
DataRun("r006") == r

abstract type DataSelector

Abstract type for data selectors like ExpSetup, DataTier, DataPeriod, DataRun, DataCategory, Timestamp and FileKey.

struct DataTier <: DataSelector

Represents a LEGEND data tier like "raw, "dsp", etc.

Example:

tier = DataTier(:raw)
tier.label == :raw
string(tier) == "raw"
DataTier("raw") == tier

struct DetectorId <: DataSelector

Represents a LEGEND detector id id.

Example:

detector = DetectorId(:V99000A)
detector.label == :V99000A
string(detector) == "V99000A"
DetectorId("V99000A") == detector

struct ExpSetup <: DataSelector

Represents a LEGEND experimental setup like "l200".

Example:

setup = ExpSetup(:l200)
setup.label == :l200
string(setup) == "l200"
ExpSetup("l200") == setup

struct FileKey <: DataSelector

Represents a LEGEND file key.

Example:

filekey = FileKey("l200-p02-r006-cal-20221226T200846Z")

See also read_filekeys and write_filekeys.

struct LegendData <: AbstractSetupData

Provides access to LEGEND data and metadata.

Constructors:

• LegendData(setup_config::SetupConfig).

• LegendData(setup::Symbol) - requires the $LEGEND_DATA_CONFIG environment variable to be set.

Examples:

config_filename = "/path/to/config.json"
config = LegendDataConfig(config_filename)
l200 = LegendData(config.setups.l200)

filekey = FileKey("l200-p02-r006-cal-20221226T200846Z")

or simply (if $LEGEND_DATA_CONFIG is set):

l200 = LegendData(:l200)

LegendData has the (virtual) properties metadata and tier.

The full path to "tier" data files can be retrieved using

(data::LegendData)[tier::Symbol, filekey::FileKey]
(data::LegendData).tier[tier::Symbol, filekey::AbstractString]

Example:

l200.tier[:raw]
l200.tier[:raw, FileKey("l200-p02-r006-cal-20221226T200846Z")]

LegendData comes with an extension for SolidStateDetectors:

l200 = LegendData(:l200)
SolidStateDetector(l200, :V99000A)

struct LegendDataConfig

Data configuration multiple experimental setups.

Contains a single field setups::PropertyDict{Symbol,SetupConfig}.

Can be read from a config file via LegendDataConfig(config_filename[s]), or simply LegendDataConfig() if the environment variable $LEGEND_DATA_CONFIG is set. $LEGEND_DATA_CONFIG may be a list of colon-separated config filenames, which are applied/merged in reverse order (analog to the order of prioritiy in $PATH and similar).

Example:

config = LegendDataConfig("/path/to/config.json")
setup = config.setups.l200
data_path(setup, "tier", "raw", "cal", "p02", "r006", "l200-p02-r006-cal-20221226T200846Z-tier_raw.lh5")

See also SetupConfig.

struct LegendReport

Represents a LEGEND report.

Don't instantiate directly, use lreport()

struct LegendDataManagement.LegendTierData

Constructors:

(data::LegendData).tier

LegendDataManagement.LegendTierData(data::LegendData)

The path to data directories and files can be accessed via getindex on tier_data::LegendTierData:

tier_data[]
tier_data[tier::DataTierLike]
tier_data[tier::DataTierLike, category::DataCategoryLike]
tier_data[tier::DataTierLike, category::DataCategoryLike, period::DataPeriodLike]
tier_data[tier::DataTierLike, category::DataCategoryLike, period::DataPeriodLike, run::DataRunLike]
tier_data[tier::DataTierLike, category::DataCategoryLike, period::DataPeriodLike, run::DataRunLike, ch::ChannelIdLike]

tier_data[tier::DataTierLike, filekey::FileKeyLike]
tier_data[tier::DataTierLike, filekey::FileKeyLike, ch::ChannelIdLike]

Examples:

```julia l200 = LegendData(:l200)

filekey = FileKey("l200-p02-r006-cal-20221226T200846Z") isfile(l200.tier[:raw, filekey])

isdir(l200.tier[:raw, :cal]) isdir(l200.tier[:raw, :cal, "p02"]) isdir(l200.tier[:raw, :cal, "p02", "r006"]) isdir(l200.tier[DataTier(:raw), DataCategory(:cal), DataPeriod(2), DataRun(6)])

struct LegendDataManagement.NoSuchPropsDBEntry

Indicates that a given property (path) of a LegendDataManagementPropsDB[@ref] does not exist.

Supports PropDicts.writeprops(missing_props::NoSuchPropsDBEntry, props::PropDicts.PropDict) to create the missing directories and file for the property path.

struct PeriodSelLike = Tuple{<:DataPeriodLike, <:DataCategoryLike}

Represents a LEGEND period selection for a specific category.

@enum ProcessStatus

May be succeeded, pending, or failed.

struct LegendDataManagement.PropsDB

A PropsDB instance, e.g. myprops, presents an on-disk directory containing JSON files or sub-directories (that contains JSON files in leaf directories) as a dictionary of properties.

PropsDB supports Base.keys and Base.getindex as well as Base.propertynames and Base.getproperty to access it's contents. getindex and getproperty will return either another PropsDB or a PropDicts.PropDict, depending on whether the accessed property is stored as a sub-directory or a JSON file. We recommend to use getproperty where the properties/keys of the PropDict are more or less standardized and where they may be arbitrary (see examples below).

The contents of PropsDB may be time- and category-dependent, determined by the presence of a "validity.json" file. In this case, use myprops(sel::LegendDataManagement.ValiditySelection) or myprops(filekey::FileKey) to select the desired time and category. The selection can be made at some point during traversal of properties or at the leaf PropsDB (see the examples below).

Examples:

l200 = LegendData(:l200)

propertynames(l200.metadata.hardware)
l200.metadata.hardware.detectors.germanium

keys(l200.metadata.hardware.detectors.germanium.diodes)
l200.metadata.hardware.detectors.germanium.diodes[:V99000A]

diodes = l200.metadata.hardware.detectors.germanium.diodes
diodes[keys(diodes)]

sel = ValiditySelection("20221226T194007Z", :cal)
filekey = FileKey("l200-p02-r006-cal-20221226T194007Z")
data.metadata.hardware(sel).configuration.channelmaps
data.metadata.hardware.configuration.channelmaps(filekey)

Use code should not instantiate PropsDB directly, use LegendDataManagement.AnyProps(path::AbstractString) instead, which may return a PropsDB or a PropDicts.PropDict depending on what on-disk content path points to.

struct RunCategorySelLike = Tuple{<:DataPeriodLike, <:DataRunLike}

Represents a LEGEND run selection for a specific category.

struct RunSelLike = Tuple{<:DataPeriodLike, <:DataRunLike}

Represents a LEGEND run selection.

struct SetupConfig

Data configuration for an experimental setup.

Supports

data_path(setup, path_components)

Examples:

data_path(setup, "tier", "raw", "cal", "p02", "r006", "l200-p02-r006-cal-20221226T200846Z-tier_raw.lh5")

struct Timestamp <: DataSelector

Represents a LEGEND timestamp.

Example:

julia timestamp = Timestamp("20221226T200846Z") timestamp.unixtime == 1672085326 string(timestamp) == "20221226T200846Z"`

struct LegendDataManagement.ValiditySelection

Representy validiy selection for a LegendDataManagement.PropsDB[@ref].

LegendDataManagement.AnyProps = Union{LegendDataManagement.PropsDB,PropDicts.PropDict}

Properties stored either in a directory managed via [LegendDataManagement.PropsDB][@ref] or loaded from one or several files into a PropDicts.PropDict.

Constructors:

LegendDataManagement.AnyProps(base_path::AbstractString; override_base::AbstractString = "")

const AnyValiditySelection = Union{ValiditySelection,FileKey}

Anything that can be used in time/category-based data selection.

ChannelIdLike = Union{ChannelId, Integer, AbstractString}

Anything that can represent a data channel, like ChannelId(1083204) or "ch1083204".

DataCategoryLike = Union{DataCategory, Symbol, AbstractString}

Anything that can represent a data category, like DataCategory(:cal), :cal or "cal".

DataPartitionLike = Union{DataPartition, Symbol, AbstractString}

Anything that can represent a data partition, like DataPartition(:calgroup001a) or "part02".

DataPeriodLike = Union{DataPeriod, Symbol, AbstractString}

Anything that can represent a data period, like DataPeriod(2) or "p02".

DataRunLike = Union{DataRun, Symbol, AbstractString}

Anything that can represent a data run, like DataRun(6) or "r006".

DataTierLike = Union{DataTier, Symbol, AbstractString}

Anything that can represent a data tier, like DataTier(:raw), :raw or "raw".

DetectorIdLike = Union{DetectorId, Symbol, AbstractString}

Anything that can represent a detector id.

ExpSetupLike = Union{ExpSetup, Symbol, AbstractString}

Anything that can represent a setup label, like ExpSetup(:l200), :l200 or "l200".

FileKeyLike = Union{FileKey, AbstractString}

Anything that can represent a file key, like FileKey("l200-p02-r006-cal-20221226T200846Z") or "l200-p02-r006-cal-20221226T200846Z".

const LJlExprLike = Union{Expr, Symbol, Integer, AbstractFloat}

Anything that can represent a parsed LEGEND Julia expression.

TimestampLike = Union{Timestamp, AbstractString, Integer}

Anything that can represent a timestamp, like Timestamp("20221226T200846Z") or "20221226T200846Z".

analysis_runs(data::LegendData)

Return cross-period analysis runs. Picks the dataset specified in data.dataset.

function atomic_fcreate(
    body, filenames::AbstractString...;
    create_dirs::Bool = true, overwrite::Bool = true, delete_on_error::Bool=true
)

Creates filenames in an atomic fashion.

Creates temporary files in the same directories as filenames, then calls body(temporary_filenames...). If body returns successfully, the files temporary_filenames are renamed to filenames. If body throws an exception, the temporary files are either deleted (if delete_on_error is true) or left in place (e.g. for debugging purposes).

If create_dirs is true, directories are created if necessary.

If all of files already exist and overwrite is false, takes no action (or, if the file is created by other code running in parallel, while body is running, does not overwrite it).

Throws an error if only some of the files exist and overwrite is false.

Returns nothing.

Example:

atomic_fcreate("foo.txt", "bar.txt") do foo, bar
    write(foo, "Hello")
    write(bar, "World")
end

bad_filekeys(data::LegendData, load_key::Symbol=:all)

Get the list of filekeys to ignore for data.

cal_groupings_default(data::LegendData)

Returns default cal_groupings runs.

channelinfo(data::LegendData, sel::AnyValiditySelection; system::Symbol = :all, only_processable::Bool = false, only_usability::Symbol = :all, extended::Bool = false)
channelinfo(data::LegendData, sel::RunCategorySelLike; system::Symbol = :all, only_processable::Bool = false, only_usability::Symbol = :all, extended::Bool = false)

Get all channel information for the given LegendData and ValiditySelection.

data_path(setup::SetupConfig, path_components::AbstractString...)

Get the full absolute path for the given path_components as configured for setup.

Converts between "/" and "\" inside of path_components if necessary (for Windows compatibility).

data_path(setup::AbstractSetupData, path_components::AbstractString...)

Get the full absolute path for the given path_components as configured for setup.

data_path(tier_data::LegendTierData, path_components::AbstractString...)

Get the full absolute path for the given path_components relative to tier_data.

data_path(pd::LegendDataManagement.PropsDB)

Return the path to the data directory that contains pd.

dataprod_config(data::LegendData)

Get the Julia data production configuration for data.

Use dataprod_config(data)(valsel::AnyValiditySelection) to also set the time/category validity selection for the configuration.

Examples:

l200 = LegendData(:l200)
filekey = FileKey("l200-p02-r006-cal-20221226T200846Z")
dataprod_config(l200)(filekey)

or

l200 = LegendData(:l200)
vsel = ValiditySelection("20221226T200846Z", :cal)
dataprod_config(l200)(vsel)

dataprod_parameters(data::LegendData)

Get the Julia data production parameters data.

Examples:

l200 = LegendData(:l200)
dataprod_config(l200)

LegendDataManagement.dataprod_pars_aoe_window(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the A/E cut window for the given data, validity selection and detector.

LegendDataManagement.dataprod_pars_lq_window(data::LegendData, sel::AnyValiditySelection, detector::DetectorId, lq_classifier::Symbol; pars_type::Symbol=:ppars, pars_cat::Symbol=:lq)

Get the LQ cut window for the given data, validity selection and detector.

find_filekey(ds::DataSet, ts::TimestampLike)
find_filekey(data::LegendData, ts::TimestampLike; kwargs...)

Find the filekey in a dataset that is closest to a given timestamp. The kwargs are passed to search_disk to generate the DataSet.

get_aux_cal_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the aux calibration function for the given data, validity selection and the aux channel referred to by detector.

get_aux_evt_chdata_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the aux channel data output PropertyFunction.

get_aux_evt_chsel_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the aux channel selection PropertyFunction.

get_aux_evt_levelname_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the aux event level name.

get_exposure(data::LegendData, det::DetectorIdLike, period::DataPeriodLike, run::DataRunLike; kwargs...)
get_exposure(data::LegendData, det::DetectorIdLike, period::DataPeriodLike; kwargs...)
get_exposure(data::LegendData, det::DetectorIdLike, partition::DataPartitionLike; kwargs...)

Calculates the exposure of a detector in a given run/period/partition.

Arguments

• data: LegendData object with information on detector geometries and runinfo / partitioninfo
• det Detector for which the exposure is calculated
• period: DataPeriod for which the exposure is calculated
• run: DataRun for which the exposure is calculated
• partition: DataPartition for which the exposure is calculated

Keyword Arguments

• is_analysis_run: If set to true, only the runs flagged as is_analysis_phy_tun == true are considered. Default is true.
• cat DataCategory for which the exposure is calculated. Default is :phy.`

Returns

• exposure: the exposure of the detector det for the time given.

Example

julia l200 = LegendData(:l200) get_exposure(l200, :V00050A, DataPeriod(3), DataRun(0)) get_exposure(l200, :V00050A, DataPeriod(3)) get_exposure(l200, :V00050A, DataPartition(:calgroup001a))`

LegendDataManagement.get_ged_aoe_cut_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId; pars_type::Symbol=:ppars, pars_cat::Symbol=:aoe)

Get the A/E cut propfuncs for the given data, validity selection and detector.

get_ged_cal_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the HPGe calibration function for the given data, validity selection and detector.

Note: Caches configuration/calibration data internally, use a fresh data object if on-disk configuration/calibration data may have changed.

get_ged_evt_chdata_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the Ge-detector channel data output PropertyFunction.

get_ged_evt_chsel_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the Ge-detector channel selection PropertyFunction.

get_ged_evt_hitchsel_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the hit Ge-detector channel selection PropertyFunction.

get_ged_evt_is_valid_hit_properties(data::LegendData, sel::AnyValiditySelection)

Get the hit Ge-detector is_valid_hit selection properties.

get_ged_evt_kwargs(data::LegendData, sel::AnyValiditySelection)

Get the Ge-detector evt kwargs.

LegendDataManagement.get_ged_lq_cut_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId; pars_type::Symbol=:ppars, pars_cat::Symbol=:lq)

Get the LQ cut propfuncs for the given data, validity selection and detector.

get_ged_psd_classifier_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the PSD cut propfuncs for the given data and validity selection.

get_ged_psd_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId; pars_type::Symbol=:ppars, pars_cat::Symbol=:aoe)

Get the HPGe psd calibration function for the given data, validity selection and detector.

Note: Caches configuration/calibration data internally, use a fresh data object if on-disk configuration/calibration data may have changed.

get_ged_qc_cuts_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the Ge-detector QC cut definitions for the given data and validity selection.

get_ged_qc_is_baseline_propfunc(data::LegendData, sel::AnyValiditySelection)

Get a PropertyFunction that returns true for events that fullfill the is_baseline definition.

get_ged_qc_is_physical_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get a PropertyFunction that returns true for events that fullfill the is_physical definition.

get_ged_qc_istrig_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the Ge-detector trigger condition for the given data and validity selection.

get_partition_combined_periods(data::LegendData, period::DataPeriodLike; chs::Vector{<:ChannelIdLike}=ChannelIdLike[])
get_partition_combined_periods(data::LegendData, period::DataPeriodLike; dets::Vector{<:DetectorIdLike}=DetectorIdLike[])

Get a list periods which are combined in any partition for the given period and list of channels.

get_pmt_cal_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the PMT calibration function for the given data, validity selection and detector.

get_pmt_is_physical_trig_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the PMT physical trigger condition for the given data and validity selection.

get_pmts_evt_chdata_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the PMT-detector channel data output PropertyFunction.

get_pmts_evt_chsel_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the PMT channel selection PropertyFunction.

get_pmts_evt_evtdata_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the PMT-detector channel data output PropertyFunction.

get_pmts_evt_kwargs(data::LegendData, sel::AnyValiditySelection)

Get the PMT evt kwargs.

get_pmts_evt_muon_cut_props(data::LegendData, sel::AnyValiditySelection)

Get the PMT muon cut properties.

LegendDataManagement.get_setup_config(data::AbstractSetupData)::SetupConfig

Must be specialized for each subtype of AbstractSetupData.

get_spm_cal_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the LAr/SPMS calibration function for the given data, validity selection and detector.

get_spm_dc_cal_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the LAr/SPMS DC calibration function for the given data, validity selection and detector.

get_spm_dc_sel_propfunc(data::LegendData, sel::AnyValiditySelection, detector::DetectorId)

Get the LAr/SPMS DC calibration selector function for the given data, validity selection and detector.

get_spms_evt_chdata_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the SiPM-detector channel data output PropertyFunction.

get_spms_evt_chsel_propfunc(data::LegendData, sel::AnyValiditySelection)

Get the SiPM channel selection PropertyFunction.

get_spms_evt_kwargs(data::LegendData, sel::AnyValiditySelection)

Get the SiPM-detector evt kwargs.

get_spms_evt_lar_cut_props(data::LegendData, sel::AnyValiditySelection)

Get the SiPM LAr cut properties.

get_uncertainties(x::Unitful.Quantity{<:Measurements.Measurement{<:Real}})
get_uncertainties(x::Unitful.Quantity{<:Real})
get_uncertainties(pd::PropDict)
get_uncertainties(A::AbstractArray)

Get the uncertainty of a Unitful.Quantity or Measurements.Measurement object or a PropDict or an array of Unitful.Quantity or Measurements.Measurement objects.

get_values(x::Unitful.Quantity{<:Measurements.Measurement{<:Real}})
get_values(x::Unitful.Quantity{<:Real})
get_values(pd::PropDict)
get_values(A::AbstractArray)

Get the value of a Unitful.Quantity or Measurements.Measurement object or a PropDict or an array of Unitful.Quantity or Measurements.Measurement objects.

is_analysis_cal_run(data::LegendData, (period::DataPeriodLike, run::DataRunLike))

Return true if run is an analysis run for data in period.

ATTENTION: This is only valid for cal runs.

is_analysis_phy_run(data::LegendData, (period::DataPeriodLike, run::DataRunLike))

Return true if run is an analysis run for data in period.

ATTENTION: This is only valid for phy runs.

is_analysis_run(data::LegendData, (period::DataPeriodLike, run::DataRunLike, cat::DataCategoryLike))

Return true if run is an cat analysis run for data in period.

is_lrun(data::LegendData, runsel::RunSelLike)

Return true if runsel is a valid run for data and therefore appears in the metadata.

livetime(data::LegendData, runsel::RunSelLike)

Get the livetime for data in physics data taking of run in period.

ljl_propfunc(expr::LJlExprLike)
ljl_propfunc(expr_string::AbstractString)

Compiles a PropertyFunctions.PropertyFunction from a LEGEND Julia expression.

See also parse_lpyexpr.

ljl_propfunc(expr_map::AbstractDict{Symbol,<:LJlExprLike})
ljl_propfunc(expr_map::AbstractDict{Symbol,<:AbstractString})
ljl_propfunc(expr_map::PropDict)

Compiles a map between output field-names and LEGEND Julia expressions to a single PropertyFunctions.PropertyFunction.

The generated function will return NamedTuples with the same property names as the keys of expr_map.

lpy_propfunc(expr_string::AbstractString)::PropertyFunctions.PropertyFunction

Generate a PropertyFunctions.PropertyFunction from a LEGEND Python expression.

See also parse_lpyexpr and ljl_propfunc.

lreport()
lreport(contents...)

Generate a LEGEND report, e.g. a data processing report.

Use lreport!(rpt, contents...) to add more content to a report.

Example:

using LegendDataManagement, StructArrays, IntervalSets, Plots

tbl = StructArray(
    col1 = rand(5), col2 = ClosedInterval.(rand(5), rand(5).+1),
    col3 = [rand(3) for i in 1:5], col4 = ProcessStatus.(rand(-1:1, 5)),
    col5 = [:a, :b, :c, :d, :e], col6 = ["a", "b", "c", "d", "e"],
    col7 = [:(a[1]), :(a[2]), :(a[3]), :(a[4]), :(a[5])]
)

rpt = lreport(
    "# New report",
    "Table 1:", tbl
)
lreport!(rpt, "Figure 1:", stephist(randn(10^3)))
lreport!(rpt, "Figure 2:", histogram2d(randn(10^4), randn(10^4), format = :png))

show(stdout, MIME"text/plain"(), rpt)
show(stdout, MIME"text/html"(), rpt)
show(stdout, MIME"text/markdown"(), rpt)

writelreport("report.txt", rpt)
writelreport("report.html", rpt)
writelreport("report.md", rpt)

See LegendDataManagement.lreport_for_show! for how to specialize the behavior of show for specific report content types.

lreport!(rpt::LegendReport, contents...)

Add more content to report rpt. See lreport for an example.

LegendDataManagement.lreport_for_show!(rpt::LegendReport, mime::MIME, content)

Add the contents of content to rpt in a way that is optimized for being displayed (e.g. via show) with the given mime type.

show(output, mime, rpt) first transforms rpt by converting all contents of rpt using lreport_for_show!(rpt::LegendReport, mime, content).

Defaults to lreport!(rpt, content), except for tables (Tables.istable(content) == true), which are converted to Markdown tables by default for uniform appearance.

lreport_for_show! is not inteded to be called by users, but to be specialized for specific types of content content. Content types not already supported will primarily require specialization of

lreport_for_show!(rpt::LegendReport, ::MIME"text/markdown", content::SomeType)

In some cases it may be desireable to specialize lreport_for_show! for MIME types like MIME"text/html" and MIME"text/plain" as well.

map_datafiles(
    f_process, f_open, data::LegendData,
    category::DataCategoryLike, filekeys::AbstractVector{<:FileKey}
)

Processes all filekeys in data for category.

Opens the files using f_open and processes them using f_process.

Returns a @NamedTuple{result::Dict{Filekey}, failed::Dict{Filekey}, success::Bool}:

• values(result) contains the results f_process(f_open(filename)) for all filenames referred to by category and filekeys.
• values(result) contains the error where processing failed.
• success equals isempty(failed)

parse_ljlexpr(expr_string::AbstractString)::LJlExprLike

Parse an LEGEND Julia expression and return a Julia syntax tree.

parse_lpyexpr(expr_string::AbstractString)::LJlExprLike

Parse an expression compatible with the LEGEND Python software and return a Julia syntax tree.

parse_runs(::AbstractVector{<:AbstractString})::Vector{DataRun}
parse_runs(::AbstractString})::Vector{DataRun}

Parse a String or a Vector{String} of runs in the format "rXXX" (single run) or "rXXX..rYYY" (range of runs).

partitioninfo(data::LegendData, det::DetectorIdLike, cat::DataCategoryLike)
partitioninfo(data, det, part::DataPartition)
partitioninfo(data, det, cat, period::DataPeriod)
partitioninfo(data, det, cat, period, run)

partitioninfo(data::LegendData, ch::ChannelId, cat::DataCategoryLike = :cal)

Return cross-period data partitions.

Arguments

• data::LegendData: The LegendData object containing the data.
• det::DetectorIdLike: The ID of the detector.
• category::DataCategoryLike: Analysis category to select groupings, e.g. :cal, :phy.

Returns

• IdDict{DataPartition, Table}: A dictionary mapping data partitions to tables.

phy_groupings_default(data::LegendData)

Returns default phy_groupings runs.

process_ljlexpr(expr::LJlExprLike, f_varsubst = identity)::LJlExprLike

Verify that expr is a valid LEGEND Julia expression and return it, with modifications if necessary.

Optionally substitute variables in expr using f_varsubst.

pydataprod_config(data::LegendData)

Get the Python data production configuration for data.

Use pydataprod_config(data)(valsel::AnyValiditySelection) to also set the time/category validity selection for the configuration.

Examples:

l200 = LegendData(:l200)
filekey = FileKey("l200-p02-r006-cal-20221226T200846Z")
pydataprod_config(l200)(filekey)

or

l200 = LegendData(:l200)
vsel = ValiditySelection("20221226T200846Z", :cal)
pydataprod_config(l200)(vsel)

pydataprod_parameters(data::LegendData)

Get the Julia data production parameters data.

Examples:

l200 = LegendData(:l200)
dataprod_config(l200)

read_filekeys(filename::AbstractString)::AbstractVector{FileKey}

Reads a list of FileKey from a text file, one file key per line.

Ignores empty lines. # may be used to start a comment in the file.

read_ldata(data::LegendData, selectors...; kwargs...)
read_ldata(f, data::LegendData, selectors...; kwargs...)
read_ldata(columns::NTuple{<:Any, Symbol}, data::LegendData, selectors::Tuple; kwargs...)
read_ldata(column::Symbol, data::LegendData, selectors::Tuple; kwargs...)

Read lh5 data from disk for a given set of selectors. After reading in, a PropertyFunction f can be applied to the data. If a tuple of Symbols is given, the properties from the tuple are selected. If the n_evts kwarg is provided, a random selection with n_evts number of events per file is performed. ch can be either a ChannelId or a DetectorId.

Examples

dsp = read_ldata(l200, :jldsp, filekey, ch)
dsp = read_ldata((:e_cusp, :e_trap, :blmean), l200, :jldsp, filekey, ch)
dsp = read_ldata(:e_cusp, l200, :jldsp, filekey, ch)
dsp = read_ldata(l200, :jldsp, :cal, :p03, :r000, ch)

dsp = read_ldata(l200, :jldsp, :cal, DataPartition(:calgroup001a), ch)
dsp = read_ldata(l200, :jldsp, :cal, DataPeriod(3), ch)
dsp = read_ldata(l200, :jldsp, :cal, runinfo(l200)[1:3], ch)

dsp = read_ldata(l200, :jldsp, filekey, ch; n_evts=1000)

readlprops(filename::AbstractString)
readlprops(filenames::Vector{<:AbstractString})

Read a PropDict from a file and parse it to Unitful.Quantity and Measurements.Measurement objects.

Returns

• pd::PropDict with all :val fields converted to Unitful.Quantity objects and all :val fields converted to Measurements.Measurement objects.

runinfo(data::LegendData)::Table
runinfo(data::LegendData, runsel::RunSelLike)::NamedTuple
runinfo(data::LegendData, filekey::FileKey)::NamedTuple

Get the run information for data based on various selection criteria.

Arguments

• data::LegendData: The dataset to query run information from.

Returns

A table of run information with one named tuple per category (e.g. :cal, :phy), each containing startkey, livetime, and is_analysis_run

Example

runinfo(data) # full table of valid runs runinfo(data, :p03) # all runs in period p03 runinfo(data, (:p03, :r005)) # single-row Table for that run runinfo(data, (:p03, :r005, :phy)) # only the :phy entry runinfo(data, fk::FileKey) # same as above via FileKey

search_disk(::Type{<:DataSelector}, path::AbstractString)
search_disk(::Type{DataSet}, data::LegendData; search_categories::Vector{<:DataCategoryLike} = DataCategory.([:cal, :phy]), search_tier::DataTierLike = DataTier(:raw), only_analysis_runs::Bool=true, save_filekeys::Bool=true, ignore_save_tier::Bool=false, save_tier::DataTierLike=DataTier(:jlfks))

Search on-disk data for data categories, periods, runs, and filekeys or whole datasets If you want to search for a whole DataSet, you have the following keyword options: - search_categories (default: [:cal, :phy]): The categories to search on disk. - search_tier (default: DataTier(:raw)): The tier to search on disk. - only_analysis_runs (default: true): Only include for analysis runs as defined in the metadata - save_filekeys (default: true): Save the filekeys to a file in the save_tier directory. - ignore_save_tier (default: false): Ignore the save_tier and do not save the filekeys. - save_tier (default: DataTier(:jlfks)): The tier to save the filekeys to.

Examples:

l200 = LegendData(:l200)

search_disk(DataCategory, l200.tier[:raw])
search_disk(DataPeriod, l200.tier[:raw, :cal])
search_disk(DataRun, l200.tier[:raw, :cal, "p02"])
search_disk(FileKey, l200.tier[DataTier(:raw), :cal, DataPeriod(2), "r006"])
search_disk(DataSet, l200)

start_filekey(data::LegendData, runsel::RunCategorySelLike)

Get the starting filekey for data in period, run, category.

LegendDataManagement.tmp_filename(fname::AbstractString)

Returns a temporary filename, based on fname, in the same directory.

Does not create the temporary file.

write_filekeys(filename::AbstractString, filekeys::AbstractVector{<:FileKey})

Writes a list of FileKey to a text file, one file key per line.

writelprops(f::IO, p::PropDict; write_units::Bool=true, write_errors::Bool=true, mutliline::Bool=true, indent::Int=4)
writelprops(filename::AbstractString, p::PropDict; multiline::Bool=true, indent::Int=4)
writelprops(db::PropsDB, key::Union{Symbol, DataSelector}, p::PropDict; kwargs...)

Write a PropDict to a file and strip it to :val and :unit fields and :val and :err fields.

writelreport(filename::AbstractString, rpt::LegendReport)
writelreport(filename::AbstractString, mime::MIME, rpt::LegendReport)

Write lreport rpt to file filename.

channel2detector(data::LegendData, sel::Union{AnyValiditySelection, RunCategorySelLike}, channel::ChannelIdLike)

Get the DetectorId for a given ChannelId input:

• data, e.g. LegendData(:l200)`
• runsel: runselection, e.g. (DataPeriod(3), DataRun(0), :cal)
• channel: ChannelId e.g. ChannelId(1080005)`

output:

• DetectorId of corresponding channel

create_logtbl(result)

Create a log table for a given result which can be added in a report.

create_metadatatbl(filekey::FileKey)
create_metadatatbl(filekey::FileKey, part::DataPartitionLike)

Create a metadata table for a given filekey which can be added in a report.

create_pars(pd::PropDict, result::Dict{ChannelInfo, ChannelResult}) -> PropDict

Create a PropDict from a result of the parallel processing

create_validity(result) -> StructArray

Create a StructArray from a result of the parallel processing

data_starttime(data::LegendData, runsel::Union{AnyValiditySelection, RunCategorySelLike})

Extract startime as DateTime from file for a given run selection Input: * data: LegendData, e.g. LegendData(:l200) * runsel: runselection, e.g. (DataPeriod(3), DataRun(0), :cal)

detector2channel(data::LegendData, sel::Union{AnyValiditySelection, RunCategorySelLike}, detector::DetectorIdLike)

Get the ChannelId for a given detectorId input:

• data, e.g. LegendData(:l200)`
• runsel: runselection, e.g. (DataPeriod(3), DataRun(0), :cal)
• detector: DetectorID e.g. DetectorId(:P00573A)OR ChannelID e.g.ChannelId(1080005)``

output:

• ChannelId of corresponding detector

get_det_type(data::LegendData, det::DetectorIdLike)

Looks up the detector type for a given DetectorID.

get_hitchfilename(data::LegendData, setup::ExpSetupLike, period::DataPeriodLike, run::DataRunLike, category::DataCategoryLike, ch::ChannelIdLike)
get_hitchfilename(data::LegendData, filekey::FileKey, ch::ChannelIdLike)

Get the filename for the hitch data for a given channel.

get_mltrainfilename(data::LegendData, period::DataPeriodLike, category::DataCategoryLike)
get_mltrainfilename(data::LegendData, filekey::FileKey)

Get the filename for the machine learning training data.

get_partitionfilekeys(data::LegendData, part::DataPartitionLike, tier::DataTierLike, category::DataCategoryLike; only_good::Bool=true)

Get filekeys for a given partition.

Arguments

• data::LegendData: data object
• part::DataPartitionLike: partition to be searched in
• tier::DataTierLike: tier
• category::DataCategoryLike: category
• only_good::Bool=true: only get good filekeys

Return

• Vector{FileKey}: filekeys

get_partitionvalidity(data::LegendData, part::DataPartitionLike) -> Vector{@NamedTuple{period::DataPeriod, run::DataRun, validity::String}}

Get partition validity for a given channel and partition.

get_peaksfilename(data::LegendData, setup::ExpSetupLike, period::DataPeriodLike, run::DataRunLike, category::DataCategoryLike, ch::ChannelIdLike)
get_peaksfilename(data::LegendData, filekey::FileKey, ch::ChannelIdLike)

Get the filename for the peaks data for a given channel.

get_plottitle(setup::ExpSetupLike, period::DataPeriodLike, run::DataRunLike, category::DataCategoryLike, det::DetectorIdLike, process::AbstractString; additional_type::AbstractString="")
get_plottitle(filekey::FileKey, det::DetectorIdLike, process::AbstractString; kwargs...)
get_plottitle(setup::ExpSetupLike, partition::DataPartitionLike, category::DataCategoryLike, det::DetectorIdLike, process::AbstractString; additional_type::AbstractString="")
get_plottitle(filekey::FileKey, partition::DataPartitionLike, det::DetectorIdLike, process::AbstractString; kwargs...)

Get the title for a plot.

get_pltfilename(data::LegendData, setup::ExpSetupLike, period::DataPeriodLike, run::DataRunLike, category::DataCategoryLike, ch::ChannelIdLike, process::Symbol)
get_pltfilename(data::LegendData, filekey::FileKey, ch::ChannelIdLike, process::Symbol)
get_pltfilename(data::LegendData, partition::DataPartitionLike, setup::ExpSetupLike, category::DataCategoryLike, ch::ChannelIdLike, process::Symbol)

Get the filename for the plot file for a given setup, period, run, category, channel and process.

get_pltfolder(data::LegendData, period::DataPeriodLike, run::DataRunLike, category::DataCategoryLike, process::Symbol)
get_pltfolder(data::LegendData, filekey::FileKey, process::Symbol)
get_pltfolder(data::LegendData, partition::DataPartitionLike, category::DataCategoryLike, process::Symbol, ch::ChannelIdOrDetectorIDLike)

Get the folder for the plot files for a given period, run, category and process.

get_preportfilename(data::LegendData, setup::ExpSetupLike, period::DataPeriodLike, category::DataCategoryLike, process::Symbol)

Get the filename for the log file for a given setup, period, category and process.

get_preportfolder(data::LegendData, period::DataPeriodLike, category::DataCategoryLike)

Get the preport folder for the log files for a given period and category.

get_rreportfilename(data::LegendData, setup::ExpSetupLike, period::DataPeriodLike, run::DataRunLike, category::DataCategoryLike, process::Symbol)
get_rreportfilename(data::LegendData, filekey::FileKey, process::Symbol)

Get the filename for the log file for a given setup, period, run, category and process.

get_rreportfolder(data::LegendData, period::DataPeriodLike, run::DataRunLike, category::DataCategoryLike)

Get the rreport folder for the log files for a given period, run and category.

get_totalTimer(result::Vector)

Get the total timer from a result vector.

load_partition_ch(open_func::Function, flatten_func::Function, data::LegendData, partinfo::StructVector, tier::DataTierLike, cat::DataCategoryLike, ch::ChannelIdLike; data_keys::Tuple=(), n_evts::Int=-1, select_random::Bool=false)
load_partition_ch(open_func::Function, flatten_func::Function, data::LegendData, part::DataPartition, tier::DataTierLike, cat::DataCategoryLike, ch::ChannelIdLike; kwargs...)

Load data for a channel from a partition.

Arguments

• open_func::Function: function to open a file
• flatten_func::Function: function to flatten data
• data::LegendData: data object
• partinfo::StructVector: partition info
• tier::DataTierLike: tier
• cat::DataCategoryLike: category
• ch::ChannelIdLike: channel
• data_keys::Tuple=(): data keys, empty tuple selects all keys
• n_evts::Int=-1: number of events, -1 selects all events
• select_random::Bool=false: select events randomly

Return

• Table: data table with flattened events

load_raw_evt(open_func::Function, data::LegendData, ch::ChannelIdLike, data_hit::Table, sel_evt::Int)
load_raw_evt(open_func::Function, data::LegendData, ch::ChannelIdLike, data_hit::Table, sel_evt::UnitRange{Int})

Load data for a channel from a hitch file for a given selected event index or index range.

Arguments

• open_func::Function: function to open a file
• data::LegendData: data object
• ch::ChannelIdLike: channel
• data_hit::Table: hitch data
• sel_evt::Int/UnitRange{Int}: selected event index

Return

• Table: data table of raw events

load_run_ch(open_func::Function, flatten_func::Function, data::LegendData, filekeys::Vector{FileKey}, tier::DataTierLike, ch::ChannelIdLike; check_filekeys::Bool=true)

Load data for a channel from a list of filekeys in a given tier.

Arguments

• open_func::Function: function to open a file
• flatten_func::Function: function to flatten data
• data::LegendData: data object
• filekeys::Vector{FileKey}: list of filekeys
• tier::DataTierLike: tier to load data from
• ch::ChannelIdLike: channel to load data for
• check_filekeys::Bool=true: check if filekeys are valid

savelfig(save_func::Function, p, data::LegendData, setup::ExpSetupLike, period::DataPeriodLike, run::DataRunLike, category::DataCategoryLike, ch::ChannelIdLike, process::Symbol; kwargs...)
savelfig(save_func::Function, p, data::LegendData, filekey::FileKey, ch::ChannelIdLike, process::Symbol; kwargs...)
savelfig(save_func::Function, p, data::LegendData, partition::DataPartitionLike, setup::ExpSetupLike, category::DataCategoryLike, ch::ChannelIdLike, process::Symbol; kwargs...)

Save a lplot.

writevalidity(props_db::LegendDataManagement.PropsDB, filekey::FileKey,
              apply::Union{String,Vector{String}}; category::Symbol = :all, mode::String = "reset")

writevalidity(props_db::LegendDataManagement.PropsDB, filekey::FileKey,
              rsel::Tuple{DataPeriod,DataRun}; category::Symbol = :all)

writevalidity(props_db::LegendDataManagement.PropsDB, filekey::FileKey,
              part::DataPartitionLike; category::Symbol = :all)

writevalidity(props_db::LegendDataManagement.PropsDB,
              validity_with_flag::NamedTuple{(:result, :skipped)}; category::Symbol = :all, impl::Symbol = :full)

Write validity information into validity.yaml.

• With apply (string or vector), one or more validity files are written using the given mode ("reset", "append", "remove", or "replace").
• A (DataPeriod, DataRun) or DataPartition is automatically converted into the corresponding validity string.
• With a NamedTuple(:result, :skipped), process a sequence of validity updates across detectors and partitions, using either the :full or :diff implementation.

All entries are merged chronologically into validity.yaml.