Configuration
The processing config file sets up the analysis chain. The config shipped with JuLeAna is config/processing_config.yaml. Without --config, juleana reads config/processing_config.yaml in the data production (the folder config next to the file named by LEGEND_DATA_CONFIG). If that file does not exist, JuLeAna first copies the config shipped with JuLeAna there; a processing_config.json in that folder is ignored with a warning. w on the Setup page of the interface writes the current settings to that file and keeps the previous file as processing_config.yaml.bak. See Running the dataflow. The file is loaded into a ProcessingConfig; all keys of the config and processing sections are required except idle_timeout_minutes, worker_restart_delay_seconds, worker_heartbeat_timeout_minutes, item_time_estimate, item_time_minimum_items, item_timeout_factor, item_timeout_minimum_minutes, processor_log_level, processor_plot_level, plot_formats and runmode_settings.slurm.worker_memory, and a missing or unknown key stops the dataflow with an error that names the key.
config
The run mode, the worker settings, environment variables and logging:
runmode:"local"or"slurm".submit_jobs: start the workers of the selected run mode automatically.runmode_settings:local.n(number of local workers) andlocal.julia_settings,slurm.flagsandslurm.julia_settings.slurm.flags: options passed to SLURM, see the SLURM documentation and the documentation of the cluster you use.slurm.worker_memory(optional):"shared"(the default) or"fixed". Each SLURM worker runs as its own job step. With"shared"the workers of a job share its memory (--mem): a worker can use more than its part while others use less, and when the job as a whole reaches its memory, the system ends the worker process that uses the most. With"fixed"each worker gets the memory of the job per node divided by the workers per node, and a worker that needs more is ended even if the others use less.julia_settings: options passed to thejuliaprocesses of the workers, such as--heap-size-hint; see thejuliacommand line documentation. Without--threadsinlocal.julia_settings, each local worker gets the number of CPU threads of the host divided bylocal.n(at least one), so the workers together use each CPU thread once.
env_variables: environment variables set for the dataflow and its workers, for example to select a data set. SetJULIA_WORKER_TIMEOUTfor a stable connection of the ParallelProcessingTools workers;JULIA_DEBUGenables debug messages of the listed packages.log_level:"debug","info","warn"or"error", the level of the records that every process writes to its log file in the log directory (head.logfor the main process,worker-<id>-<host>.logfor each worker)."debug"adds the debug records of the packages listed inJULIA_DEBUG. The terminal user interface also writes the error output of the main process tohead-stderr.logthere, each line after its UTC time; Distributed and the cluster manager that adds the workers print their errors there.processor_log_level(optional):"debug","info","warn"or"error". Without it, the processors' own records are written atlog_level. With it, it replaceslog_levelfor them; seelog_levelof the processor entries below. The command line option--processor-log-levelreplaces the value of the config, and the "Processor log level" row of the Setup page replaces both. The level in effect is the one of the Setup page, else the command line, elseprocessor_log_level, else thelog_levelof each processor entry.processor_plot_level(optional):"info"or"debug". Without it, each processor saves plots at theplot_levelof its entry. With it, it replacesplot_levelfor all processors; seeplot_levelof the processor entries below. The command line option--processor-plot-levelreplaces the value of the config, and the "Processor plot level" row of the Setup page replaces both.plot_formats(optional, default["png"]): a list of the formats in which plots are saved,"png"and"jls", at least one, each at most once. With"jls", the Makie figure is also saved withSerializationnext to the PNG, under the same name with the extension.jls. The file lets you open the figure again later, but only a Julia session with the same package versions can read it.idle_timeout_minutes(optional, default20): when no processing has run for this many minutes, the worker allocations started from the terminal user interface are cancelled, so that idle workers do not hold a machine or a SLURM allocation. The time counts from the end of the last processing run, the start of the newest allocation or the last postponement, whichever is latest.0never cancels them.worker_restart_delay_seconds(optional, default30): the worker start script starts a new worker whenever a worker ends, for example after a timeout or after running out of memory, and waits this many seconds before it does. The pause keeps a worker that cannot start from being started again and again in quick succession.worker_heartbeat_timeout_minutes(optional, default5, at least2): while it has worker allocations,juleanatouches a heartbeat file every minute. When a worker ends and the file has not been touched for this many minutes, for example becausejuleanacrashed, the allocation ends itself instead of starting a new worker.item_time_estimate(optional, default"median"):"median","p95"(the 95th percentile) or"max", the statistic of the item times of a processor by which the processing run estimates how long its steps take. When every selected step runs withreprocess = true, the steps whose chain of waiting steps is expected to take longest start first, and their items get free workers first. The item times of a processor come from its items of the current run onceitem_time_minimum_itemsof them have finished, before that from the latest run in the history of the production that recorded them.item_time_minimum_items(optional, default100, at least1): the number of finished items of a processor in the current run from which their times replace the recorded ones, for the estimates above and fortimeout: auto. Fewer items give an unreliable statistic, and the first items of a run include the compilation.item_timeout_factor(optional, default2): a processor withtimeout: autoin itskwargs(see below) gets this factor times the 99th percentile of its item times as timeout per item.item_timeout_minimum_minutes(optional, default5): such a timeout is at least this many minutes.
processing
The periods and runs to process, each either a list of numbers or "all":
processing:
periods: [3]
runs: all
...processes all available runs of period 3. "periods": "all" processes all periods of the dataset. The other keys are flags:
analysis_runs_only: process only runs marked as analysis runs inlegend-metadata, i.e. runs that passed the quality criteria for the final analysis.only_runs: run only the run-level processors.only_partitions: run only the partition-level processors.
Two keys are optional:
order:"dependencies"or"rank"(the default). With"dependencies"thedependenciesof the processors alone decide the order, and processors that do not depend on each other run at the same time. With"rank"the processors of each level run one after another in the order ofrank, then name: when the config is loaded, every processor gets a dependency on the processor before it, and a warning asks to switch to"dependencies".max_parallel_steps: the maximum number of processor steps (one processor for one run, or for one period at partition level) that run at the same time;0, the default, sets no limit. The steps share the workers either way: a free worker goes to the waiting item of the step with the longest chain of selected steps after it, then with the most selected steps after it, then by rank.
processors and p_processors
The run-level processors (processors) and partition-level processors (p_processors) of the chain, keyed by processor name. At least one processor must be enabled. An entry looks like this:
process_dsp_cal:
enabled: true
category: cal
rank: 5
log_level: info
plot_level: info
detectors: "processable"
kwargs:
reprocess: true
max_wvfs: 1000
timeout: auto
dependencies: [p_process_aoe_optimization_cal]All fields are required:
enabled: whether the processor runs.category:"cal"or"phy", the data category the processor works on.rank: position in listings, and the tie rule of the scheduling: among steps that are ready to start, lower ranks start first.kwargs: keyword arguments passed to the processor; the Processors page lists the keywords of each processor. Every enabled processor must accept all of its configured keywords, plusreprocessand, for partition processors,only_first_period, which the dataflow passes on its own. This is checked when the dataflow starts. The--reprocesscommand line flag setsreprocesstotruefor every processor. The keywordtimeoutof the processors that have one is the number of seconds an item may run before its worker is ended,0for no limit, orauto. Withauto, each step of the processor getsitem_timeout_factortimes the 99th percentile of the item times of the processor, at leastitem_timeout_minimum_minutes. The 99th percentile comes from the current run onceitem_time_minimum_itemsitems of the processor have finished, before that from the latest run in the history of the production that recorded it. While neither is known, the step runs without a timeout. The timeout of a step is set when it starts and is written to the log.dependencies: processors of either level that must finish first. A run processor waits for a run processor in the same run and for a partition processor in its period. A partition processor waits for a run processor in all runs of its period and of the other selected periods that share a default partition with it. A processor that is not selected passes its dependencies on: a selected processor waits for the selected processors that its unselected dependencies depend on. Every dependency must name a configured processor, the dependencies must not form a cycle, and processor names must be unique acrossprocessorsandp_processors.log_level:"debug","info","warn"or"error", the level of the processor's records in the log files. With"debug", the debug records of JuLeAna, the Legend* packages, BAT*, RadiationDetectorDSP, RadiationDetectorSignals, RadiationSpectra, SolidStateDetectors and their extensions are written as well while the processor runs.config.processor_log_level, when set, replaces this value for all processors.plot_level:"info"or"debug". With"info", the processor saves its normal plots; with"debug", it also saves its debug plots.config.processor_plot_level, when set, replaces this value for all processors.
Some processors select detectors with the detectors field:
detectors: the detectors the processor works on, as a string in the syntax ofLegendDataManagement.ljl_propfunc. The processor loads the channel info of its system and keeps the rows for which the expression is true, so the expression can use any column of the channel info table, for exampleprocessable,usability,system,det_type,low_aoe_statusorlq_status. Examples:"processable && usability != :off"or"processable && usability == :on && (low_aoe_status == :valid || low_aoe_status == :present)". Use"true"to keep every row. The expression is checked when the config is loaded; a column that does not exist throws an error when the processor starts. A processor that selects detectors and has nodetectorsentry stops with an error when it starts.process_evt_phydoes not select detectors and ignores the field. The shipped config sets"processable"for all processors except the PSD processors, which also require a validlow_aoe_statusorlq_status(andusability == :onat run level), andprocess_dsp_aux_phyandp_process_skm_phy, which keep all rows.
Two fields are optional:
report_status: how the rows of the processor report give theProcessor Status, see Report status. A table from the detector usability (on,acoroff) to a table from the row status (warningorfailure) to the status the row gives the processor (success,warningorfailure). Entries that are not set keep the default ofJuLeAna.DEFAULT_REPORT_STATUS; an unknown usability, row status or status is an error. Example, which makes a failingoffdetector a success and a warning of anacdetector a failure:report_status: ac: {warning: failure} off: {failure: success}error_plots:trueorfalse(the default). Withtrue, a failed step of the processor saves the debug plot of itson_errorhook, seeJuLeAna.@process_step. The plot is saved with the other plots of the detector, with a name starting witherror_.
A key or dependency that uses a former processor name from JuLeAna.RENAMED_PROCESSORS (for example process_decay_time for process_decay_time_cal) is read as the current name, and a warning names both once when the config is loaded. A config that contains both names of one processor is an error.
String lists in the kwargs of a processor become Julia symbol vectors when the config is loaded: YAML subsystems: [geds, spms, pmts, aux] is passed as [:geds, :spms, :pmts, :aux]. Use plain strings in YAML, without a leading colon. Scalar strings, and lists outside processor kwargs, keep their types.
Building a config in Julia
ProcessingConfig can also be built from keywords, which is convenient for tests. run_steps and partition_steps return the enabled processors in processing order:
julia> cfg = ProcessingConfig(;
processors = Dict(
:process_dsp_cal => ProcessorSpec(; name = :process_dsp_cal, level = :run,
category = :cal, rank = 5, dependencies = [:p_process_decay_time_cal]),
:process_peak_split_cal => ProcessorSpec(; name = :process_peak_split_cal, level = :run,
category = :cal, rank = 1)),
p_processors = Dict(
:p_process_decay_time_cal => ProcessorSpec(; name = :p_process_decay_time_cal,
level = :partition, category = :cal, rank = 2)));
julia> run_steps(cfg)
2-element Vector{Symbol}:
:process_peak_split_cal
:process_dsp_cal
julia> partition_steps(cfg)
1-element Vector{Symbol}:
:p_process_decay_time_cal
julia> check_processor_kwargs(cfg) # all processors are registered and accept their kwargsAn inconsistent config is rejected when it is built:
julia> ProcessingConfig(;
processors = Dict(:process_dsp_cal => ProcessorSpec(; name = :process_dsp_cal,
level = :run, category = :cal, rank = 5, dependencies = [:p_process_unknown])),
p_processors = Dict{Symbol,ProcessorSpec}())
ERROR: ArgumentError: processing_config: key `processors.process_dsp_cal.dependencies` names unknown processor `p_process_unknown`