API: steps, plots and logging

root_exception(e)

Return the exception that caused e: CompositeException (its first entry), TaskFailedException, RemoteException and CapturedException are unwrapped, also when nested. Other exceptions are returned unchanged.

@process_step name block
@process_step name on_error = hook block

Run block as one named step of a processor and return its value. If block throws, log "<name> failed for <detector>: <error>" as an @error and throw a ProcessStepError with the root_exception of the failure. The detector is that of the item parallel is processing; without one, the message has no for <detector>.

name is a string, usually with interpolations; it is only evaluated when block fails. When steps are nested, only the innermost failing step logs: an outer step passes the ProcessStepError on unchanged. block has the variable scope of a try block: it can assign variables of the enclosing function, and variables first assigned in block are local to it.

Inside run_step with debug = true, the innermost failing step also captures a Failure with the local variables of the processor at the step, see JuLeAna.failure. With debug = :infiltrate, it also stops there in an Infiltrator prompt. With pause, a step whose name is selected stops after it succeeded, with the value of block as the variable step_value.

hook is a function without arguments that saves a debug plot of the failure, with the variables of the processor, and a plot name starting with error_; the report of run_step links such plots instead of embedding them. The innermost failing step calls it when the error_plots setting of the processor (or of run_step) is on, before the ProcessStepError is thrown. If hook throws, a warning is logged and the error of the step is thrown unchanged. hook is only evaluated when block fails.

Examples

result_simple, report_simple = @process_step "$e_type simple calibration" begin
    simple_calibration(e_uncal, th228_lines, left_window_sizes, right_window_sizes)
end

result_simple, report_simple = @process_step "$e_type simple calibration" on_error = () ->
        save_plot(lplot(e_uncal), l200, filekey, det, Symbol("error_simple_calibration_$e_type")) begin
    simple_calibration(e_uncal, th228_lines, left_window_sizes, right_window_sizes)
end

ProcessStepError <: Exception

The error thrown by @process_step when its block fails: the step name name, the item of parallel that was processed (nothing outside of parallel) and the root_exception exception of the failure. It prints as "<name>: <exception>".

Failure

A failed @process_step, captured by run_step with debug = true. The local variables of the processor at the step are its properties, e.g. f.aoe for a variable aoe, and propertynames(f) lists them. failure_info gives the step name, processor, item and error, failure_variables all variables, and JuLeAna.@locals defines them in the REPL.

The variables are those visible where the error is caught: the arguments and variables of the processor, also those assigned in enclosing blocks, but not the variables first assigned inside the failing block itself. The values are the objects themselves, not copies.

JuLeAna.failure()::Failure

The first Failure captured by the last call of run_step with debug = true, or the last failure of JuLeAna.rerun that replaced it. Throws an ErrorException if that call captured none.

Examples

run_step(:process_energy_calibration_cal, 3, 0; detectors = [:V01234A], debug = true)
f = JuLeAna.failure()
f.e_uncal           # a variable of the processor at the failed step
JuLeAna.@locals f   # all of them as variables in the REPL
JuLeAna.rerun(f; e_uncal = f.e_uncal[f.e_uncal .> 100])   # the step again, with a changed variable

JuLeAna.failures()::Vector{Failure}

The Failures captured by the last call of run_step, in the order they happened; empty unless that call had debug = true. JuLeAna.failure gives the first one.

failure_info(f::Failure)::NamedTuple

The fields of the Failure f besides the variables: name, the step name; step and args, the processor and its period and run; item, the item of parallel (nothing outside of it) and detector, its detector (nothing if it has none); exception, the root_exception of the failure, and backtrace; block, the expression of the step, and mod, the module it is in; file and line, where the step is.

failure_variables(f::Failure)::Dict{Symbol,Any}

The local variables captured with the Failure f, by name.

JuLeAna.@locals f

Define the variables captured with the Failure f as global variables of the calling module, e.g. the REPL, and return their names.

JuLeAna.rerun(f::Failure; overrides...)

Run the block of the failed step f again, with the variables captured in f and the variables in overrides set to the given values, and return the value of the block. The block runs in the module of the processor, with the item of f as the current item, and writes no parameters or validity, see write_pars.

If the block fails again, the new Failure replaces f in JuLeAna.failures, and the root error is thrown. To check a fix in the configuration, run run_step again, which reads everything anew.

Throws an ArgumentError, without running the block, if a name in overrides is not a captured variable, or if the block contains a return, a break or continue outside of a loop in the block, or a @goto, all of which leave the block.

Examples

f = JuLeAna.failure()
JuLeAna.rerun(f)                         # fails again in the same way
JuLeAna.rerun(f; window = (100, 200))    # with a changed variable

save_plot(fig::Makie.Figure, l200::LegendData, args...; cleanup::Bool = true)

Save fig at the plot path that savelfig of LegendDataManagement builds from l200 and args, e.g. save_plot(fig, l200, filekey, det, :fwhm) or save_plot(fig, l200, partition, filekey, det, :fwhm). The figure is written in each of the plot_formats of the processing config (see ProcessingConfig): png, and jls, the figure serialized with Serialization next to the PNG under the same name. A jls file can be read only with the same package versions and holds the whole Makie theme, several MB per figure.

With cleanup = true the figure is emptied after it is written, which releases its memory: it cannot be shown or saved again. cleanup = false keeps it, e.g. to add it to a report afterwards; the caller then empties it with empty!(fig) when done.

@debug_plot block

Run block, which builds and saves a plot, only when the plot level of the running processor is debug (see ProcessorSpec and run_step); otherwise skip it, so that the figure is not built. Return the value of block, or nothing when it is skipped.

Examples

@debug_plot begin
    fig = LegendMakie.lplot(report_simple, title = get_plottitle(filekey, det, "Simple Calibration"))
    save_plot(fig, l200, filekey, det, :simple_calibration)
end

processing_logger(; log_file=nothing, log_level=Logging.Info, console_level=Logging.Info)

Build the logger that init_process installs.

• With log_file, records are appended to that file, one timestamped entry per record, and the file is flushed after every record. Progress bar records of ProgressLogging (see show_progress) are not written. The file gets the records at or above the level of the running processor, and outside of a processor those at or above log_level (see ProcessorLevelLogger); at Logging.Debug this includes the debug records of the modules listed in the JULIA_DEBUG environment variable, and inside a processor those of the DEBUG_LOG_PACKAGES.
• With console_level not nothing, records go to a TerminalLogger wrapped in an UnderscoreSafeLogger. console_level = Logging.Info shows the debug records of the JULIA_DEBUG modules as well; a higher level shows only the records at or above it.

Throw an ArgumentError when neither sink is requested.

ProcessorLevelLogger(inner::AbstractLogger, level::LogLevel)

Logger that passes to inner the records at or above the log level of the running processor (see ProcessorSpec), or at or above level outside of a processor. Debug records pass when that level is Logging.Debug and their module is listed in JULIA_DEBUG, or, inside a processor, belongs to one of the DEBUG_LOG_PACKAGES. Records of other modules are rejected before their message is built.

@debug_progress desc for x in iterable
    ...
end

Run the loop and, when the log level of the running processor is debug (see ProcessorSpec and run_step), print a line [ Progress: <desc> <i>/<n> to stdout at the start of iteration i of n, and <desc> <n>/<n> after the loop. The lines are plain output, not log records, so that a worker's lines reach the head process; the Workers page of the terminal user interface shows only the newest line of each desc. The loop body may use continue.

Examples

@debug_progress "Filekey HPGe: $fk" for chinfo_det in chinfo
    detector_selected(chinfo_det.detector) || continue
    ...
end

DEBUG_LOG_PACKAGES

Name prefixes of the packages whose debug records a processor at log level debug writes: JuLeAna, the Legend packages, BAT and the JuliaPhysics packages of the dataflow. A module counts when its name or the name of one of its parent modules starts with one of them, so submodules and package extensions such as LegendMakieExt are included.

process_log_file(log_path::AbstractString)

Return the log file of the current process in the directory log_path: head.log on the main process, worker-<id>-<hostname>.log on a worker.

parse_log_level(level::AbstractString)

Return the Logging.LogLevel named by level: "debug", "info", "warn" or "error". Throw an ArgumentError for any other value.

Examples

julia> JuLeAna.parse_log_level("warn")
Warn

UnderscoreSafeLogger(inner::AbstractLogger)

Logger that passes every record to inner, with string messages escaped by escape_log_underscores. Wrap a TerminalLogger in it so that names like process_dsp_cal are shown as written instead of in italics. Level filtering and exception handling are those of inner.

Examples

using Logging, TerminalLoggers
global_logger(UnderscoreSafeLogger(TerminalLogger()))

escape_log_underscores(message::AbstractString)

Return message with every unescaped _ outside of code spans replaced by \_, so that a Markdown renderer such as TerminalLogger does not read underscores in names as emphasis.

Examples

julia> JuLeAna.escape_log_underscores("process_dsp_cal in `p_process_energy_cal`")
"process\\_dsp\\_cal in `p_process_energy_cal`"