API: controller and workers

The headless state of an interactive dataflow session, the commands that change it, and the workers it starts and ends.

Controller

JuLeAna.DataflowController — Type
mutable struct DataflowController

Headless state of an interactive dataflow session. All changes go through dispatch! with a ControllerCommand; snapshot returns a read-only view of the state and progress the progress of the current or last processing run. Commands may be sent from any task; long-running ones (loading the session, reloading the config, processing) run in the background.

The controller is in one of the states

  • :idle: no session loaded yet,
  • :loading: the session or config is being loaded,
  • :ready: a session is loaded and no processing runs,
  • :running: processing runs,
  • :stopping: processing was asked to stop and runs until the running items finish.

Constructors:

  • DataflowController(args::AbstractDict; processors = PROCESSORS, console = true) starts :idle with the parsed command line options args; LoadSession sets up the session, with terminal log output of this process only if console.
  • DataflowController(session::DataflowSession; processors = PROCESSORS) starts :ready with an existing session.

processors maps processor names to their functions. Both constructors take the keyword workers, the JuLeAna.WorkerManager of the worker allocations, and record_software: whether each processing run first records the software of this process in the setup/software folder of the production (see JuLeAna.record_software), and record_metadata: whether it records the metadata and the config files (the data config files and the processing config args["config"]) in its setup/metadata folder (see JuLeAna.record_metadata). A run fails if a record fails.

source
JuLeAna.dispatch! — Function
dispatch!(ctrl::DataflowController, cmd::ControllerCommand)

Apply cmd to ctrl and return nothing. Throw an ArgumentError that explains why when the command is not possible in the controller's state or its arguments are invalid; the controller is then unchanged. Commands that run in the background return as soon as they are started; see snapshot for their state and wait(ctrl) to block until they are done.

source
JuLeAna.snapshot — Function
snapshot(ctrl::DataflowController)

Return a JuLeAna.ControllerSnapshot, a NamedTuple with fixed field types, with a copy of the controller state:

  • state, outcome (:finished, :failed, :stopped or nothing of the last processing run), last_error (message or nothing), started, finished (UTC times of the last processing run or nothing);
  • config_file, log_path, dataset (the name of the dataset, "default" for every run);
  • available_periods, available_runs (Dict period => runs of the dataset), periods (selected periods), runs (selected runs: nothing for all runs, a vector for the same runs in every period, or a dict of runs per period where a period without an entry keeps all its runs), run_steps, partition_steps (selected steps in processing order);
  • processors: one NamedTuple per configured processor, run level first and then by rank, with name, level, category, rank, enabled, selected, dependencies, kwargs (with the overrides applied), config_kwargs (as in the config), accepted (the keyword arguments the processor function accepts besides the ones the dataflow passes itself, nothing if it accepts any), overridden (names of the overridden kwargs, and :log_level and :plot_level if overridden), log_level, plot_level (with the overrides of SetStepLevel applied) and config_log_level, config_plot_level (as in the config);
  • reprocess, processor_log_level, processor_plot_level, analysis_runs_only and runmode (the settings in effect), runmode_settings (local_n, local_julia_settings, slurm_flags and slurm_julia_settings in effect, or nothing without a session), check_dependencies, nworkers (workers in the pool), allocations (the worker allocations, see JuLeAna.allocations, without their output), idle_deadline (UTC time at which idle allocations are cancelled, or nothing, see JuLeAna.idle_deadline).
source
Base.wait — Method
wait(ctrl::DataflowController)

Block until the background task of the last loading or processing command has ended.

source
JuLeAna.progress — Function
progress(store::ProgressStore)
progress(ctrl::DataflowController)

Return a copy of the progress of the current or last processing run as a JuLeAna.ProgressSummary, a NamedTuple with fixed field types:

  • started, finished: UTC times of the run, nothing if not (yet) reached.
  • fraction: overall progress in [0, 1], the mean of the step fractions. A finished, failed, skipped, stopped or cancelled step counts as 1; a waiting step as 0; a running step as (done + failed + notrun) / (total + 1) of its items, so that the overall progress reaches 1 only after the last step has finished.
  • eta: estimated seconds until the run finishes, from the elapsed time and fraction, or nothing before any progress or after the run.
  • counts: number of steps per state, a NamedTuple with one field per state (waiting, running, done, failed, skipped, stopped, cancelled).
  • steps: one NamedTuple per step in schedule order with period, run (nothing for partition-level steps), step, state, total, done, failed, notrun, workers (sorted workers of the running items), started, finished, error (the error message of a failed step, else nothing), nmessages (number of messages kept for the step, see step_messages), fraction, rate (items done or failed per second since the step queued its first item, up to the step's end; nothing before the first item ended) and eta (seconds for the remaining items of a running step at that rate, or nothing).
  • groups: one NamedTuple per period (partition level) and per run, in schedule order, with period, run, nsteps, counts and fraction over its steps.
source
JuLeAna.busy_workers — Function
busy_workers(store::ProgressStore)
busy_workers(ctrl::DataflowController)

Return the items that workers are running, oldest first, as NamedTuples with worker, item (the item's label), period, run (nothing for partition-level steps), step, since (UTC time the worker got the item) and abandoned. An abandoned item belongs to a processing run that was stopped without waiting for it; its worker stays busy until the item's call returns, which for an unresponsive worker never happens. For a controller, the items of workers that have ended are left out: they wait for a retry on another worker.

source
JuLeAna.step_messages — Function
step_messages(store::ProgressStore, key)
step_messages(ctrl::DataflowController, key)

Return a copy of the messages logged on this process while step key of the current or last processing run ran, oldest first, as NamedTuples with level (a Logging.LogLevel), time (UTC) and text (possibly several lines). key is (period, run, step) with run = nothing for a partition-level step. Only the last 500 messages of a step are kept. Returns an empty vector for a step that is not part of the run.

source
JuLeAna.ProgressStore — Type
ProgressStore()

Thread-safe record of the progress of a processing run, filled by run_processing and parallel when run_processing is given the store as progress_store. It holds one entry per scheduled step of a period (partition level) or of a run (run level) with the step's state, the number of items handed to parallel and how many of them finished, failed, were not run or are running on which worker. Messages logged on this process while a step runs are kept with the step (see step_messages) when the logger in use passes them to the store, as JuLeAna.MessageBuffer, the logger of the terminal user interface, does.

The store also records the busy workers: every item that a worker runs, from the moment the item is handed to the worker until its call returns. This record is kept across processing runs, so that items whose run was stopped without waiting for them stay listed until their workers are free again. When run_processing times a processing run, the store keeps the duration of every item that ends done, per processor over all periods and runs.

progress returns a copy of the current state with the aggregated progress, busy_workers the busy workers and JuLeAna.item_timings the item durations.

source
JuLeAna.item_timings — Function
JuLeAna.item_timings(store::ProgressStore)

Return the durations in seconds of the items of the current or last processing run that ended done, per processor over all periods and runs, as a vector of (; process, items, min, median, p95, p99, max) sorted by process, where items is the number of timed items, p95 the 95th and p99 the 99th percentile. A processor without a done item is left out, and so is every processor of a processing run that is not timed. The duration of an item runs from the moment its last worker got it until its call returned.

source
JuLeAna.show_progress — Function
show_progress(f, store::ProgressStore; interval::Real = 1)

Call f() and, while it runs, log the progress in store every interval seconds as progress records of ProgressLogging: one bar for the overall progress and one below it for each running step. A TerminalLogger draws these bars below the log messages and keeps a line with the final state of each step whose bar it closes; other loggers receive them as Info records whose message is a ProgressLogging.Progress. All bars are closed when f returns or throws. Returns the value of f().

source
JuLeAna.LoadSession — Type
LoadSession()

Set up the session from the controller's command line options with setup_session: read the config and the LEGEND data and start the workers. Runs in the background; the controller is :loading until it is :ready. Allowed when the controller is :idle.

source
JuLeAna.ReloadConfig — Type
ReloadConfig()

Read the processing config file again with the original command line options, keeping the log directory, and reset the selection to the config's periods, runs and enabled processors. Drops all kwarg overrides. Runs in the background like LoadSession.

source
JuLeAna.SelectPeriods — Type
SelectPeriods(periods)

Select the periods to process, given as DataPeriods or anything DataPeriod accepts. Every period must be in the runinfo of the dataset. Periods that stay selected keep their runs.

source
JuLeAna.SelectRuns — Type
SelectRuns(runs)

Select the runs to process.

  • nothing selects all runs of the selected periods.
  • A vector of DataRuns (or anything DataRun accepts) selects the same runs in every selected period; every run must be available in at least one selected period.
  • A dict of period => runs (keys and values as DataPeriod and DataRun accept) selects exactly these periods, each with its runs, where nothing stands for all runs of the period. Every period and every run of it must be in the dataset.
source
JuLeAna.SelectSteps — Type
SelectSteps(run_steps, partition_steps)

Select the run-level and partition-level processors to run, by name. Every name must be a processor of that level in the config, enabled or not. The steps run in the order of their dependencies, see run_processing.

source
JuLeAna.SetReprocess — Type
SetReprocess(reprocess::Union{Nothing,Bool})

Set the reprocess keyword passed to every processor: true or false override the config, nothing keeps the configured values.

source
JuLeAna.SetProcessorLogLevel — Type
SetProcessorLogLevel(level::Union{Nothing,Logging.LogLevel})

Set the level at which the records of every processor are written to the log files, for example Logging.Debug, replacing the log_level of each processor entry. nothing keeps the levels of the processor entries. The level starts as processor_log_level of the config.

source
JuLeAna.SetProcessorPlotLevel — Type
SetProcessorPlotLevel(level::Union{Nothing,Symbol})

Set the plot level of every processor, :info or :debug, replacing the plot_level of each processor entry. nothing keeps the levels of the processor entries. The level starts as processor_plot_level of the config.

source
JuLeAna.SetAnalysisRunsOnly — Type
SetAnalysisRunsOnly(only::Bool)

Choose whether processing skips the runs that are not analysis runs (true) or processes every run in the run info (false), instead of the setting of the config.

source
JuLeAna.SetRunmode — Type
SetRunmode(runmode::Symbol)

Use the run mode runmode, :local or :slurm, instead of the one of the config, for the next worker allocations; running allocations are not changed.

source
JuLeAna.SetRunmodeSettings — Type
SetRunmodeSettings(; local_n = nothing, local_julia_settings = nothing,
    slurm_flags = nothing, slurm_julia_settings = nothing)

Use the given settings of the run modes instead of those of the config, for the next worker allocations and the start script; running allocations are not changed. Also possible during a processing run, which keeps its config. Each argument that is nothing keeps the setting in effect. local_n is the number of local workers and must be positive; local_julia_settings, slurm_flags and slurm_julia_settings are vectors of strings, the extra options of the local Julia processes, the flags of the SLURM job and the options of the Julia processes it starts.

source
JuLeAna.SetCheckDependencies — Type
SetCheckDependencies(check::Bool)

Choose whether a processing run rebuilds the dependency graph for the selected steps and waits for their dependencies (true), or runs the selected steps on the current graph without waiting (false).

source
JuLeAna.SetKwarg — Type
SetKwarg(step, key, value)

Override the keyword argument key of the processor step for this session. The processor function must accept key.

source
JuLeAna.SetStepLevel — Type
SetStepLevel(step, key, value)

Override the log level (key = :log_level, value a LogLevel or one of "debug", "info", "warn" and "error") or the plot level (key = :plot_level, value :info or :debug) of the processor step for this session. A level set for all processors with SetProcessorLogLevel or SetProcessorPlotLevel takes precedence.

source
JuLeAna.ResetKwargs — Type
ResetKwargs(step = nothing, key = nothing)

Drop the kwarg and level overrides (see SetKwarg and SetStepLevel) of all processors, of the processor step, or only the override of key (a kwarg, :log_level or :plot_level) of the processor step.

source
JuLeAna.StartProcessing — Type
StartProcessing()

Run the selected steps over the selected periods and runs in the background with run_processing, using the config with the session's kwarg overrides. The controller is :running until the run ends; the outcome is :finished, :failed or :stopped. progress shows the progress of the run. When the run ends, its steps are added to the processing history of the data production (see JuLeAna.append_history and JuLeAna.history_path). Throws an ArgumentError when no step is selected or a selected processor does not accept its keyword arguments.

source
JuLeAna.StopProcessing — Type
StopProcessing(; force::Bool = false)

Stop the processing run: no further step or item starts, items running on workers finish, and the run ends as :stopped. With force = true, the run ends without waiting for the running items. They count as not run, go on in the background and keep their workers busy until they end; busy_workers lists them as abandoned. No worker is stopped.

source
JuLeAna.WriteConfig — Type
WriteConfig(path::AbstractString)
WriteConfig()

Write the processing config file of the session to path with the current settings of the session: the kwarg overrides are applied to the processor entries, every processor is enabled exactly when it is selected, and processing.analysis_runs_only is the setting in effect. Without path, the file is processing_config.yaml in the JuLeAna.production_config_dir, which is created if needed. An existing file at the path is first copied to path * ".bak". The file is written as YAML. The written file becomes the config file of the session, which ReloadConfig reads.

source
JuLeAna.CreateProduction — Type
CreateProduction(spec::ProductionSpec)

Create the data production spec describes with JuLeAna.create_production in the background and switch the session to it: cancel all worker allocations (their workers use the data of the previous production), set LEGEND_DATA_CONFIG of this process to the new config.yaml, and load a session with the JuLeAna.install_default_config file of the new production and a new log folder. The lines of the creation are kept for production_log. Possible while no session is loaded or no processing runs; the controller is :loading meanwhile and returns to its previous state if the creation or the load fails.

source
JuLeAna.SelectProduction — Type
SelectProduction(path::AbstractString)

Switch the session to the existing data production in the folder path, which must hold a config.json or config.yaml (config.json if both exist): cancel all worker allocations, set LEGEND_DATA_CONFIG of this process to that file, reset the dataset to default, and load a session with the JuLeAna.install_default_config file of the production and a new log folder. Possible while no session is loaded or no processing runs; the controller is :loading meanwhile and is :idle without a session if the load fails.

source
JuLeAna.SelectDataset — Type
SelectDataset(dataset::Symbol)

Reload the session for the dataset dataset, a name in the metadata datasets/runlists of the data production or :default (every run), and cancel all worker allocations. The dataset is the --dataset option. Possible while no session is loaded or no processing runs; a dataset that is not in the metadata makes the load fail, leaving the controller :idle without a session.

source
JuLeAna.check_idle! — Method
JuLeAna.check_idle!(ctrl::DataflowController)

Cancel the worker allocations of ctrl if they have been idle for the idle_timeout_minutes of the config (see JuLeAna.check_idle!); allocations are never idle while processing runs or is stopping. Return whether they were cancelled. The terminal user interface calls it with every read of the controller state.

source

Workers

JuLeAna.WorkerManager — Type
WorkerManager(; squeue::Cmd = `squeue`, scancel::Cmd = `scancel`, sinfo::Cmd = `sinfo`,
              sacct::Cmd = `sacct`, sstat::Cmd = `sstat`)

The worker allocations of this process and the worker start script they run.

The start script is generated from the processing config of the session by JuLeAna.start_script! and saved as startjlworkers.sh in the JuLeAna.production_config_dir. It holds the cluster cookie and the address of this process, so it is generated anew for every config (JuLeAna.set_worker_config!) and can be edited for the rest of the session (JuLeAna.edit_start_script!). JuLeAna.add_workers! runs it as a new allocation, JuLeAna.cancel_allocation! and JuLeAna.cancel_all! end allocations, JuLeAna.allocations lists them.

Running allocations are cancelled when nothing has used them for the idle_timeout_minutes of the config: see JuLeAna.idle_deadline and JuLeAna.check_idle!. JuLeAna.check_joining! warns when started workers wait to join this process.

squeue, scancel, sinfo, sacct and sstat are the SLURM commands, without arguments; SLURM counts as available when the program of squeue is found. JuLeAna.workers_overview collects everything the Workers page shows. All functions may be called from any task.

source
JuLeAna.WorkerAllocation — Type
WorkerAllocation

One run of the worker start script, started by JuLeAna.add_workers!. Fields:

  • id::Int: number of the allocation, counted from 1 per JuLeAna.WorkerManager.
  • backend::Symbol: :local or :slurm.
  • job_name: the SLURM job name of the allocation's workers, nothing for :local.
  • started, finished: UTC times; finished is nothing while the script runs.
  • process: the Base.Process of the shell that runs the script, the leader of its own process group.
  • state::Symbol: :running, :ended (the script exited with code 0), :failed (any other exit) or :cancelled.
  • exit_code: the exit code of the script, nothing while it runs.
  • job_ids::Vector{String}: the SLURM job IDs of the allocation at the last query.
  • worker_errors::Int: the number of workers of the allocation that ended with an exit code other than 0 while it ran, as the lines Worker slot <n>: the worker ended after <s> s with exit code <code> of the script report them (see JuLeAna.worker_start_script).
  • output::Vector{String}: the last 1000 lines of the standard and error output of the script.
source
JuLeAna.set_worker_config! — Function
JuLeAna.set_worker_config!(mgr::WorkerManager, config::ProcessingConfig)

Use the run mode of config for the next allocations of mgr. The start script is generated anew on its next use; running allocations are not changed.

source
JuLeAna.worker_start_script — Function
JuLeAna.worker_start_script(cfg::ProcessingConfig)

Return the text of the bash script that starts the workers of the run mode of cfg and connects them to this process. A worker that ends is started again in its slot after cfg.worker_restart_delay_seconds, and the script prints Worker slot <n>: the worker ended after <s> s with exit code <code> for its end, so the script runs until it is ended, as long as this process runs: when a worker ends and this process has not touched its heartbeat file for cfg.worker_heartbeat_timeout_minutes (see JuLeAna.add_workers!), the script ends the allocation, the local workers with SIGTERM to its process group, the SLURM job with scancel.

In the local run mode the script starts cfg.local_n workers on the host it runs on.

In the slurm run mode it submits one job with sbatch --wait, the job name julia-<pid of this process>-0 (see JuLeAna.add_workers!) and the SLURM flags of cfg except --cpu-bind, and prints the output of the job until the job ends (it is cancelled or reaches its time limit); ended with SIGTERM, it cancels the job by its name. The output goes to julia-workers-<pid of the script>.out in the log directory of cfg, or in the current directory if cfg has none. The job runs every worker as its own job step on one node, with the CPUs per task and the --cpu-bind and --mem-bind flags of cfg. With cfg.slurm_worker_memory === :shared the steps share the memory of the job; with :fixed each step gets the memory of the job per worker. Throws an ArgumentError for :fixed without --mem or --mem-per-cpu in the flags, and for a --cpu-bind or --mem-bind flag without =.

source
JuLeAna.edit_start_script! — Function
JuLeAna.edit_start_script!(mgr::WorkerManager, text::AbstractString)

Replace the start script of mgr by text and save it to the script file. For the slurm run mode the script must name its SLURM job --job-name=julia-<n>-<m>: each allocation replaces the name by its own, by which it finds and cancels its jobs. Throws an ArgumentError for a SLURM script without such a name.

source
JuLeAna.add_workers! — Function
JuLeAna.add_workers!(mgr::WorkerManager)

Run the start script of mgr (see JuLeAna.start_script!) with bash in the background as a new allocation and return its JuLeAna.WorkerAllocation. The shell leads its own process group, so it keeps running when this process ends unless it is cancelled. Its standard and error output are kept in the allocation. For the slurm run mode, the job name of the script is replaced by julia-<pid of this process>-<n> with a new n. The started workers connect to this process and join its worker pool.

From the first allocation on, this process touches its heartbeat file juleana-<host>-<pid>.alive in the JuLeAna.production_config_dir every minute until it exits; the start scripts end their allocation when the file has not been touched for the worker_heartbeat_timeout_minutes of the config (see JuLeAna.worker_start_script).

source
JuLeAna.update_job_ids! — Function
JuLeAna.update_job_ids!(mgr::WorkerManager)

Query squeue for the jobs of the running SLURM allocations of mgr and store their IDs in the allocations. Does nothing without such allocations.

source
JuLeAna.update_steps! — Function
JuLeAna.update_steps!(mgr::WorkerManager; at::Real = time())

Query sacct and sstat for the job steps of the workers of the SLURM allocations of mgr (not the batch and extern steps) and keep them as JuLeAna.StepInfos by job ID. Queries the jobs of the running allocations (see JuLeAna.update_job_ids!) and, until their steps have ended, the jobs that had running steps at the previous query. Does nothing less than 10 seconds after the previous query (at is the current time()) or without such jobs.

source
JuLeAna.WorkerAttributes — Type
JuLeAna.WorkerAttributes

The attributes a worker process reports once (see JuLeAna.worker_attributes), a NamedTuple with the fields

  • host::String: the host name;
  • os_pid::Int: the process ID on the host;
  • threads::Int: the number of Julia threads;
  • memory::Int: the bytes of memory the process may use: the smallest memory limit of its control group and the groups above it (such as its SLURM job step and job), else Sys.total_memory();
  • job, step: the SLURM job and step ID, "" outside SLURM;
  • tasks: the number of tasks of the SLURM step (SLURM_STEP_NUM_TASKS), "" outside SLURM;
  • cpus: the CPUs per task of the SLURM step (SLURM_CPUS_PER_TASK), "" if not set;
  • gpus: the GPUs of the SLURM step (SLURM_STEP_GPUS), "" without GPUs.
source
JuLeAna.StepInfo — Type
JuLeAna.StepInfo

A SLURM job step of a worker allocation as sacct and sstat report it (see JuLeAna.update_steps!), a NamedTuple with the fields

  • id::String: <job ID>.<step ID>;
  • node::String: the node of the step;
  • state::String: the state, such as RUNNING, COMPLETED, CANCELLED or OUT_OF_MEMORY;
  • exit_code::String: <exit code>:<signal>;
  • elapsed::String: the run time as sacct prints it;
  • rss: the resident memory in bytes at the last sample, nothing once the step ended;
  • max_rss: the peak of the resident memory in bytes, nothing if not reported. SLURM samples the memory every few seconds, so a short peak can be missing;
  • cpu_load: the CPUs the step used over the last minute, its CPU time since the oldest query of the last minute divided by the time since then; nothing at its first query and once the step ended;
  • cpu_mean: the CPUs the step used over its whole run, its CPU time divided by its run time; nothing once the step ended.

The CPU time is the AveCPU of sstat, the CPU time of the step's task, which is all of the step's CPU time for a worker step with its one task.

source
JuLeAna.worker_attributes — Function
JuLeAna.worker_attributes(pid::Integer)

Return the JuLeAna.WorkerAttributes of the worker process pid. The first call asks the worker, later calls return the same attributes without asking. The worker must answer, so the first call must come while it is not busy.

source
JuLeAna.kill_worker — Function
JuLeAna.kill_worker(pid::Integer; scancel::Cmd = `scancel`)

End the process of the worker pid at once with SIGKILL, also while it computes and does not answer: a worker that is the only task of a SLURM job step (other than the step of this process) with scancel --signal=KILL <job>.<step>, a worker on this host with kill -9. Its JuLeAna.worker_attributes must have been asked for before. Distributed then removes the worker, and the calls running on it throw a ProcessExitedException.

Throws an ArgumentError for the main process, for a worker whose attributes are not known, and for a worker on another host that is not the only task of its job step.

source
JuLeAna.worker_job_ended — Function
JuLeAna.worker_job_ended(pid::Integer; squeue::Cmd = `squeue`)

Whether the SLURM job of the worker pid no longer runs, so that a worker that ended ended with its allocation, at its time limit or cancelled, and not through what it ran. squeue reports the state of the job; any state but RUNNING counts as ended. false for a worker whose JuLeAna.worker_attributes are not known or that is not in a SLURM job, and when squeue reports no state or cannot run.

source
JuLeAna.cancel_allocation! — Function
JuLeAna.cancel_allocation!(mgr::WorkerManager, id::Integer)

End the allocation id of mgr if it is running: for a SLURM allocation, scancel the jobs that squeue lists for this user under the allocation's job name (and no others), then send SIGTERM to the process group of its script, which ends the local workers or sbatch. The workers of the allocation leave the worker pool; items running on them fail. Throws an ArgumentError for an unknown id.

source
JuLeAna.allocations — Function
JuLeAna.allocations(mgr::WorkerManager; output::Bool = true)

Return a copy of the allocations of mgr as NamedTuples with the fields of JuLeAna.WorkerAllocation except process, and without output unless output: a Vector{JuLeAna.AllocationOutputInfo}, or a Vector{JuLeAna.AllocationInfo} without output.

source
JuLeAna.mark_activity! — Function
JuLeAna.mark_activity!(mgr::WorkerManager, time::DateTime = now(UTC))

Record that the workers of mgr were used at time (UTC), such as by the end of a processing run or a postponement: the idle time of JuLeAna.idle_deadline counts from the latest such time.

source
JuLeAna.idle_deadline — Function
JuLeAna.idle_deadline(mgr::WorkerManager; processing::Bool)

Return the UTC time at which JuLeAna.check_idle! cancels the running allocations of mgr: the last activity (see JuLeAna.mark_activity!; a new allocation counts too) plus the idle_timeout_minutes of the config. Return nothing while processing, if no allocation runs, without a config, or if the timeout is 0.

source
JuLeAna.joining_workers — Function
JuLeAna.joining_workers(mgr::WorkerManager)

Return the number of started workers that have connected to this process and wait for the cluster manager of ParallelProcessingTools to add them: the manager adds the waiting workers in batches, one batch at a time, and the workers of the batch it is adding are not counted. 0 before the first allocation of mgr.

source
JuLeAna.check_joining! — Function
JuLeAna.check_joining!(mgr::WorkerManager; at::Real = time(), joining::Integer = JuLeAna.joining_workers(mgr))

Warn once when workers have waited to join this process (joining, see JuLeAna.joining_workers) at every call for 60 seconds (at is the current time()): a worker that is not added within its JULIA_WORKER_TIMEOUT ends. After such a warning, log when no worker waits anymore. Return joining. The terminal user interface calls it with every read of the controller state.

source
JuLeAna.workers_overview — Function
JuLeAna.workers_overview(mgr::WorkerManager)

Collect what the Workers page shows and return a JuLeAna.WorkersOverview (; allocations, script, workers, joining, steps, slurm):

Runs SLURM commands, so it can take a while; call it in the background.

source