API: controller and workers
The headless state of an interactive dataflow session, the commands that change it, and the workers it starts and ends.
Controller
JuLeAna.DataflowController — Type
mutable struct DataflowControllerHeadless state of an interactive dataflow session. All changes go through dispatch! with a ControllerCommand; snapshot returns a read-only view of the state and progress the progress of the current or last processing run. Commands may be sent from any task; long-running ones (loading the session, reloading the config, processing) run in the background.
The controller is in one of the states
:idle: no session loaded yet,:loading: the session or config is being loaded,:ready: a session is loaded and no processing runs,:running: processing runs,:stopping: processing was asked to stop and runs until the running items finish.
Constructors:
DataflowController(args::AbstractDict; processors = PROCESSORS, console = true)starts:idlewith the parsed command line optionsargs;LoadSessionsets up the session, with terminal log output of this process only ifconsole.DataflowController(session::DataflowSession; processors = PROCESSORS)starts:readywith an existing session.
processors maps processor names to their functions. Both constructors take the keyword workers, the JuLeAna.WorkerManager of the worker allocations, and record_software: whether each processing run first records the software of this process in the setup/software folder of the production (see JuLeAna.record_software), and record_metadata: whether it records the metadata and the config files (the data config files and the processing config args["config"]) in its setup/metadata folder (see JuLeAna.record_metadata). A run fails if a record fails.
JuLeAna.dispatch! — Function
dispatch!(ctrl::DataflowController, cmd::ControllerCommand)Apply cmd to ctrl and return nothing. Throw an ArgumentError that explains why when the command is not possible in the controller's state or its arguments are invalid; the controller is then unchanged. Commands that run in the background return as soon as they are started; see snapshot for their state and wait(ctrl) to block until they are done.
JuLeAna.snapshot — Function
snapshot(ctrl::DataflowController)Return a JuLeAna.ControllerSnapshot, a NamedTuple with fixed field types, with a copy of the controller state:
state,outcome(:finished,:failed,:stoppedornothingof the last processing run),last_error(message ornothing),started,finished(UTC times of the last processing run ornothing);config_file,log_path,dataset(the name of the dataset,"default"for every run);available_periods,available_runs(Dictperiod => runs of the dataset),periods(selected periods),runs(selected runs:nothingfor all runs, a vector for the same runs in every period, or a dict of runs per period where a period without an entry keeps all its runs),run_steps,partition_steps(selected steps in processing order);processors: oneNamedTupleper configured processor, run level first and then by rank, withname,level,category,rank,enabled,selected,dependencies,kwargs(with the overrides applied),config_kwargs(as in the config),accepted(the keyword arguments the processor function accepts besides the ones the dataflow passes itself,nothingif it accepts any),overridden(names of the overridden kwargs, and:log_leveland:plot_levelif overridden),log_level,plot_level(with the overrides ofSetStepLevelapplied) andconfig_log_level,config_plot_level(as in the config);reprocess,processor_log_level,processor_plot_level,analysis_runs_onlyandrunmode(the settings in effect),runmode_settings(local_n,local_julia_settings,slurm_flagsandslurm_julia_settingsin effect, ornothingwithout a session),check_dependencies,nworkers(workers in the pool),allocations(the worker allocations, seeJuLeAna.allocations, without their output),idle_deadline(UTC time at which idle allocations are cancelled, ornothing, seeJuLeAna.idle_deadline).
JuLeAna.progress — Function
progress(store::ProgressStore)
progress(ctrl::DataflowController)Return a copy of the progress of the current or last processing run as a JuLeAna.ProgressSummary, a NamedTuple with fixed field types:
started,finished: UTC times of the run,nothingif not (yet) reached.fraction: overall progress in[0, 1], the mean of the step fractions. A finished, failed, skipped, stopped or cancelled step counts as1; a waiting step as0; a running step as(done + failed + notrun) / (total + 1)of its items, so that the overall progress reaches1only after the last step has finished.eta: estimated seconds until the run finishes, from the elapsed time andfraction, ornothingbefore any progress or after the run.counts: number of steps per state, aNamedTuplewith one field per state (waiting,running,done,failed,skipped,stopped,cancelled).steps: oneNamedTupleper step in schedule order withperiod,run(nothingfor partition-level steps),step,state,total,done,failed,notrun,workers(sorted workers of the running items),started,finished,error(the error message of a failed step, elsenothing),nmessages(number of messages kept for the step, seestep_messages),fraction,rate(items done or failed per second since the step queued its first item, up to the step's end;nothingbefore the first item ended) andeta(seconds for the remaining items of a running step at that rate, ornothing).groups: oneNamedTupleper period (partition level) and per run, in schedule order, withperiod,run,nsteps,countsandfractionover its steps.
JuLeAna.busy_workers — Function
busy_workers(store::ProgressStore)
busy_workers(ctrl::DataflowController)Return the items that workers are running, oldest first, as NamedTuples with worker, item (the item's label), period, run (nothing for partition-level steps), step, since (UTC time the worker got the item) and abandoned. An abandoned item belongs to a processing run that was stopped without waiting for it; its worker stays busy until the item's call returns, which for an unresponsive worker never happens. For a controller, the items of workers that have ended are left out: they wait for a retry on another worker.
JuLeAna.step_messages — Function
step_messages(store::ProgressStore, key)
step_messages(ctrl::DataflowController, key)Return a copy of the messages logged on this process while step key of the current or last processing run ran, oldest first, as NamedTuples with level (a Logging.LogLevel), time (UTC) and text (possibly several lines). key is (period, run, step) with run = nothing for a partition-level step. Only the last 500 messages of a step are kept. Returns an empty vector for a step that is not part of the run.
JuLeAna.ProgressStore — Type
ProgressStore()Thread-safe record of the progress of a processing run, filled by run_processing and parallel when run_processing is given the store as progress_store. It holds one entry per scheduled step of a period (partition level) or of a run (run level) with the step's state, the number of items handed to parallel and how many of them finished, failed, were not run or are running on which worker. Messages logged on this process while a step runs are kept with the step (see step_messages) when the logger in use passes them to the store, as JuLeAna.MessageBuffer, the logger of the terminal user interface, does.
The store also records the busy workers: every item that a worker runs, from the moment the item is handed to the worker until its call returns. This record is kept across processing runs, so that items whose run was stopped without waiting for them stay listed until their workers are free again. When run_processing times a processing run, the store keeps the duration of every item that ends done, per processor over all periods and runs.
progress returns a copy of the current state with the aggregated progress, busy_workers the busy workers and JuLeAna.item_timings the item durations.
JuLeAna.item_timings — Function
JuLeAna.item_timings(store::ProgressStore)Return the durations in seconds of the items of the current or last processing run that ended done, per processor over all periods and runs, as a vector of (; process, items, min, median, p95, p99, max) sorted by process, where items is the number of timed items, p95 the 95th and p99 the 99th percentile. A processor without a done item is left out, and so is every processor of a processing run that is not timed. The duration of an item runs from the moment its last worker got it until its call returned.
JuLeAna.show_progress — Function
show_progress(f, store::ProgressStore; interval::Real = 1)Call f() and, while it runs, log the progress in store every interval seconds as progress records of ProgressLogging: one bar for the overall progress and one below it for each running step. A TerminalLogger draws these bars below the log messages and keeps a line with the final state of each step whose bar it closes; other loggers receive them as Info records whose message is a ProgressLogging.Progress. All bars are closed when f returns or throws. Returns the value of f().
JuLeAna.effective_config — Function
effective_config(ctrl::DataflowController)Return the session's ProcessingConfig with the kwarg overrides of the controller merged into the processor kwargs, its level overrides (see SetStepLevel) applied to the processors, and its analysis_runs_only, runmode and run mode settings (see SetRunmodeSettings) applied.
JuLeAna.ControllerCommand — Type
abstract type ControllerCommandSupertype of the commands that dispatch! applies to a DataflowController.
JuLeAna.LoadSession — Type
LoadSession()Set up the session from the controller's command line options with setup_session: read the config and the LEGEND data and start the workers. Runs in the background; the controller is :loading until it is :ready. Allowed when the controller is :idle.
JuLeAna.ReloadConfig — Type
ReloadConfig()Read the processing config file again with the original command line options, keeping the log directory, and reset the selection to the config's periods, runs and enabled processors. Drops all kwarg overrides. Runs in the background like LoadSession.
JuLeAna.SelectPeriods — Type
SelectPeriods(periods)Select the periods to process, given as DataPeriods or anything DataPeriod accepts. Every period must be in the runinfo of the dataset. Periods that stay selected keep their runs.
JuLeAna.SelectRuns — Type
SelectRuns(runs)Select the runs to process.
nothingselects all runs of the selected periods.- A vector of
DataRuns (or anythingDataRunaccepts) selects the same runs in every selected period; every run must be available in at least one selected period. - A dict of period => runs (keys and values as
DataPeriodandDataRunaccept) selects exactly these periods, each with its runs, wherenothingstands for all runs of the period. Every period and every run of it must be in the dataset.
JuLeAna.SelectSteps — Type
SelectSteps(run_steps, partition_steps)Select the run-level and partition-level processors to run, by name. Every name must be a processor of that level in the config, enabled or not. The steps run in the order of their dependencies, see run_processing.
JuLeAna.SetReprocess — Type
SetReprocess(reprocess::Union{Nothing,Bool})Set the reprocess keyword passed to every processor: true or false override the config, nothing keeps the configured values.
JuLeAna.SetProcessorLogLevel — Type
SetProcessorLogLevel(level::Union{Nothing,Logging.LogLevel})Set the level at which the records of every processor are written to the log files, for example Logging.Debug, replacing the log_level of each processor entry. nothing keeps the levels of the processor entries. The level starts as processor_log_level of the config.
JuLeAna.SetProcessorPlotLevel — Type
SetProcessorPlotLevel(level::Union{Nothing,Symbol})Set the plot level of every processor, :info or :debug, replacing the plot_level of each processor entry. nothing keeps the levels of the processor entries. The level starts as processor_plot_level of the config.
JuLeAna.SetAnalysisRunsOnly — Type
SetAnalysisRunsOnly(only::Bool)Choose whether processing skips the runs that are not analysis runs (true) or processes every run in the run info (false), instead of the setting of the config.
JuLeAna.SetRunmode — Type
SetRunmode(runmode::Symbol)Use the run mode runmode, :local or :slurm, instead of the one of the config, for the next worker allocations; running allocations are not changed.
JuLeAna.SetRunmodeSettings — Type
SetRunmodeSettings(; local_n = nothing, local_julia_settings = nothing,
slurm_flags = nothing, slurm_julia_settings = nothing)Use the given settings of the run modes instead of those of the config, for the next worker allocations and the start script; running allocations are not changed. Also possible during a processing run, which keeps its config. Each argument that is nothing keeps the setting in effect. local_n is the number of local workers and must be positive; local_julia_settings, slurm_flags and slurm_julia_settings are vectors of strings, the extra options of the local Julia processes, the flags of the SLURM job and the options of the Julia processes it starts.
JuLeAna.SetCheckDependencies — Type
SetCheckDependencies(check::Bool)Choose whether a processing run rebuilds the dependency graph for the selected steps and waits for their dependencies (true), or runs the selected steps on the current graph without waiting (false).
JuLeAna.SetKwarg — Type
SetKwarg(step, key, value)Override the keyword argument key of the processor step for this session. The processor function must accept key.
JuLeAna.SetStepLevel — Type
SetStepLevel(step, key, value)Override the log level (key = :log_level, value a LogLevel or one of "debug", "info", "warn" and "error") or the plot level (key = :plot_level, value :info or :debug) of the processor step for this session. A level set for all processors with SetProcessorLogLevel or SetProcessorPlotLevel takes precedence.
JuLeAna.ResetKwargs — Type
ResetKwargs(step = nothing, key = nothing)Drop the kwarg and level overrides (see SetKwarg and SetStepLevel) of all processors, of the processor step, or only the override of key (a kwarg, :log_level or :plot_level) of the processor step.
JuLeAna.ResetDependencyGraph — Type
ResetDependencyGraph()Mark the selected steps as unfinished for the selected periods and runs.
JuLeAna.StartProcessing — Type
StartProcessing()Run the selected steps over the selected periods and runs in the background with run_processing, using the config with the session's kwarg overrides. The controller is :running until the run ends; the outcome is :finished, :failed or :stopped. progress shows the progress of the run. When the run ends, its steps are added to the processing history of the data production (see JuLeAna.append_history and JuLeAna.history_path). Throws an ArgumentError when no step is selected or a selected processor does not accept its keyword arguments.
JuLeAna.StopProcessing — Type
StopProcessing(; force::Bool = false)Stop the processing run: no further step or item starts, items running on workers finish, and the run ends as :stopped. With force = true, the run ends without waiting for the running items. They count as not run, go on in the background and keep their workers busy until they end; busy_workers lists them as abandoned. No worker is stopped.
JuLeAna.WriteConfig — Type
WriteConfig(path::AbstractString)
WriteConfig()Write the processing config file of the session to path with the current settings of the session: the kwarg overrides are applied to the processor entries, every processor is enabled exactly when it is selected, and processing.analysis_runs_only is the setting in effect. Without path, the file is processing_config.yaml in the JuLeAna.production_config_dir, which is created if needed. An existing file at the path is first copied to path * ".bak". The file is written as YAML. The written file becomes the config file of the session, which ReloadConfig reads.
JuLeAna.AddWorkers — Type
AddWorkers()Start a worker allocation: run the worker start script of the session in the background (see JuLeAna.add_workers!). The workers join the worker pool when they have started.
JuLeAna.CancelAllocation — Type
CancelAllocation(id::Integer)Cancel the worker allocation id (see JuLeAna.cancel_allocation!): its workers end and items running on them fail.
JuLeAna.CancelAll — Type
CancelAll()Cancel every running worker allocation of the session (see JuLeAna.cancel_all!).
JuLeAna.PostponeIdleCancel — Type
PostponeIdleCancel()Restart the idle time after which the worker allocations are cancelled (see JuLeAna.check_idle!) from now.
JuLeAna.EditStartScript — Type
EditStartScript(text::AbstractString)Replace the worker start script of the session by text for the next allocations (see JuLeAna.edit_start_script!).
JuLeAna.CreateProduction — Type
CreateProduction(spec::ProductionSpec)Create the data production spec describes with JuLeAna.create_production in the background and switch the session to it: cancel all worker allocations (their workers use the data of the previous production), set LEGEND_DATA_CONFIG of this process to the new config.yaml, and load a session with the JuLeAna.install_default_config file of the new production and a new log folder. The lines of the creation are kept for production_log. Possible while no session is loaded or no processing runs; the controller is :loading meanwhile and returns to its previous state if the creation or the load fails.
JuLeAna.SelectProduction — Type
SelectProduction(path::AbstractString)Switch the session to the existing data production in the folder path, which must hold a config.json or config.yaml (config.json if both exist): cancel all worker allocations, set LEGEND_DATA_CONFIG of this process to that file, reset the dataset to default, and load a session with the JuLeAna.install_default_config file of the production and a new log folder. Possible while no session is loaded or no processing runs; the controller is :loading meanwhile and is :idle without a session if the load fails.
JuLeAna.SelectDataset — Type
SelectDataset(dataset::Symbol)Reload the session for the dataset dataset, a name in the metadata datasets/runlists of the data production or :default (every run), and cancel all worker allocations. The dataset is the --dataset option. Possible while no session is loaded or no processing runs; a dataset that is not in the metadata makes the load fail, leaving the controller :idle without a session.
JuLeAna.check_idle! — Method
JuLeAna.check_idle!(ctrl::DataflowController)Cancel the worker allocations of ctrl if they have been idle for the idle_timeout_minutes of the config (see JuLeAna.check_idle!); allocations are never idle while processing runs or is stopping. Return whether they were cancelled. The terminal user interface calls it with every read of the controller state.
Workers
JuLeAna.WorkerManager — Type
WorkerManager(; squeue::Cmd = `squeue`, scancel::Cmd = `scancel`, sinfo::Cmd = `sinfo`,
sacct::Cmd = `sacct`, sstat::Cmd = `sstat`)The worker allocations of this process and the worker start script they run.
The start script is generated from the processing config of the session by JuLeAna.start_script! and saved as startjlworkers.sh in the JuLeAna.production_config_dir. It holds the cluster cookie and the address of this process, so it is generated anew for every config (JuLeAna.set_worker_config!) and can be edited for the rest of the session (JuLeAna.edit_start_script!). JuLeAna.add_workers! runs it as a new allocation, JuLeAna.cancel_allocation! and JuLeAna.cancel_all! end allocations, JuLeAna.allocations lists them.
Running allocations are cancelled when nothing has used them for the idle_timeout_minutes of the config: see JuLeAna.idle_deadline and JuLeAna.check_idle!. JuLeAna.check_joining! warns when started workers wait to join this process.
squeue, scancel, sinfo, sacct and sstat are the SLURM commands, without arguments; SLURM counts as available when the program of squeue is found. JuLeAna.workers_overview collects everything the Workers page shows. All functions may be called from any task.
JuLeAna.WorkerAllocation — Type
WorkerAllocationOne run of the worker start script, started by JuLeAna.add_workers!. Fields:
id::Int: number of the allocation, counted from 1 perJuLeAna.WorkerManager.backend::Symbol::localor:slurm.job_name: the SLURM job name of the allocation's workers,nothingfor:local.started,finished: UTC times;finishedisnothingwhile the script runs.process: theBase.Processof the shell that runs the script, the leader of its own process group.state::Symbol::running,:ended(the script exited with code 0),:failed(any other exit) or:cancelled.exit_code: the exit code of the script,nothingwhile it runs.job_ids::Vector{String}: the SLURM job IDs of the allocation at the last query.worker_errors::Int: the number of workers of the allocation that ended with an exit code other than 0 while it ran, as the linesWorker slot <n>: the worker ended after <s> s with exit code <code>of the script report them (seeJuLeAna.worker_start_script).output::Vector{String}: the last 1000 lines of the standard and error output of the script.
JuLeAna.set_worker_config! — Function
JuLeAna.set_worker_config!(mgr::WorkerManager, config::ProcessingConfig)Use the run mode of config for the next allocations of mgr. The start script is generated anew on its next use; running allocations are not changed.
JuLeAna.worker_start_script — Function
JuLeAna.worker_start_script(cfg::ProcessingConfig)Return the text of the bash script that starts the workers of the run mode of cfg and connects them to this process. A worker that ends is started again in its slot after cfg.worker_restart_delay_seconds, and the script prints Worker slot <n>: the worker ended after <s> s with exit code <code> for its end, so the script runs until it is ended, as long as this process runs: when a worker ends and this process has not touched its heartbeat file for cfg.worker_heartbeat_timeout_minutes (see JuLeAna.add_workers!), the script ends the allocation, the local workers with SIGTERM to its process group, the SLURM job with scancel.
In the local run mode the script starts cfg.local_n workers on the host it runs on.
In the slurm run mode it submits one job with sbatch --wait, the job name julia-<pid of this process>-0 (see JuLeAna.add_workers!) and the SLURM flags of cfg except --cpu-bind, and prints the output of the job until the job ends (it is cancelled or reaches its time limit); ended with SIGTERM, it cancels the job by its name. The output goes to julia-workers-<pid of the script>.out in the log directory of cfg, or in the current directory if cfg has none. The job runs every worker as its own job step on one node, with the CPUs per task and the --cpu-bind and --mem-bind flags of cfg. With cfg.slurm_worker_memory === :shared the steps share the memory of the job; with :fixed each step gets the memory of the job per worker. Throws an ArgumentError for :fixed without --mem or --mem-per-cpu in the flags, and for a --cpu-bind or --mem-bind flag without =.
JuLeAna.start_script! — Function
JuLeAna.start_script!(mgr::WorkerManager)Return (; path, text) of the start script of mgr. The first call after JuLeAna.set_worker_config! generates the script for the run mode of the config with JuLeAna.worker_start_script and writes it to startjlworkers.sh in the JuLeAna.production_config_dir, readable only by the user, keeping an existing file as startjlworkers.sh.bak. Throws an ArgumentError if no config is set.
JuLeAna.edit_start_script! — Function
JuLeAna.edit_start_script!(mgr::WorkerManager, text::AbstractString)Replace the start script of mgr by text and save it to the script file. For the slurm run mode the script must name its SLURM job --job-name=julia-<n>-<m>: each allocation replaces the name by its own, by which it finds and cancels its jobs. Throws an ArgumentError for a SLURM script without such a name.
JuLeAna.add_workers! — Function
JuLeAna.add_workers!(mgr::WorkerManager)Run the start script of mgr (see JuLeAna.start_script!) with bash in the background as a new allocation and return its JuLeAna.WorkerAllocation. The shell leads its own process group, so it keeps running when this process ends unless it is cancelled. Its standard and error output are kept in the allocation. For the slurm run mode, the job name of the script is replaced by julia-<pid of this process>-<n> with a new n. The started workers connect to this process and join its worker pool.
From the first allocation on, this process touches its heartbeat file juleana-<host>-<pid>.alive in the JuLeAna.production_config_dir every minute until it exits; the start scripts end their allocation when the file has not been touched for the worker_heartbeat_timeout_minutes of the config (see JuLeAna.worker_start_script).
JuLeAna.update_job_ids! — Function
JuLeAna.update_job_ids!(mgr::WorkerManager)Query squeue for the jobs of the running SLURM allocations of mgr and store their IDs in the allocations. Does nothing without such allocations.
JuLeAna.update_steps! — Function
JuLeAna.update_steps!(mgr::WorkerManager; at::Real = time())Query sacct and sstat for the job steps of the workers of the SLURM allocations of mgr (not the batch and extern steps) and keep them as JuLeAna.StepInfos by job ID. Queries the jobs of the running allocations (see JuLeAna.update_job_ids!) and, until their steps have ended, the jobs that had running steps at the previous query. Does nothing less than 10 seconds after the previous query (at is the current time()) or without such jobs.
JuLeAna.WorkerAttributes — Type
JuLeAna.WorkerAttributesThe attributes a worker process reports once (see JuLeAna.worker_attributes), a NamedTuple with the fields
host::String: the host name;os_pid::Int: the process ID on the host;threads::Int: the number of Julia threads;memory::Int: the bytes of memory the process may use: the smallest memory limit of its control group and the groups above it (such as its SLURM job step and job), elseSys.total_memory();job,step: the SLURM job and step ID,""outside SLURM;tasks: the number of tasks of the SLURM step (SLURM_STEP_NUM_TASKS),""outside SLURM;cpus: the CPUs per task of the SLURM step (SLURM_CPUS_PER_TASK),""if not set;gpus: the GPUs of the SLURM step (SLURM_STEP_GPUS),""without GPUs.
JuLeAna.StepInfo — Type
JuLeAna.StepInfoA SLURM job step of a worker allocation as sacct and sstat report it (see JuLeAna.update_steps!), a NamedTuple with the fields
id::String:<job ID>.<step ID>;node::String: the node of the step;state::String: the state, such asRUNNING,COMPLETED,CANCELLEDorOUT_OF_MEMORY;exit_code::String:<exit code>:<signal>;elapsed::String: the run time assacctprints it;rss: the resident memory in bytes at the last sample,nothingonce the step ended;max_rss: the peak of the resident memory in bytes,nothingif not reported. SLURM samples the memory every few seconds, so a short peak can be missing;cpu_load: the CPUs the step used over the last minute, its CPU time since the oldest query of the last minute divided by the time since then;nothingat its first query and once the step ended;cpu_mean: the CPUs the step used over its whole run, its CPU time divided by its run time;nothingonce the step ended.
The CPU time is the AveCPU of sstat, the CPU time of the step's task, which is all of the step's CPU time for a worker step with its one task.
JuLeAna.worker_attributes — Function
JuLeAna.worker_attributes(pid::Integer)Return the JuLeAna.WorkerAttributes of the worker process pid. The first call asks the worker, later calls return the same attributes without asking. The worker must answer, so the first call must come while it is not busy.
JuLeAna.kill_worker — Function
JuLeAna.kill_worker(pid::Integer; scancel::Cmd = `scancel`)End the process of the worker pid at once with SIGKILL, also while it computes and does not answer: a worker that is the only task of a SLURM job step (other than the step of this process) with scancel --signal=KILL <job>.<step>, a worker on this host with kill -9. Its JuLeAna.worker_attributes must have been asked for before. Distributed then removes the worker, and the calls running on it throw a ProcessExitedException.
Throws an ArgumentError for the main process, for a worker whose attributes are not known, and for a worker on another host that is not the only task of its job step.
JuLeAna.worker_job_ended — Function
JuLeAna.worker_job_ended(pid::Integer; squeue::Cmd = `squeue`)Whether the SLURM job of the worker pid no longer runs, so that a worker that ended ended with its allocation, at its time limit or cancelled, and not through what it ran. squeue reports the state of the job; any state but RUNNING counts as ended. false for a worker whose JuLeAna.worker_attributes are not known or that is not in a SLURM job, and when squeue reports no state or cannot run.
JuLeAna.cancel_allocation! — Function
JuLeAna.cancel_allocation!(mgr::WorkerManager, id::Integer)End the allocation id of mgr if it is running: for a SLURM allocation, scancel the jobs that squeue lists for this user under the allocation's job name (and no others), then send SIGTERM to the process group of its script, which ends the local workers or sbatch. The workers of the allocation leave the worker pool; items running on them fail. Throws an ArgumentError for an unknown id.
JuLeAna.cancel_all! — Function
JuLeAna.cancel_all!(mgr::WorkerManager)Cancel every running allocation of mgr with JuLeAna.cancel_allocation!.
JuLeAna.allocations — Function
JuLeAna.allocations(mgr::WorkerManager; output::Bool = true)Return a copy of the allocations of mgr as NamedTuples with the fields of JuLeAna.WorkerAllocation except process, and without output unless output: a Vector{JuLeAna.AllocationOutputInfo}, or a Vector{JuLeAna.AllocationInfo} without output.
JuLeAna.mark_activity! — Function
JuLeAna.mark_activity!(mgr::WorkerManager, time::DateTime = now(UTC))Record that the workers of mgr were used at time (UTC), such as by the end of a processing run or a postponement: the idle time of JuLeAna.idle_deadline counts from the latest such time.
JuLeAna.idle_deadline — Function
JuLeAna.idle_deadline(mgr::WorkerManager; processing::Bool)Return the UTC time at which JuLeAna.check_idle! cancels the running allocations of mgr: the last activity (see JuLeAna.mark_activity!; a new allocation counts too) plus the idle_timeout_minutes of the config. Return nothing while processing, if no allocation runs, without a config, or if the timeout is 0.
JuLeAna.check_idle! — Method
JuLeAna.check_idle!(mgr::WorkerManager; processing::Bool, time::DateTime = now(UTC))Cancel all running allocations of mgr with JuLeAna.cancel_all! if time (UTC) has reached the JuLeAna.idle_deadline. Return whether they were cancelled.
JuLeAna.joining_workers — Function
JuLeAna.joining_workers(mgr::WorkerManager)Return the number of started workers that have connected to this process and wait for the cluster manager of ParallelProcessingTools to add them: the manager adds the waiting workers in batches, one batch at a time, and the workers of the batch it is adding are not counted. 0 before the first allocation of mgr.
JuLeAna.check_joining! — Function
JuLeAna.check_joining!(mgr::WorkerManager; at::Real = time(), joining::Integer = JuLeAna.joining_workers(mgr))Warn once when workers have waited to join this process (joining, see JuLeAna.joining_workers) at every call for 60 seconds (at is the current time()): a worker that is not added within its JULIA_WORKER_TIMEOUT ends. After such a warning, log when no worker waits anymore. Return joining. The terminal user interface calls it with every read of the controller state.
JuLeAna.workers_overview — Function
JuLeAna.workers_overview(mgr::WorkerManager)Collect what the Workers page shows and return a JuLeAna.WorkersOverview (; allocations, script, workers, joining, steps, slurm):
allocations:JuLeAna.allocationswith their output, afterJuLeAna.update_job_ids!;script: the(; path, text)ofJuLeAna.start_script!, or the message of the error that kept it from being generated;workers: one(; pid, attributes)per worker process with itsJuLeAna.WorkerAttributes,nothinguntil they are known andmissingif the worker did not answer; the attributes of a new worker are asked for in the background;joining: theJuLeAna.joining_workers;steps: theJuLeAna.StepInfos of the SLURM jobs, afterJuLeAna.update_steps!, by job and step ID;slurm:nothingwithout SLURM, else(; queue, idle), the output ofsqueue -u <user>and ofsinfo -t idle.
Runs SLURM commands, so it can take a while; call it in the background.