Terminal interface

mutable struct JuLeAnaTUI <: Tachikoma.Model

State of the terminal user interface opened by JuLeAna.run_tui.

JuLeAnaTUI(ctrl::DataflowController, messages::MessageBuffer = MessageBuffer()) shows the dataflow of ctrl. Controller commands run one after another in a background task, in the order they were sent, and state reads run in background tasks of the interface's Tachikoma.TaskQueue, so the interface never waits for the controller; the latest state read from it is kept in the fields snap, progress, busy and step_msgs (the messages of the step open on the Progress page, with the step's key).

A left click on a page's tab opens the page; other mouse events go to the visible page.

Keys: the visible page gets each key first (see JuLeAna.setup_key! for the Setup page, JuLeAna.graph_key! for the Graph page, JuLeAna.progress_key! for the Progress page and JuLeAna.log_key! for the Log page, JuLeAna.reports_key! for the Reports page and JuLeAna.history_key! for the History page and JuLeAna.workers_key! for the Workers page); otherwise 1 to 7 switch pages and q or Ctrl-C quits. While processing runs, quitting asks first: s stops processing after the running items and quits when the run has ended, q quits at once, ending the run and the workers connected to this process, and c or Escape keeps the interface open.

JuLeAnaTUI(ctrl, messages; splash_seconds = 1.5) shows a startup screen with the logo, the version and the link to the documentation in place of the page when the interface starts, until the session has loaded (or loading failed) and splash_seconds have passed; any key ends it and is not passed on. ? opens a panel that lists the keys of the visible page; Escape, ? or q closes it. The Setup page's filter and the kwargs editor receive ? as text.

N on the Setup page opens the JuLeAna.ProductionForm for a new data production. The form replaces the visible page and takes every key (see JuLeAna.production_key!). It is also open when the interface starts without a session: when the parsed options have no config or new_production is set, start! does not load a session.

The field timing is the JuLeAna.InterfaceTiming that records the times of the view and Tachikoma.update! calls; once a session is loaded it appends a row every 10 seconds to interface_timing.tsv in the log folder.

JuLeAna.TUI_THEMES

The themes of the interface, in the order in which T cycles through them: night-owl (the default), juleana and the themes of Tachikoma.

JuLeAna.tui_theme(name::AbstractString)

The theme of JuLeAna.TUI_THEMES called name. Throws an ArgumentError that lists the names for an unknown one.

JuLeAna.theme_preference_path()

The file that keeps the name of the theme chosen with T: prefs/JuLeAna/theme in the first Julia depot.

JuLeAna.load_theme_name(path::AbstractString = JuLeAna.theme_preference_path())

The name of the theme saved in path, or night-owl when nothing is saved. A saved name that is not a theme of JuLeAna.TUI_THEMES is logged as a warning and replaced by night-owl.

JuLeAna.next_theme!(path::AbstractString = JuLeAna.theme_preference_path())

Switch the interface to the theme after the active one in JuLeAna.TUI_THEMES and save its name in path for the next start. Returns the new theme. A failed save is logged as a warning; the theme is switched anyway.

JuLeAna.send_command!(m::JuLeAnaTUI, cmd::ControllerCommand)

Queue cmd for the controller of m; the commands are dispatched one after another in a background task. A rejected command shows its error in the status panel.

MessageBuffer(capacity::Integer = 1000; worker_capacity::Integer = 200)

Logger that keeps the last capacity lines of log messages and of captured terminal output as text, for the terminal user interface. Each line has the log level of its message and the number of its record: the lines of one log message share a record, each line of captured output is a record of its own. Output lines containing Warning: or Error: get the level Warn or Error, other output lines Info. Records below Info and progress bar records of ProgressLogging are dropped. A message that is not a string but can be shown as Markdown, such as a processor's report, is kept as that Markdown if it has at most 20 lines, else as one line with its section heading, processor status and length. A log message of a processing step (inside the step's scope set by run_processing) is also kept with the step in its ProgressStore (see step_messages). The newest warning or error, from a log record or from an output line containing Warning: or Error:, is kept as the alert of the buffer.

Output lines that Distributed relays from a worker (From worker <pid>:) are kept apart, the last worker_capacity lines per worker without the prefix, so that the output of many workers does not push out the other lines. Of the progress lines of @debug_progress, only the newest one of each description is kept. Relayed lines containing Warning: or Error: are also kept with the other lines.

JuLeAna.message_lines returns the kept lines as text, JuLeAna.message_entries with their levels and records, and JuLeAna.worker_output the lines of a worker.

JuLeAna.append_output!(buf::MessageBuffer, stream::Symbol, text::AbstractString)

Add text written to the captured output stream (:stdout or :stderr) to buf, line by line; an unfinished last line waits for the rest of its text. A line relayed from a worker goes to the lines of that worker (see JuLeAna.worker_output), and only with Warning: or Error: also to the other lines.

JuLeAna.message_lines(buf::MessageBuffer)

Return a copy of the lines kept in buf, oldest first.

JuLeAna.message_entries(buf::MessageBuffer)

Return a copy of the entries kept in buf, oldest first: named tuples with the fields level (a Logging.LogLevel), record (the number of the log message or output line the line belongs to) and text (the line as shown).

JuLeAna.worker_output(buf::MessageBuffer, pid::Integer)

Return a copy of the output lines kept in buf for the worker with process ID pid, oldest first, as entries like those of JuLeAna.message_entries with the text without the From worker <pid>: prefix.

Checklist(entries; title = "", checked = (), focused = true, tag_style = tag -> nothing)

Terminal widget that displays a list of items with checkboxes and lets the user select any number of them.

entries is either a vector of JuLeAna.ChecklistItem (flat list) or a vector of JuLeAna.ChecklistGroup (two-level list). Item keys must be unique; checked is an iterable of the keys of the initially selected items and must only contain keys of items of the checklist. Only items are selected, never groups. The cursor row is highlighted only while focused is true. tag_style(tag::String) returns the Tachikoma.Style of a tag, or nothing for the dimmed default.

Selected boxes are drawn in the success color and group labels in bold primary color. A group row shows [x] if all of its visible items are selected, [~] if some are, and [ ] if none are. The title shows the number of selected items out of all items.

Keys

Tachikoma.handle_key!(w, evt) returns true if the widget used the key and false otherwise.

In normal mode:

• Up, Down, PageUp, PageDown, Home, End move the cursor.
• Space toggles the item under the cursor. On a group row it toggles all visible items of the group: if all are selected they are deselected, otherwise all are selected.
• a selects and n deselects all visible items.
• Left collapses the group under the cursor (the cursor moves to the group row if it is on one of its items), Right expands it, and Enter on a group row toggles it.
• / starts editing the filter.
• Escape clears a non-empty filter.
• Other keys are not used.

While editing the filter, every key is used: printable characters are appended, Backspace deletes the last character, Enter keeps the filter and returns to normal mode, and Escape clears the filter and returns to normal mode.

The filter keeps the items whose label or any tag contains the filter text, ignoring case. A group is shown with all its items if its own label or a tag matches, and with only its matching items if just some of its items match. Visible items are the items shown under the filter, whether or not their group is collapsed.

Mouse

Tachikoma.handle_mouse!(w, evt) handles clicks and the scroll wheel using the area of the last Tachikoma.render(::JuLeAna.Checklist, ...).

ChecklistItem(key, label::AbstractString; tags = String[])

Selectable entry of a JuLeAna.Checklist. key identifies the item and must be unique among the items of a checklist, label is the displayed text and tags are short strings displayed dimmed at the right end of the row. Labels and tags are searched by the filter of the checklist.

ChecklistGroup(key, label::AbstractString, items::AbstractVector{ChecklistItem};
               tags = String[], expanded = true)

Group of items of a two-level JuLeAna.Checklist. A group is never selected itself: toggling it toggles its visible items. expanded states whether the items are displayed below the group; the checklist changes it when the group is collapsed or expanded.

Tachikoma.handle_key!(w::JuLeAna.Checklist, evt::Tachikoma.KeyEvent)::Bool

Apply the key event evt to w as described for JuLeAna.Checklist. Return true if the widget used the key and false otherwise.

Tachikoma.handle_mouse!(w::JuLeAna.Checklist, evt::Tachikoma.MouseEvent)::Bool

Apply the mouse event evt to w. A left button press on a row moves the cursor to the row and toggles it like Space does. The scroll wheel moves the cursor by one row while the pointer is inside the area of the last render. Return true if the event was used.

Tachikoma.render(w::JuLeAna.Checklist, rect::Tachikoma.Rect, buf::Tachikoma.Buffer)

Draw w into rect of buf. The view scrolls to keep the cursor row visible. Rows never extend beyond rect: long labels are shortened, and the tags of a row are dropped if they would leave less than a short label.

JuLeAna.checked(w::Checklist)

Keys of the selected items of w, in the order of the unfiltered list.

JuLeAna.set_checked!(w::Checklist, keys)

Replace the selection of w with the items whose keys are in keys. Throws an ArgumentError naming the keys that belong to no item; the selection is unchanged then.

JuLeAna.set_checked!(f::Function, w::Checklist)

Replace the selection of w with the items for which f(item::ChecklistItem) is true.

JuLeAna.current(w::Checklist)

The JuLeAna.ChecklistItem or JuLeAna.ChecklistGroup under the cursor of w, or nothing if no rows are visible.

SetupPage()

State of the Setup page of the terminal user interface: a JuLeAna.Checklist of periods with their runs, one of the run-level and one of the partition-level processors, the options pane, the pane with the production root, the data production and the dataset, and the runinfo of the period or run under the cursor of the periods. The checklists are built from the controller snapshots by JuLeAna.sync_setup!; keys are handled by JuLeAna.setup_key!.

focus is the focused pane: 1 periods, 2 run processors, 3 partition processors, 4 options, 5 production pane, where source is the chosen line (1 root, 2 production, 3 dataset). editor is the open JuLeAna.KwargsEditor or JuLeAna.RunmodeEditor, confirm is :write while writing the config waits for confirmation, and picker is the open JuLeAna.SetupPicker. root is the folder the productions are listed under, or nothing for the parent folder of the current data production (see JuLeAna.production_root); root_edit is the text of the root while it is edited in place, nothing otherwise, and root_hint a message below the lines of the pane (the completion candidates or the error of the root) with whether it is an error. choices holds the names the pickers list, read by JuLeAna.read_choices: nothing before they are read, the message of the read error, or a NamedTuple with productions and datasets.

SetupPicker(kind::Symbol, items::Vector{String}, cursor::Integer)

Open list of names on the Setup page: kind is :production or :dataset, cursor the index of the chosen item in items, and top the index of the first item shown at the last render.

JuLeAna.read_choices(ctrl::DataflowController, root)

Read the names the pickers of the Setup page list, and return them as a NamedTuple with productions, the data productions under the folder root (see JuLeAna.find_productions), and datasets, "default" and the names in the metadata datasets/runlists of the loaded session. Throws if root is empty or the run lists cannot be read.

JuLeAna.find_productions(root)

Return the sorted paths, relative to the folder root, of the folders under root that hold a config.json or config.yaml; "." is root itself. The search is breadth-first with one readdir per folder, goes at most 4 levels below root, skips hidden folders and symbolic links, and does not descend into a folder that holds a config file. Folders that cannot be read are skipped.

JuLeAna.production_root(page::SetupPage)

Return the folder the productions of the Setup page are listed under: page.root if it was set, else the parent folder of the current data production, or "" if none is set.

JuLeAna.root_key!(page::SetupPage, evt::Tachikoma.KeyEvent)

Handle the key evt while the production root of page is edited in page.root_edit. Printable characters are appended and Backspace removes the last one. Tab completes the text with JuLeAna.complete_path: it extends the text by the common prefix of the folders it matches and, if there are several, shows up to 8 of them in page.root_hint. Enter sets the root to the text, with ~ expanded, if that is an existing absolute folder, which makes the productions be read again, and shows an error and keeps editing otherwise. Escape ends the edit and keeps the root. Every other key is ignored.

JuLeAna.sync_setup!(page::SetupPage, snap; pending::Bool = false)

Bring page in line with the controller snapshot snap. The checklists are rebuilt when the available periods, runs or processors change, keeping their cursor and filter otherwise. Unless commands of the page are pending, the checked items are set to the selection of snap.

JuLeAna.setup_key!(page::SetupPage, evt::Tachikoma.KeyEvent, snap, send)

Handle the key evt on the Setup page and return whether it was used. snap is the latest controller snapshot (or nothing) and send(cmd) sends a controller command.

An open kwargs or run mode editor gets every key (see JuLeAna.kwargs_key! and JuLeAna.runmode_key!); while writing the config waits for confirmation, y sends WriteConfig for the config folder of the data production and any other key cancels. An open picker gets every key: Up and Down move the cursor, Enter sends SelectProduction or SelectDataset for the chosen name unless it is the current one, and Escape closes the picker. While the production root is edited every key goes to its text (see JuLeAna.root_key!). Otherwise the focused checklist gets the key first; a change of its selection sends SelectRuns or SelectSteps. Then: Tab and Shift-Tab move between the panes; in a processor pane c and p select the cal and phy processors of the pane, d adds the steps the selected steps wait for and e opens the kwargs editor for the processor under the cursor; in the options pane Up and Down choose a row, Space or Enter cycles its value and e on the Run mode row opens the editor of the settings of the run modes; in the production pane Up and Down choose the root, the production or the dataset, Space or Enter edits the root in place or opens the picker of the production or dataset once the names are read; s starts processing, x stops it after the running items, X stops it without waiting for them, r reloads the config, g rebuilds the dependency graph, W starts a worker allocation with AddWorkers and w asks to write the current settings to config/processing_config.yaml of the data production. N sets page.new_production, which asks the interface to open the new-production form (see JuLeAna.ProductionForm); the checklists use n to uncheck all items.

JuLeAna.setup_mouse!(page::SetupPage, evt::Tachikoma.MouseEvent, snap, send)

Pass the mouse event evt to the checklist under the pointer, focus that pane, and send the new selection if it changed. A left click in the options pane focuses it; on an option row it also chooses the row and sends its next value. A left click in the production pane focuses it and edits the root or opens the picker of the line. While the root is edited, a click is used without effect. While a picker is open, a left click on an item chooses it and every other event is used without effect. Return whether the event was used.

JuLeAna.render_setup(page::SetupPage, snap, area::Tachikoma.Rect, buf::Tachikoma.Buffer)

Draw the Setup page into area: periods and runs on the left, the run and partition processors in the middle, and on the right the options with a summary of what a start would process and the largest logo that fits below it (centered at the bottom of the pane), the data production and dataset, and the runinfo of the period or run under the cursor of the periods (see JuLeAna.dataset_lines); the production pane shows JuLeAna.production_root, the data production and the dataset. The open picker is drawn over the page.

JuLeAna.dataset_lines(page::SetupPage)

Return the title and the lines, as vectors of (text, style) spans, of the box with the runinfo of the period or run under the cursor of the periods of page. A run shows one line per category with file keys: the number of file keys, the livetime, and no analysis if it is not an analysis run of the category. A period shows the totals of its runs per category (see JuLeAna.period_summary) with the number of runs. The first line is the earliest start time. The box has as many lines as the period with the most categories needs.

ProgressPage()

State of the Progress page of the terminal user interface: the row under the cursor, the first displayed row, whether the cursor follows the running step, and which groups the user folded or unfolded. The page lists the steps of the current or last processing run grouped by period and run, with the busy workers below. Keys are handled by JuLeAna.progress_key! and JuLeAna.progress_mouse!, the page is drawn by JuLeAna.render_progress.

A row is identified by its key: (period, run) for a group and (period, run, step) for a step, so that the cursor stays on its row when the list changes. expanded holds the group keys the user folded (false) or unfolded (true); the other groups are unfolded if a step runs, failed or stopped, or has items that were not run.

open is the key of the step whose messages pane is shown (nothing if closed). The pane lists the messages that step_messages returns for that step; while pane_follow is true it shows the newest lines, else it starts at line pane_top. While the pane is open, the cursor moving to another step row switches the pane to that step.

JuLeAna.progress_key!(page::ProgressPage, evt::Tachikoma.KeyEvent, prog)::Bool

Apply the key event evt to page for the progress prog (as returned by progress, or nothing). Return true if the key was used and false otherwise.

• Up, Down, PageUp, PageDown, Home and End move the cursor and turn following off.
• Enter on a group row toggles the group. Left folds the group of the cursor row and Right unfolds it. Folding a group moves the cursor to the group row. These keys turn following off.
• Enter on a step row opens the messages pane for the step and turns following off; Enter on the open step's row closes the pane. Escape closes the pane.
• While the pane is open, PageUp and PageDown scroll it instead of moving the cursor and End shows the newest line. Moving the cursor to another step row switches the pane to that step.
• f turns following on: the cursor goes to the first running step and the view scrolls to keep it visible. An open pane follows the cursor's step.
• Other keys are not used.

JuLeAna.progress_mouse!(page::ProgressPage, evt::Tachikoma.MouseEvent, prog)::Bool

Apply the mouse event evt to page: the scroll wheel moves the cursor by one row while the pointer is inside the steps block of the last render, and a left click on a row moves the cursor to it, folds or unfolds a group row and opens the messages pane for a step row. Both turn following off. Return true if the event was used.

JuLeAna.render_progress(page::ProgressPage, prog, busy, msgs, area::Tachikoma.Rect, buf::Tachikoma.Buffer)

Draw the Progress page into area of buf from the progress prog (as returned by progress, or nothing), the busy workers busy (see busy_workers) and msgs, a named tuple with the step key and the messages that step_messages returned for it. The page shows the overall progress and the step counts, a block with the steps grouped by period and run, and a block with the busy workers. While page follows, the cursor moves to the first running step. The view scrolls to keep the cursor row visible.

While page.open is a step key, the lower half of the space between the overview and the busy workers (if at least 5 rows) shows the messages pane of that step: the error of a failed step, then the messages oldest first with local time and level, colored by level. It shows the newest lines while page.pane_follow is true. msgs is used only if its key is page.open, else the pane says that the messages are being read.

LogPage()

State of the Log page of the terminal user interface: whether the view follows the newest line, the index of the first displayed line, the level filter and the text filter. The page lists the entries of a JuLeAna.MessageBuffer (see JuLeAna.message_entries). Keys are handled by JuLeAna.log_key! and JuLeAna.log_mouse!, the page is drawn by JuLeAna.render_log.

The level filter shows the records of at least the level min_level. The text filter filter shows the records with a line that contains it, ignoring case. A record is shown with all of its lines. top indexes the shown lines and is used while follow is false.

JuLeAna.log_key!(page::LogPage, evt::Tachikoma.KeyEvent, entries)::Bool

Apply the key event evt to page for the message entries entries (see JuLeAna.message_entries). Return true if the key was used and false otherwise.

• Up, Down, PageUp, PageDown and Home scroll and turn following off.
• End shows the newest line and turns following on. f turns following on.
• l cycles the level filter: all records, warnings and errors, errors.
• / starts editing the text filter.
• Other keys are not used.

While editing the filter, every key is used: printable characters are appended, Backspace deletes the last character, Enter keeps the filter and returns to normal mode, and Escape clears the filter and returns to normal mode.

JuLeAna.log_mouse!(page::LogPage, evt::Tachikoma.MouseEvent, entries)::Bool

Apply the mouse event evt to page: the scroll wheel scrolls by three lines and turns following off while the pointer is inside the block of the last render. Return true if the event was used.

JuLeAna.render_log(page::LogPage, entries, area::Tachikoma.Rect, buf::Tachikoma.Buffer;
    capacity = typemax(Int), log_path = nothing)

Draw the Log page into area of buf from the message entries entries (see JuLeAna.message_entries): a block that lists the lines passing the filters of page, colored by level, and a line with the number of lines shown. The title shows the level filter, the text filter and whether the view follows the newest line. While page follows, the last lines are displayed. Long lines are cut at the edge of the block.

If entries holds capacity lines, older lines were dropped, and the bottom line points to the log file of this process in the directory log_path (if not nothing).

GraphPage()

State of the Graph page of the terminal user interface: the processor under the cursor, the first displayed row, whether unselected processors are shown, and the ProcessorGraph of the processors of the latest snapshot. The page draws the run-level processors in a left lane and the partition-level processors in a right lane, one row per processor in the order topological_order gives. Keys are handled by JuLeAna.graph_key! and JuLeAna.graph_mouse!, the page is drawn by JuLeAna.render_graph.

The state icon of a processor is that of the last processing run, or, while disk is true, that of its reports and processing history on disk.

The cursor is identified by the processor name, so that it stays on its processor when the rows change. The graph is rebuilt only when the names, levels, ranks or dependencies of the processors change.

JuLeAna.graph_key!(page::GraphPage, evt::Tachikoma.KeyEvent, snap, send)::Bool

Apply the key event evt to page for the controller snapshot snap (or nothing); send(cmd) sends a controller command. Return true if the key was used and false otherwise.

• Up, Down, PageUp, PageDown, Home and End move the cursor over the processors.
• Space adds the processor under the cursor to the selected steps, or removes it, and sends the new selection as SelectSteps.
• d adds the processors that the selected steps require, as the Setup page does.
• h hides or shows the processors that are not selected.
• v switches the state icons between the last run and the state on disk.
• Other keys and key releases are not used.

JuLeAna.graph_mouse!(page::GraphPage, evt::Tachikoma.MouseEvent, snap)::Bool

Apply the mouse event evt to page: the scroll wheel moves the cursor by one row while the pointer is inside the block of the last render, and a left click on a processor row moves the cursor to it. Return true if the event was used.

JuLeAna.render_graph(page::GraphPage, snap, prog, area::Tachikoma.Rect, buf::Tachikoma.Buffer; grid = nothing)

Draw the Graph page into area of buf from the controller snapshot snap, the progress prog (as returned by progress, or nothing) and the grid of the reports and the processing history grid (as returned by JuLeAna.history_grid, or nothing while it is not read).

Run-level processors are in the left lane and partition-level processors in the right lane, one row per processor in topological order, labeled with its display_name and category. A processor shows [x] if it is a selected step, else [ ], and the icon of its state in the last run: running, failed, done, or stopped. While page.disk is true the icon is instead the JuLeAna.disk_state of the processor over the selected periods and runs of snap: ✗ failed, – missing, ! warning or ✓ ok, and the last line also shows the counts of the processor under the cursor. Unselected processors are dim, and are left out while page.show_unselected is false. Each declared dependency is an edge from the required processor down to the one that waits for it, in the space between the lanes, also between processors of the same lane; every processor with edges to later processors has a vertical track there, and tracks with overlapping spans have different columns.

The processor under the cursor is bold; the processors it requires are yellow and the processors that require it are purple, as are the edges between them. The view scrolls to keep the cursor row visible, and the last line shows the number of required and requiring processors and the tags of the processor under the cursor. Names are cut with … if the area is too narrow.

ReportsPage()

State of the Reports page of the terminal user interface: the reports grouped by processor, the processing history, the row under the cursor, which groups the user folded or unfolded, the open report and the requests for reading the file system. The page lists all reports on disk (source is :all, report_overview), which it starts with, or the reports of the steps of the current or last processing run (:run, rows of report_index), and the failures of the processor under the cursor. Keys are handled by JuLeAna.reports_key! and JuLeAna.reports_mouse!, the page is drawn by JuLeAna.render_reports.

The page never reads the file system. It queues requests (see JuLeAna.take_requests!) that the interface carries out in the background and answers with JuLeAna.reports_result!. A row is identified by its key, (group,) for a group and (group, period, run) for a report, where group is (name, level, category), so that the cursor stays on its row when the list is read anew. expanded holds the group keys the user folded (false) or unfolded (true); the other groups are unfolded if one of their reports failed.

JuLeAna.reports_key!(page::ReportsPage, evt::Tachikoma.KeyEvent)::Bool

Apply the key event evt to page. Return true if the key was used and false otherwise.

While a report is open, the viewer uses every key: Up, Down, PageUp, PageDown, Home and End scroll it, Escape and q close it, and other keys are ignored.

Otherwise:

• Up, Down, PageUp, PageDown, Home and End move the cursor over the groups and reports, or scroll the failures while the failures have the focus. Tab and Shift-Tab switch the focus.
• Enter on a group row toggles the group, on a report row opens the report. Left folds the group of the cursor row and moves the cursor to it, Right unfolds it.
• a switches between all reports on disk and the reports of the steps of the current or last processing run, and reads them.
• r reads the reports anew.
• Other keys, such as 1 to 6 and q, are not used.

JuLeAna.reports_mouse!(page::ReportsPage, evt::Tachikoma.MouseEvent)::Bool

Apply the mouse event evt to page. While a report is open, the viewer gets the event (the scroll wheel and the scroll bar). Otherwise the scroll wheel moves the cursor by one row over the list of reports and scrolls the failures over the failures block, and a left click on a row moves the cursor to it and folds or unfolds a group row. Return true if the event was used.

JuLeAna.render_reports(page::ReportsPage, area::Tachikoma.Rect, buf::Tachikoma.Buffer)

Draw the Reports page into area of buf. The left block, about 45% of the width, lists the reports grouped by processor: a group row shows the fold arrow, the name, the level, the category and the number of reports that are ok, have warnings, failed and (for the current run) are missing; the rows of an unfolded group show the period and run, the state and the processing date in UTC. The right block shows the failures of the processor under the cursor, per detector the runs it failed in and its distinct errors. The block with the focus has a heavy border. While a read is under way the page says Reading reports…, and the error of a failed read replaces the list.

While a report is open the viewer, a Tachikoma.MarkdownPane titled with the file name, fills the whole area; a missing file shows No report file: <path>.

JuLeAna.reports_result!(page::ReportsPage, answer)

Take the answer to a request of page as JuLeAna.read_reports returns it. The answer to a read of the reports replaces the groups of page, or sets its error, unless the page has read the reports anew since; likewise an answer with failures is dropped if the groups changed. The viewer of the report file opens if the report is still the open one, else it is dropped.

JuLeAna.reports_visit!(page::ReportsPage, snap)

Queue a read of the reports if the page is visible for the first time or the controller snapshot snap (or nothing) shows a loaded session or a processing run that started or finished since the shown reports were read.

JuLeAna.take_requests!(page::ReportsPage)

Return the requests of page for reading the file system and clear them. A request is a named tuple with kind :index (read the reports of source), :failures (read the failures of the reports reports of the processor group) or :open (read the report file path). JuLeAna.read_reports carries a request out and JuLeAna.reports_result! takes its answer.

JuLeAna.take_requests!(page::HistoryPage)

Return the requests of page for reading the file system and clear them. A request is a named tuple with kind :grid (read the grid) or :open (read the report file path). JuLeAna.read_history_page carries a request out and JuLeAna.history_result! takes its answer.

JuLeAna.take_requests!(page::WorkersPage)

Return the requests of page for reading the overview and clear them. A request is a named tuple with kind :overview. JuLeAna.read_workers_page carries a request out and JuLeAna.workers_result! takes its answer.

JuLeAna.read_reports(ctrl::DataflowController, request)

Carry out a request of a JuLeAna.ReportsPage (see JuLeAna.take_requests!) and return (; request, value), or (; request, error) with the message of the exception if the read failed. The value of an :index request is the rows of report_index(ctrl) or the entries of report_overview(ctrl) (see report_index), that of a :failures request the rows (; process, period, run, failures) of its reports, and that of an :open request the viewer of the file's text, or nothing if there is no such file. The answer to an :index request also has the field history, the path of the processing history file (JuLeAna.history_path) or nothing if the file does not exist. The viewer is built here, so that converting the text does not hold up the interface.

HistoryPage()

State of the History page of the terminal user interface: the grid of the processing state of every processor (rows) in every run (columns, grouped by period) as JuLeAna.history_grid returns it, the cursor cell, the scroll position, the open report and the requests for reading the file system.

The columns are the periods and runs that the session has selected, or, with all_runs (toggled by a), all periods and runs of the data production. Everything the drawing needs about the grid (column headers, row labels, the text and color of each cell) is computed when the grid, all_runs or the selection changes, not when the page is drawn. Keys are handled by JuLeAna.history_key! and JuLeAna.history_mouse!, the page is drawn by JuLeAna.render_history.

The page never reads the file system. It queues requests (see JuLeAna.take_requests!) that the interface carries out in the background and answers with JuLeAna.history_result!. The cursor is a row and a run column; on a partition-level row the cell is that of the period of the column.

JuLeAna.history_key!(page::HistoryPage, evt::Tachikoma.KeyEvent)::Bool

Apply the key event evt to page. Return true if the key was used and false otherwise.

While a report is open, the viewer uses every key: Up, Down, PageUp, PageDown, Home and End scroll it, Escape and q close it, and other keys are ignored.

Otherwise:

• Up and Down move the cursor by one row, PageUp and PageDown by a page of rows.
• Left and Right move the cursor by one run, on a partition-level row by one period. Home and End move it to the first and the last run.
• Enter on a cell that comes from a report opens the report.
• r reads the grid anew.
• a switches between the selected periods and runs of the session (the default) and all periods and runs of the data production.
• Other keys, such as 1 to 7 and q, are not used.

JuLeAna.history_mouse!(page::HistoryPage, evt::Tachikoma.MouseEvent)::Bool

Apply the mouse event evt to page. While a report is open, the viewer gets the event (the scroll wheel and the scroll bar). Otherwise a left click on a cell moves the cursor to it, the scroll wheel moves the cursor by one row, and sideways scrolling (a horizontal wheel or touchpad, or the wheel with Shift held) moves it by one run column. Return true if the event was used.

JuLeAna.render_history(page::HistoryPage, area::Tachikoma.Rect, buf::Tachikoma.Buffer)

Draw the History page into area of buf. The grid has a row per processor, named after the processor and its category (p marks a partition-level processor), and a column per run with the period over its runs. A run cell shows an icon: for a report ✓ ok, ! warning or ✗ failed; for a step of the processing history that left no newer report ■ failed, s stopped or c cancelled; – if the processor never processed the run. A partition-level cell spans the runs of its period, as [ ✓ ]. The cell under the cursor is highlighted, the grid scrolls to keep it visible while the names stay in place, and the last two lines show the legend and the description of the cell under the cursor: processor, period and run, state, processing date in UTC and source. While no session is loaded or a read is under way the page says so, and the error of a failed read replaces the grid. The title says whether the columns are the selected runs or all runs (a).

While a report is open the viewer, a Tachikoma.MarkdownPane titled with the file name, fills the whole area.

JuLeAna.history_result!(page::HistoryPage, answer)

Take the answer to a request of page as JuLeAna.read_history_page returns it. The answer to a read of the grid replaces the grid of page, or sets its error, unless the page has read the grid anew since. The viewer of the report file opens if the report is still the open one, else it is dropped.

JuLeAna.history_visit!(page::HistoryPage, snap)

Queue a read of the grid if the controller snapshot snap (or nothing) shows a loaded session or a processing run that started or finished since the shown grid was read. While no session is loaded the page drops its grid. A change of the selected periods and runs of snap changes the shown columns without a new read.

JuLeAna.history_selection(page::HistoryPage)

Return the cell under the cursor of page as (; row, period, run, label, cell), where row is the row of the grid, run is nothing on a partition-level row, label is p02-r008 or p02 and cell is the entry of the grid or nothing if the processor never processed it. Return nothing if the page has no cells.

JuLeAna.read_history_page(ctrl::DataflowController, request)

Carry out a request of a JuLeAna.HistoryPage (see JuLeAna.take_requests!) and return (; request, value), or (; request, error) with the message of the exception if the read failed. The value of a :grid request is history_grid(ctrl) (see JuLeAna.history_grid), that of an :open request the viewer of the report file's text, or nothing if there is no such file. The viewer is built here, so that converting the text does not hold up the interface.

WorkersPage()

State of the Workers page of the terminal user interface: the latest overview of the worker allocations, the workers and SLURM as JuLeAna.workers_overview returns it, the focused pane, the selected allocation and worker, the scroll positions of the output panes, a pending confirmation, the start script editor, the form of the SLURM options (see JuLeAna.SlurmForm) and the requests for reading the overview. Keys are handled by JuLeAna.workers_key! and JuLeAna.workers_mouse!, the page is drawn by JuLeAna.render_workers.

The page never runs the commands of the overview. It queues requests (see JuLeAna.take_requests!) that the interface carries out in the background and answers with JuLeAna.workers_result!. The selections are kept by allocation number and process ID, so they stay on their item when the lists change.

JuLeAna.workers_key!(page::WorkersPage, evt::Tachikoma.KeyEvent, send; snap = nothing)::Bool

Apply the key event evt to page. send(cmd) queues the controller command cmd and snap is the latest controller snapshot. Return true if the key was used and false otherwise.

• Tab and Shift-Tab switch between the allocations and the workers or SLURM pane.
• Up and Down move the selection in the focused list; over the SLURM text they scroll it. PageUp, PageDown and End scroll the output of the focused pane, End to the newest line.
• s switches the right pane between the workers and SLURM, if the overview has SLURM output.
• a sends AddWorkers.
• c asks for confirmation to cancel the selected allocation if it is running, C for all allocations. The question says that items running on these workers fail; y sends CancelAllocation or CancelAll, any other key cancels the question.
• p sends PostponeIdleCancel.
• r reads the overview again.
• e opens the start script in a full-page editor, or shows why there is none. In the editor Ctrl-O sends EditStartScript with the text and closes it, Escape closes it without saving, and every other key edits the text. A shown message closes with any key.
• o opens the form of the SLURM options, such as the time limit and the number of nodes, if snap has the run mode settings; JuLeAna.slurm_form_key! handles the keys while it is open.
• Other keys, such as 1 to 7 and q, are not used while no editor, form, message or question is open.

JuLeAna.workers_mouse!(page::WorkersPage, evt::Tachikoma.MouseEvent)::Bool

Apply the mouse event evt to page: a left click on a row of the allocations or the workers selects it and focuses its pane, and the scroll wheel scrolls the list, the output pane or the SLURM text under the pointer by three lines; scrolling a list keeps its selection. The mouse does nothing while the editor, a message or a question is open. Return true if the event was used.

JuLeAna.render_workers(page::WorkersPage, snap, busy, messages::JuLeAna.MessageBuffer, area::Tachikoma.Rect,
    buf::Tachikoma.Buffer; at::DateTime = now(UTC))

Draw the Workers page into area of buf. snap is the controller snapshot (or nothing), busy the busy workers (see busy_workers), messages the message buffer that keeps the output of the workers (see JuLeAna.worker_output) and at the current UTC time.

A line at the top shows the number of workers and the countdown to the cancel of idle allocations (idle cancel in mm:ss, or no idle cancel). Two columns follow. The left one lists the allocations (number, backend, state, SLURM job name and job IDs, run time or the time it ran, the exit code of a failed one) with the output of the script of the selected allocation below, the newest line at the bottom. The right one lists the workers (process ID, host, the item a busy worker runs or idle) with the lines the selected worker wrote below; while that output is scrolled up, it stays on the same lines as new ones arrive. If the overview has SLURM output, the right pane can show it instead (squeue, then sinfo -t idle) over the whole column. The focused pane has the heavy border.

While the start script is edited the editor fills the whole area. A message about the start script, the form of the SLURM options and the question before a cancel are drawn over the page.

JuLeAna.workers_result!(page::WorkersPage, answer)

Take the answer to a request of page as JuLeAna.read_workers_page returns it: the overview replaces the one of page, or the message of a failed read becomes its error and the shown overview stays.

JuLeAna.workers_visit!(page::WorkersPage; at::Real = time())

Queue a read of the overview unless one is under way or the latest was queued less than five seconds before at.

JuLeAna.workers_stale!(page::WorkersPage)

Mark the overview of page as out of date, so that the next JuLeAna.workers_visit! queues a read at once, even if a read is under way.

JuLeAna.workers_selected_allocation(page::WorkersPage)

Return the selected allocation of page (the first one if none was selected) as the named tuple of the overview, or nothing if there is none.

JuLeAna.workers_selected_worker(page::WorkersPage)

Return the selected worker of page (the first one if none was selected) as (; pid, host), or nothing if there is none.

ProductionForm(; parent = <folder that holds the current production>, shell, home)

State of the form of the terminal user interface that collects a ProductionSpec: the text of each field, the yes/no choices, the rows of the lists of extra tier paths and of parameter folders, the field under the cursor, the message of the last rejected input and the progress of the creation. Keys are handled by JuLeAna.production_key!, the form is drawn by JuLeAna.render_production.

phase is :edit while the form is filled in, :creating from the submit until the controller has created and loaded the production (see JuLeAna.production_sync!), and :done once it has. shell and home select the startup file that set_default writes (see ProductionSpec).

JuLeAna.production_fields(form::ProductionForm)

Return the fields of form in the order of the cursor, as (kind, row) pairs: row is the index into the list of a :tier_label, :tier_path or :par field and 0 for the others. The branch fields are present only if the branches are checked out.

JuLeAna.production_spec(form::ProductionForm)

Return the ProductionSpec that form describes. Surrounding blanks of the text are removed; the reference and the branches are nothing unless they are checked out and not empty, and a tier row or parameter row without any text is left out.

JuLeAna.complete_path(text; home = homedir())

Complete the folder path text against the file system: return (; text, candidates) where candidates are the names (with a trailing /) of the folders in the parent folder of text whose names start with the last component of text, and text is the input extended by the longest common prefix of the candidates. A leading ~ is replaced by home. Hidden folders are candidates only if the last component starts with a .. A relative path, or a parent folder that does not exist or cannot be read, has no candidates and text is returned unchanged. Reads one folder and does not descend into any.

JuLeAna.production_key!(form::ProductionForm, evt::Tachikoma.KeyEvent, send)

Handle the key evt in the new-production form and return :close when the form closes, :finished when a key ends the display of the created production, true otherwise: the form uses every key. send(cmd) sends a controller command.

• Up, Down and Shift-Tab move between the fields. Tab on a path field (parent folder, tier path, parameter folder) completes the text with JuLeAna.complete_path and lists the folders that remain; when there is nothing left to complete it moves to the next field.
• Printable characters and Backspace edit the text field under the cursor.
• Space or Enter flips a yes/no field. Enter on a text field moves to the next field.
• Ctrl-N, or Enter on a [+] field, adds a row to the list under the cursor (extra tier paths, parameter folders) and Ctrl-D or Delete removes the row under the cursor.
• Ctrl-O, or Enter on the [ Create ] field, checks the input with JuLeAna.check_production and sends CreateProduction for it. A problem is shown in the form and nothing is sent.
• Escape closes the form.

While the production is created no key does anything. Once it is created and loaded, any key returns :finished.

JuLeAna.production_answer!(form::ProductionForm, answered::Integer, error)

Take the answer to a command: when it is the answer to the CreateProduction command of form (the answered-th answer, see the field answer_at), a rejection with the exception error becomes the error of the form and its fields can be corrected, else the command is accepted and the creation runs.

JuLeAna.production_sync!(form::ProductionForm, snap, lines)

Bring form in line with the controller snapshot snap and the lines of the creation log (see JuLeAna.production_log). Once the command is accepted and the controller is no longer :loading, the form is :done or, if the controller kept an error, back in :edit with that error.

JuLeAna.render_production(form::ProductionForm, snap, area::Tachikoma.Rect, buf::Tachikoma.Buffer; ticks = 0)

Draw the new-production form as a page filling area: one field per row with the label at the left and the text or the yes/no value at the right, the field under the cursor highlighted, scrolled to keep it in view; below them the first line of the error of the last input, wrapped to at most three rows, and a line of keys. While the production is created, the lines of its log (the newest that fit) replace the fields, and a finished creation shows the path of the production.

JuLeAna.read_workers_page(ctrl::DataflowController, request)

Carry out a request of a JuLeAna.WorkersPage (see JuLeAna.take_requests!) and return (; request, value) with the JuLeAna.workers_overview of the workers of ctrl, or (; request, error) with the message of the exception if the read failed. Runs SLURM commands, so it belongs in a background task.

JuLeAna.strip_span_tags(text::AbstractString)

Return text without the HTML <span …> and </span> tags that the processors put around the colored status words of their reports, so that the words remain.

JuLeAna.tables_as_code(text::AbstractString; width = JuLeAna.REPORT_TABLE_WIDTH)

Return text with each Markdown table (consecutive lines that start with |) replaced by a fenced code block that holds the table as aligned text, the header followed by a rule. The Markdown renderer of Tachikoma leaves tables out; a code block shows their cells. HTML tags in the cells are removed and their text is kept. A table wider than width characters is narrowed by limiting its widest columns, down to at least JuLeAna.REPORT_MIN_COLUMN_WIDTH characters each, and a cell longer than its column continues on the following lines of its row.

JuLeAna.logo_size(rows)

Width in terminal cells and number of rows of the logo rows, as in JuLeAna.render_logo.

JuLeAna.render_logo(rows, x, y, buf::Tachikoma.Buffer; max_x, max_y)

Draw the logo rows with its top left cell at column x and row y of buf, clipped at column max_x and row max_y (default: the edges of buf). Each row is a string in the text color of the theme or a vector of (text, color) segments with the color :outline (the text color of the theme), :red, :green or :purple. Spaces are transparent: they leave the cell as it is.

JuLeAna.bar_segments(counts, width::Integer)

Widths in cells of the segments of a bar of width cells that shows counts (a non-negative number per segment, in the order shown). The segment boundaries are the rounded cumulative shares of the total, so the widths add up to width (to 0 if all counts are zero) and a segment is empty only if its share is below half a cell.

KwargsEditor(step::Symbol)

State of the overlay that edits the kwargs of the processor step on the Setup page: the cursor row, the text being typed while a value is edited (nothing otherwise), and the error of the last typed value.

JuLeAna.kwargs_key!(editor::KwargsEditor, evt::Tachikoma.KeyEvent, snap, send)

Handle the key evt in the kwargs editor and return :close when the editor closes, true otherwise; the editor uses every key. Up and Down move, Enter starts editing the value under the cursor and, while editing, sends it as SetKwarg once it parses (see JuLeAna.parse_kwarg_value); on the log or plot level row, Enter sets the next level with SetStepLevel. Escape stops editing or closes the editor; u resets the kwarg or level under the cursor and U all kwargs and levels of the processor with ResetKwargs.

JuLeAna.render_kwargs(editor::KwargsEditor, snap, area::Tachikoma.Rect, buf::Tachikoma.Buffer)

Draw the kwargs editor as a box in the middle of area: a row for the log and the plot level of the processor, then one row per kwarg in alphabetical order with its value, * and the config value for a changed level or kwarg, and dimmed the accepted kwargs that are not set; the text being typed and the last error below.

RunmodeEditor()

State of the overlay that edits the settings of the run modes on the Setup page: the cursor row, the text being typed (nothing when no value is edited), whether that text is a new entry of a list, and the error of the last typed value.

JuLeAna.runmode_key!(editor::RunmodeEditor, evt::Tachikoma.KeyEvent, snap, send)

Handle the key evt in the run mode editor and return :close when the editor closes, true otherwise; the editor uses every key. Up and Down move. Enter edits the value or entry under the cursor, and on the heading of a list adds an entry; a adds an entry to the list under the cursor and d deletes the entry under the cursor. While a text is typed, Enter sends it as SetRunmodeSettings once it is valid (a positive integer for local_n, a text that is not empty for an entry of a list) and Escape drops it; otherwise Escape closes the editor.

JuLeAna.render_runmode(editor::RunmodeEditor, snap, area::Tachikoma.Rect, buf::Tachikoma.Buffer)

Draw the run mode editor as a box in the middle of area: the settings of the local and the slurm run mode, with the number of workers on one row and each list as a heading with one row per entry; the text being typed and the last error below.

SlurmForm()

State of the form on the Workers page that edits single options of the SLURM job, such as the time limit and the number of nodes: the cursor row, the text being typed (nothing when no value is edited) and the error of the last typed value.

JuLeAna.slurm_option(flags::AbstractVector{<:AbstractString}, name::AbstractString)

The value of the SLURM option name set as --name=value among flags, or nothing.

JuLeAna.set_slurm_option(flags::AbstractVector{<:AbstractString}, name::AbstractString, value::AbstractString)

Return a copy of flags with the SLURM option name set to value as --name=value, in place of a flag that set it before or else at the end. An empty value removes the option. Throws an ArgumentError if value is not valid for the option, or if flags set the option in another form, such as a short option.

JuLeAna.slurm_form_key!(form::SlurmForm, evt::Tachikoma.KeyEvent, snap, send)

Handle the key evt in the SLURM form and return :close when the form closes, true otherwise; the form uses every key. Up and Down move, Enter edits the option under the cursor. While a value is typed, Enter sends the flags with the new value as SetRunmodeSettings once it is valid, an empty value removes the option, and Escape drops the text; otherwise Escape closes the form.

JuLeAna.render_slurm_form(form::SlurmForm, snap, area::Tachikoma.Rect, buf::Tachikoma.Buffer)

Draw the SLURM form as a box in the middle of area: one row per option with its value or not set, the other SLURM flags, the text being typed and the last error.

JuLeAna.parse_kwarg_value(text::AbstractString, current)

Parse the text text typed for a processor kwarg whose value is current (nothing for a kwarg that is not set). The text follows the type of current: true/false for a Bool, a number for an Integer or a float, the text itself for a string, and names separated by commas for a vector of symbols. Otherwise the text must be a Julia literal: a number, true/false, a string in quotes, a :symbol, or a vector of these such as [:a, :b]; it is only parsed, never evaluated. Throws an ArgumentError naming the expected form when the text does not fit.

Examples

julia> JuLeAna.parse_kwarg_value("12", 10)
12

julia> JuLeAna.parse_kwarg_value("V01, V02", [:V00])
2-element Vector{Symbol}:
 :V01
 :V02

julia> JuLeAna.parse_kwarg_value("[1, 2]", nothing)
2-element Vector{Int64}:
 1
 2

JuLeAna.InterfaceTiming(; window_seconds = 10, clock = time_ns, utc = () -> now(UTC), capacity = 8192, pages = 16)

Recorder of how responsive the terminal interface is, one window of window_seconds at a time. It keeps, in fixed-size buffers of capacity entries per kind of measurement (measurements beyond that are counted but not stored):

• the interval between the starts of consecutive view calls (frame gaps),
• the time spent in each view call and in each Tachikoma.update! call, with the maximum per event kind (key, mouse, task) and the kind or task id of the slowest call,
• per page 1:pages, the number of view calls and the time spent in them, and the page of the slowest view call,
• the key-to-redraw latency: from the start of the update! call for a key press to the end of the next view call. It does not include the time a key waited in the terminal input before the application loop read it; that shows up as a long frame gap,
• the number of garbage collections, the time spent in them and the longest pause, from Base.gc_num(),
• the thread and thread pool of the first view call.

The functions JuLeAna.timing_view_begin!, JuLeAna.timing_view_end!, JuLeAna.timing_update_begin! and JuLeAna.timing_update_end! feed the recorder; each takes one time reading and writes into the buffers without allocating. JuLeAna.timing_flush! closes the window.

clock returns the time in nanoseconds and utc the wall-clock time of a row; both can be replaced to feed synthetic times.

JuLeAna.TIMING_COLUMNS

Column names of the file interface_timing.tsv, in order: UTC time of the end of the window, controller state, number of frames, frame gap median, 95th percentile and maximum, view time 95th percentile and maximum, update! time 95th percentile and maximum, number of key presses, key-to-redraw latency median, 95th percentile and maximum, garbage collections in the window (count, time, longest pause), the longest update! for key, mouse and task events, the event kind or task id of the slowest update!, the length of the window in seconds, threadid:threadpool of the thread that draws, the page shown during the slowest view and, for each page shown, page:frames:mean view time separated by spaces. Times are in milliseconds; NA marks a statistic without measurements.

JuLeAna.timing_view_begin!(t::InterfaceTiming)

Record the start of a view call in t: the gap since the start of the previous call and, on the first call, the thread it runs on (also logged). Returns the start time.

JuLeAna.timing_view_end!(t::InterfaceTiming, snap; page = 0)

Record the end of a view call in t that showed the page page (0 for none): its duration, per page if page is one of the pages of t, and the key-to-redraw latency of the key presses since the previous call. When the window of t has elapsed, close it with JuLeAna.timing_flush!, using the controller state and log folder from the controller snapshot snap (nothing before the first read); returns the row of the window, else nothing.

JuLeAna.timing_update_begin!(t::InterfaceTiming, evt)

Record the start of a Tachikoma.update! call for evt in t; the start of a key press is kept until the end of the next view. Returns the start time, to pass to JuLeAna.timing_update_end!.

JuLeAna.timing_update_end!(t::InterfaceTiming, start::UInt64, evt)

Record the end of a Tachikoma.update! call for evt that started at start in t.

JuLeAna.timing_flush!(t::InterfaceTiming, state, log_path)

Close the window of t: format its statistics as one tab-separated row (see JuLeAna.TIMING_COLUMNS for the columns), start a new window and return the row. state is the state of the controller. If log_path is a folder name, the row is appended to interface_timing.tsv in it by a separate task, preceded by the header if the file does not exist; the caller never waits for the file. With log_path = nothing or "" nothing is written.

Frame gaps, view times and update! times are in milliseconds. The garbage collector counts and time are those of the window. gc_max_pause_ms is the longest pause of the window if it exceeds all earlier pauses of the process, else an upper bound: the lesser of the longest earlier pause and the time spent in garbage collection in the window.