console-automation

Recipe card from the charly-check plugin (Commands — runtime CLI verbs).

Console automation — transport-neutral screenshot+OCR+keyboard driving

Overview

Much of what an IP-KVM or a VM console is used for is the SAME task: drive a machine’s text console with nobody at the keyboard, reading each screen by OCR and sending keyboard input. The MECHANISM is shared — it lives in sdk/kit — so ONE authored recipe/flow/session drives either transport:

Layer	Where	What it owns
kit.ConsoleTransport	sdk/kit	the 4 primitives: Capture / PressKey / PressCombo / Type
kit.ConsoleWizard	sdk/kit	one-shot wizard: OCR-wait an anchor, send one action, next step
kit.ConsoleSession	sdk/kit	an OPEN terminal: run commands, read results, sudo, passphrase
kit.ConsoleFlow	sdk/kit	a bounded state machine: continuous OCR + control flow
plugin-jetkvm	candy/plugin-jetkvm	the USB-HID transport (jetkvm:)
plugin-spice	candy/plugin-spice	the SPICE-keyboard transport (spice: wizard, the generic actions)

The CUE AUTHORING SURFACE lives in each plugin’s own schema/*.cue (SDD); the shared engine holds NO wire type. A kind: jetkvm entity is the transport-neutral home of a recipe (both verbs read it).

open-terminal / run-command / close-terminal

Drive an interactive shell and READ EACH RESULT BY OCR:

- check: the installed system is Omarchy
  context: [runtime]
  jetkvm:
      method: run-command
      allow_control: true
      sudo_password_secret: OMARCHY_KVM_PASSWORD   # credential store
      commands:
          - {command: "cat /etc/os-release | grep ID=", expect: "omarchy"}
          - {command: "efibootmgr", sudo: true, expect: "BootOrder"}
      close_terminal: true

open-terminal sends terminal_combo (default super+Return; ctrl+alt+F3 for a bare TTY) and waits for a prompt_anchors match. close-terminal sends exit.

Completion is a MARKER LINE, not a prompt (RCA). A shell reuses one terminal across commands; a prompt is neither screen-unique nor stable. Worse, the terminal ECHOES the typed line, so waiting for the command text — or for the marker as a SUBSTRING — passes the instant the command is typed, BEFORE it runs, certifying completion that never happened. So the engine types command; echo <opaque-marker> and waits for a screen LINE whose trimmed text EQUALS the marker: only the shell’s OWN output produces that. It is a condition poll, never a sleep (R4).

luks-unlock — the encrypted-install first boot

A default Omarchy install is encrypted and stops at an initramfs prompt (A password is required to access the root volume:). There is no shell there, so completion is a screen condition: luks-unlock enters passphrase: / passphrase_secret: and waits for outcomes: (default login:, Welcome, succeeded, Booting). A wrong passphrase is caught by a built-in failure anchor (No key available, wrong password) and FAILS the step — never a silent “outcome”. Secrets only ever come from *_secret: credential-store keys.

flow — continuous OCR with control flow

flow drives a BOUNDED state machine. It replaces guessed timeouts and brittle single anchors with explicit DATA:

• CONTINUOUS OCR-until-condition — each node’s wait: lists NAMED outcomes; the engine re-captures + OCRs (a real poll) until ANY match appears, reporting WHICH. No guessed wait time.
• if/then/else + case/switch — transitions: maps an outcome NAME to the next node id; the OBSERVED outcome selects the branch. No transition → next:.
• while loop — a transition pointing BACK to an earlier node is a loop, bounded by flow_max_loops (per node) and flow_max_steps (whole flow). A condition that never becomes true FAILS naming max_loops — the R4-safe “repeat until”.

A node action may be a raw input (key/combo/text) or a shell command: (run in the open terminal, output OCR-read). The flow validates WITHOUT a device.

- check: log in and read the EFI boot order
  context: [runtime]
  jetkvm:
      method: flow
      allow_control: true
      flow_start: login
      flow_nodes:
          login: {wait: [{name: prompt, match: "login:"}], text: "root", next: submit}
          submit:
              wait: [{name: shell, match: "archiso"}, {name: pw, match: "Password:", failure: true}]
              key: Return
              transitions: {shell: listboot}
          listboot: {wait: [{name: shell, match: "~"}], command: "efibootmgr", expect: "BootOrder"}

Reference-screenshot outcomes — match by PIXELS, not OCR

An outcome can match by a REFERENCE SCREENSHOT instead of (or as well as) an OCR substring: reference: names a host path to a PREVIOUSLY-CAPTURED screenshot, and the outcome matches when the current frame’s perceptual hash is within max_distance: (default 5) of it. This is the right tool for a screen that OCRs badly — a firmware menu, a graphical lock, a splash — and it lets an action trigger “when the current screen matches the reference”:

- check: act when the login screen reappears
  jetkvm:
      method: flow
      allow_control: true
      flow_start: see
      flow_nodes:
          see:
              wait: [{name: same, reference: "/tmp/ref_login.png"}]
              command: "echo READY"

Capture the reference once with a screenshot: step (its artifact: is the reference path). A reference-only node pays NO OCR. Live-proven: a flow whose only wait was a reference matched the login screen and ran its command. Same screen→small Hamming distance; a >2× resolution difference is a hard no-match.

Auto-resume — recover to the right step

flow_resume: true AUTO-DETECTS the node whose wait matches the CURRENT screen and starts there, instead of at flow_start — so a flow re-run after a stall or restart recovers to the right step rather than replaying from the beginning. flow_resume_order: (node ids earliest→latest) disambiguates when a screen matches several nodes (the latest-listed match wins); an ambiguous match without it FAILS naming the candidates. Live-proven: a resumed flow detected the password prompt was step2 and started there, skipping step1/step3.

Two failure bounds every long flow must know

• Prompt preflight. run-command (and a flow’s command:) types BLINDLY unless prompt_anchors: is set. Without it, if the target is at a pager/menu/login screen, the command and its marker are swallowed and the wait burns its whole timeout. Set prompt_anchors: ["~", "#", "$"] so a command is only typed once a shell prompt is on screen — otherwise it fails FAST naming what was there.
• The per-step never-hang bound. The host SIGKILLs a check step that exceeds its per-attempt ceiling (2m by default) — a long flow dies with a bare “context deadline exceeded”. Declare timeout: on the step (a longer value is honoured over the floor) so the plugin survives to return its evidence; the flow’s own wall-clock budget then stops it CLEANLY between nodes if it still runs long.

boot-order — the OS-side EFI boot manager

boot-order sets the UEFI boot order from INSIDE the running system via efibootmgr (the OS-side counterpart to the firmware boot-menu key): list (read entries), next (--bootnext, a ONE-TIME boot — ideal for booting an installer medium once then returning to the disk), set (--bootorder, persist). next/set write NVRAM and need root.

Measured hardware facts (a real ASRock X670E)

• The disk boots first: booting a live installer ISO needs the firmware boot-menu key OR an OS-side efibootmgr --bootnext — a plain reboot boots the installed disk.
• The firmware boot menu / setup is reached by hammering the key during POST (a macro of repeated F11/F2/Delete); a single press usually misses the window.
• A dark screenshot is NOT proof input is dead. An earlier investigation wrongly concluded “HID does not reach the target” because a graphical lock screen / a Plymouth splash did not visibly change. The terminal DID accept input, and a run-command typed into a root shell and read its output back. Always confirm with an input-echoed check before concluding input is dead (R1).

OCR must UPSCALE — the anchor-misread RCA

The console engines upscale a capture 2× before tesseract, and that is load-bearing, not cosmetic: at 1× the Omarchy GUI installer’s form labels OCR as Usernane> / Confirn> / Hostnane> (tesseract confuses the word-final m> for n>), so a recipe anchored on Username> / Confirm> / Hostname> never matches and the drive stalls on the form. At 2× they read correctly; a framebuffer TTY also reads better while its shell anchors (~, #, $) survive (at 4× they do not — ~→R). So: author anchors that are robust, and know the engine’s 2× scale. A stalled wizard whose anchor “never appears” while the screen clearly shows it is this class of bug — capture and OCR the frame manually at 1× and 2× to confirm before blaming the recipe.

The install is not done at the reboot

A default Omarchy install is ENCRYPTED and stops at an initramfs prompt for the disk passphrase (A password is required to access the root volume:). To install WITHOUT encryption (the unattended-friendly path), press Ctrl+C on the overwrite-confirm screen — it flips the affirmative to “Yes, install without encryption”; encryption defaults ON otherwise. After Reboot Now, a stock Omarchy still ships sshd DISABLED and the firewall CLOSED, so reachability needs omarchy-setup-security-sshd --key="<pubkey>" (Omarchy’s OWN command: installs openssh, enables sshd, opens the firewall, authorizes the key) driven over the console after first boot.

Cross-References

• /charly-check:jetkvm — the JetKVM verb (USB-HID transport + IP-KVM device ops).
• /charly-check:spice — the SPICE verb (VM console transport; spice: wizard).
• /charly-check:check — parent router; the verbs dispatch out-of-process.
• /charly-internals:plugin — the plugin/provider model; the shared kit layer.
• /charly-internals:go — the schema→params generation recipe.

When to Use This Skill

MUST be invoked before authoring any console-driving recipe, session, or flow (an OS installer, a first-boot provisioning flow, a shell command over a console, a LUKS unlock, a boot-order change), and whenever an action must work identically over a JetKVM (USB HID) and a VM (SPICE). Invoke it BEFORE reading source or launching Explore agents.