Deployment start diagnostics

Every Yoke self-release run — the stage run over HTTPS and the serving-API production self-deploy — prepares a detached checkout at the run's release lineage before launching its engine. Locating that checkout reads only each machine-config projects entry's checkout and project_id, so the installed CLI finds a release that changes the machine-config contract. Preparation runs uv sync --frozen --no-dev against that checkout's own uv.lock; the execute adapter and deploy watcher both launch its .venv Python with pinned source imports. Ambient uv environment selectors cannot redirect the sync into the installed tool environment. Preparation is bounded to five minutes. A missing lock, uv executable, failed sync, timeout, or missing interpreter refuses as DEPLOY-DRIVER-ENVIRONMENT, with the pinned path, repair recipe, and same-run re-drive instructions. Watcher failures remain in its raw and progress captures. Every other project's delivery keeps the installed engine environment.

A driver connected straight to the control-plane database runs the release's own code against it before that release has booted. Before its first in-process call it applies boot's creation phase — tables, indexes and additive columns, never the ordered migration history — so a release that adds a column can start. It runs as the role that owns the database's tables, so a table it creates stays convergeable by the serving build. The phase reads the catalog first and runs only the statements whose object is missing, so a release with no schema change takes no table locks; it commits one table at a time, so it never holds one table's lock while waiting on another's. Every run on one control plane converges the same database, so one driver at a time holds the release-start schema lock; a second driver waits, then finds every object present. A driver still waiting after two minutes refuses as release_start_schema_lock_unavailable, naming the run that holds the lock. A concurrent creation of the same object, such as by a booting server, is retried; one that persists refuses as release_start_schema_convergence_failed without blaming grants. Each statement waits at most the run-lock timeout; a busy table is retried with backoff and then refuses as release_start_schema_lock_unavailable, naming the table, and any other failure refuses as release_start_schema_convergence_failed. Each refusal names the re-drive step and comes before anything about the run changes. A re-drive of a run that already started skips the phase: its first start converged the schema and its candidate is frozen. A driver relaying through HTTPS skips this step, because the serving build answers every call.

A normal driver start builds its execution context once. Composition completes and validates membership before the driver reads it, and returns that pass’s enrolled references. Branch verification and QA seeding use those members. The driver reuses the context’s containment basis when starting the run. That basis includes merged open items and items finished since the latest succeeded release started on the same project and target environment; before the first success, it includes all recorded finishers. The same anchor covers bound projects and bound-source revalidation. Elapsed time alone never changes the question set; the Done band's 24-hour display window is separate.

The executing transition still locks stable membership, validates mutable item bindings, freezes requirements and intent, verifies the containment digest, and stamps execution in one transaction. A stale digest rolls that transaction back. The driver refreshes only the basis and re-attests, with at most three retries; other refusals stop immediately and name the repair and re-drive step. deployment_runs.execution.containment_basis is the narrow registered read:

yoke deployment-runs execution containment-basis RUN-ID --json

It refuses while another session is the run's live driver. A serving build below its declared minimum version refuses by name; use the same-universe owner’s local authority when driving the control plane’s own upgrade. Ordinary projects use HTTPS.

Coverage resolves member completion flows as a set, with one project/workflow default resolution per distinct pair. The flow and source record are shared within a composition operation, and so are the repository reads: each bound branch head resolves once, and each project's commit comparison against its preceding release runs once however many readers ask. The execution context records bound sources and validates composition on one connection, so its validation reuses the heads the record resolved. A re-drive of a run whose composition is frozen skips that validation. It only sets aside or refuses members that left delivery readiness since the freeze, and refuses a run left with no member its item-scoped QA can prove (timing step context_frozen_composition). A comparison whose preceding release changed is a different answer and is derived again. Catalog probes share table and column answers only inside that operation and only for the same connection. Exiting, including on refusal, discards the scope; no process-global or cross-request cache exists. Mutable member status/intent is read afresh. Freeze reuses one custody resolution for both membership and completion-authority checks; unknown custody still refuses.

Create validates provisionally before the creation lock, rolls that transaction back, then validates again under the lock. The locked pass reuses each project's carried-work comparison from the provisional pass when its commits and preceding release are unchanged; custody, membership and readiness are read again under the lock. Each enrollment pass first pins members' completion flows, reads their outstanding intake together, and locks selected requirements in ID order. It reads attachments together, locks distinct plans in ID order, and reads their cases once. Every member still passes the existing ownership, waiver, retraction, attachment, project, retirement and case-chain checks. These mutable inputs belong only to that enrollment; the locked pass reads them again after provisional rollback. Membership and takeover still commit atomically, and derived selections are still derived again at composition freeze. Old runs continue to use their saved QA contracts.

After each member removal, the orphaned plan-group check reads the remaining members' live attachments. A group with no attaching member left is retracted; its passed rows stay. A removed member cannot keep a group alive through cached membership. The outer schema/source cache is a separate scope.

Completion publishes exact run/member/environment/candidate attribution in the existing delivery-stamp facts, in the same transaction as the item's done status. An independently completed member stays delivered even if its siblings or run later fail or cancel. Dependency and dashboard readers consume this durable answer. Accepted QA and a preflight stamp alone do not mean the item completed.

The driver raw capture contains Deployment start timing: JSON lines from preflight pin reading onward. Each step has a UTC start/end timestamp, monotonic elapsed milliseconds, outcome, run ID, source SHA, member count, and transport. Failed steps also record reason_code: the refusal's code when present, otherwise the exception type, so a planned retry remains distinguishable from a defect. SHA and member count remain unset until their authoritative reads. Pin reading, worktree retirement/ensure, child launch, context substeps, branch verification, QA seed, containment attestation, freeze, and executing stamp are named separately. HTTPS returns server substeps with its response for the driver to flush; local execution collects them through the same contract. The watcher preserves preflight lines when it appends child output. Records carry no secrets and are never operational authority.

Each run-scoped driver scope (preflight, watched child, pinned re-exec, pipeline) also stores its start/end records in deployment_run_timings through deployment_runs.timings.record when it closes; the pipeline also flushes at each stage, so a running run's earlier steps are readable. Any machine can read them:

yoke deployment-runs timings RUN-ID [--json]

It prints per-step elapsed totals, then every record; --json returns each record as the driver wrote it. A write that cannot land never fails the run: the driver prints Deployment timing record refused: with the run, the named error and the recovery, the capture still holds the lines, and the next flush retries them without duplicating any that landed. An empty answer names that no timings were recorded.

For a start-gap measurement, subtract the earliest preflight-start timestamp from the successful executing-transition end (both on the driver’s clock) in that same run’s records. Report the sum of context elapsed time, freeze elapsed time, and any refresh attempts alongside the total. Compare matched source/transport/member-count conditions; missing markers are incomplete evidence, never a zero-time result.

The read-only report computes it from the recorded timings:

python3 -m yoke_core.tools.deployment_start_report --run-id RUN-ID --baseline-seconds SECONDS

The report refuses incomplete markers or more than one context build, and prints the measured total and every step’s elapsed time; a slower start still reports its measured difference.

Source/provider and containment preparation precede item/run locks. The short commit phase rereads membership, workflow, evidence, flow and source pins; only an identical repository question reuses its prepared answer. Carried-work attribution reads lane heads only for landed items, so workers committing in open lanes during preparation ask no new question. A changed source question (for example an item that landed meanwhile) rolls the transaction back, answers only that question with every lock released (timing step start_source_refresh) and retries the commit phase, up to eight preparations. After that the start refuses as deployment_source_preparation_changed, naming the exact source key; nothing was dispatched, so re-drive the same run.

Creation prepares against the waiting run when its takeover would settle that run succeeded; a cancelled takeover keeps the preceding succeeded release as its base. This unlocked prediction moves unclosed members into provisional membership so validation prepares their source questions too. Rollback restores the waiting run without removal audits, notices or settlement; the locked pass rechecks and applies the takeover. If creation exhausts source preparation, repeat the original yoke deployment-runs create or yoke deployment-runs start-for-item command with the same arguments and idempotency key, if supplied; no new run was created.

QA occupancy uses the first real blocker and shares peer-plan facts within that evaluation; full diagnostic reads still report all blockers. Item/run/target lock contention refuses as deployment_start_lock_busy after two seconds and names same-run recovery. It is not a measured diagnosis of which older transaction held a lock.

An item-QA cursor followed by QA/approval gates and the declared final auto completion step is eligible for takeover. Any remaining execution step is not. Containment, same flow/target and driver ownership still apply. Recreating a run alone cannot repair those facts. A delivered run can settle while its member still owes manual acceptance or post-deploy QA on another named target; that requirement stays unmet and the item stays at release until it is accepted. Same-target or untargeted acceptance still blocks completion.

A mutation timeout or connection loss is deployment_outcome_unknown, not a deployment failure. Bound-source preparation commits its exact request receipt beside the consumed sources on the run; the driver reads that receipt before proceeding after response loss. An absent, mismatched or unreadable receipt leaves the attempt pending, with no retry, provider dispatch or failed transition. Inspect yoke deployment-runs get RUN-ID and the existing driver's capture, and reconcile the request or provider attempt before re-driving the same run; never use --fresh to recover a lost response. The live attachment's phase and capture show preparation even while run status remains created.

QA dispatch closes members within a time budget and recovers response timeouts over HTTPS using the existing request reservation and result.

A lost answer leaves the run in its QA stage with status executing, so live QA executions keep writing, and the next re-drive continues that stage. A genuine stage failure records run status failed before the <stage>-failed stage. When that status write is refused, for example qa_abort_restore_required while host QA work is in flight, the driver reports run_failure_unrecorded with the refusal's recovery and leaves both status and stage unchanged.

A scoped-QA dispatch whose database lock wait is cancelled or times out, for example when an operator cancels a statement stuck behind a held run row, is not a stage failure either. It answers qa_stage_dispatch_interrupted, naming the SQLSTATE and the current deployment_runs lock holder. Members settled before the interruption keep their result, the run stays executing at its QA stage, and the driver re-dispatches that stage inside its bounded QA wait.

A member's QA plan configuration refusal is not a stage failure either. When the gate refuses one member's saved plan, for example qa_plan_source_snapshot_missing, the stage records a qa_configuration_refused blocker for that member and keeps waiting on the others. The driver does not exit for it. The run stays executing at its QA stage and every recorded QA result is kept. Fix the named configuration, then re-drive the same run with yoke watch deploy -- RUN-ID. A correction already frozen on that member counts as selected, so attaching the same plan does not raise qa_frozen_plan_snapshot_incomplete.

The scoped-QA wait polls a run fingerprint every 15 seconds and re-dispatches only when QA facts change, plus once at the deadline. It announces its poll and status cadence and deadline, then prints Still waiting for scoped QA with elapsed time and blocker count every four polls (one minute by default). Each fingerprint read or re-dispatch gets only the time left on that deadline: a read or dispatch still unanswered then ends the wait with a named timeout. When the bounded scoped-QA wait ends with blockers, the driver returns awaiting QA (exit code 5). If the following fresh diagnostic report read fails, it explicitly reports that fresh QA details are unavailable and retains the last known blocker diagnostic. That read failure does not change the established awaiting-QA outcome, supply fresh counts, waive a gate or record success. Inspect the existing driver's capture and scoped QA state, resolve the remaining obligations, then re-drive the same run.

Whole-run reads project each stored field directly. Shell pipes, newlines and quotes in frozen member instructions remain part of the complete receipt; they cannot split its fields or displace source identities and timestamps.

Timing records distinguish start_source_preparation, start_admission_reads, start_item_run_locks, start_admission_recheck, start_source_refresh, target_occupancy, composition_freeze, bound_source_provider, bound_source_composition, bound_source_enrollment, bound_source_judgement, bound_source_item_run_locks, bound_source_recheck, bound_source_commit and executing_stamp. The start_admission_reads, bound_source_composition, bound_source_enrollment and bound_source_judgement steps read every member, and judge the members enrollment would add, before the run row lock; the two _recheck steps prove those reads current under it, and a changed input repeats the reads while the lock is held. A lock timeout names the phase, not an invented blocker; missing response/timing evidence stays unknown. Real-run verification must name the exact deployed source and representative members; a small fixture is a regression check, not proof of production startup time.

Fleet preparation reports the actual subphase under the declared deployment stage: declarations, receipt authority, admin connection, roster, ownership, extensions, copy dump/restore, boot convergence, invariants, product readers, cleanup and receipt writing. Long subphases emit phase=running, outcome=pending and elapsed milliseconds every thirty seconds. Restored-copy reader diagnostics identify sessions, frontier, board, launch pages/details, message attempts and items, with record_count and completed_count for each record family. Every copied record still reaches its existing handler or decoder. Bounded batches reuse the canonical session projection and item text expressions, and fetch launch and attempt rows in sets; no check is sampled or skipped. Batch data is discarded on exit and cannot cross copy authorities. Connections opened under a bound copy still physically close. Every timing line stays in the driver raw capture, which is where preparation is identified before any provider dispatch. Outcome-only watchers do not forward each line on its own: ordinary lines join the progress digest, at most one digest line per flush window, which yoke watch tail holds until the next urgent line or sentinel rather than waking on it; a line whose outcome is failed is urgent and is forwarded immediately. Other watchers still emit timing lines as they arrive. They carry run/source identity when supplied and no DSNs. A missing end marker is unfinished evidence, never a successful receipt.

Validation reads member workflow pins and skipped-candidate status in sets, interprets each distinct pinned workflow version once per set, and resolves refusal and rework labels together. Mutable statuses are read again on the locked pass; no validation result survives provisional rollback. The nested steps validation_bound_sources, validation_custody, validation_carried_work, validation_enrollment_plan and validation_judgement show where validation spends its time.

Creation uses the same timing format for create_source, create_provisional_composition, create_lock, create_locked_composition, create_takeover, create_validation, enrollment_members, create_commit, create_takeover_settlement and create_response. Source refresh attempts have their own marker. Long phases emit the same thirty-second pending heartbeat; member validation also reports record/completed counts. The creation handler collects these diagnostics and returns them in start_timings on both local and HTTPS transports. The CLI saves the timing records to complete private evidence and prints its path beside the compact creation receipt; it does not repeat the records as an inline table. Deployment execution keeps its streamed driver timings. Nested phase durations overlap and must not be added as independent wall time. A missing response or end marker does not authorize another create; keep the request's idempotency key and reconcile the existing attempt.

An unchanged fingerprint retains the previous blocker result. A dispatch reads gate facts once across the run's members; acceptance writes re-check their own subject under the run lock. The local driver resolves its actor at startup and reuses that binding until it exits. A closed database connection, including a failed actor lookup, reports control_plane_connection_lost; a lost write answer leaves the run resumable with its pending receipt preserved. Restore the control-plane connection, inspect the run and its driver capture, then re-drive the same run using yoke watch deploy -- RUN-ID.

Before a dispatch begins its database writes, it resolves each project's immutable release pin and published installer record with autocommit reads, so Git, GitHub contents and HTTP work hold no database transaction open. The answers are reused for that dispatch. A newly changed source input refuses as deployment_qa_source_inputs_changed; re-drive the same QA stage to refresh its inputs. If a connection drops while handling another error, the driver reports both the original error and the rollback loss as a resumable connection loss. An idle-in-transaction timeout is named as the original cause.