Executable Procedural Memory Graphs: Remembering Procedures, Not Policies

Vishnu E, Sankar Nivas M

Abstract

Deployed robot work almost always already has a procedure. Aviation, surgery, pharmaceutical manufacturing and regulated maintenance specify standard operating procedures: an order that must hold, checkpoints that must be observed, actions that must never be taken. A system that finds a better route has still failed to follow the procedure. This constraint is organizational, not epistemic, so no improvement in model capability removes it.

The requirement this creates is narrower than it appears. It is not that the robot repeats identical motions — conditions vary and adaptation is what a learned controller is for. It is that every execution satisfies the procedure’s invariants, and that satisfaction be checkable afterwards. Determinism is required in conformance, not in trajectory.

We describe a system that places determinism in the decision function rather than in the motion. A Procedural Memory Graph (PMG) holds persistent, generalized procedural knowledge for a task. A symbolic reasoning engine plans over it by backward chaining against grounded scene facts, composes a per-episode execution graph, and walks that graph, handing a single action at a time to an arbitrary low-level controller. Vision-language models appear only as sources of facts, never of procedural decisions, behind a typed boundary where prose stops. Given the same grounded situation the planner returns the same plan, so any behavioural difference between two runs is attributable to a specific fact that differed.

Execution is closed-loop, which is not new; what the loop produces is. Each step yields three separable signals — a predicted effect, an execution report, and an independent grounded verdict — and each condition declares how it is verified, so divergence can be classified rather than merely recovered from: mechanics error, grounding error, verification error, transition error, action selection error, execution error, rig capability. Attribution of this kind needs somewhere to land, which is why knowledge in a PMG carries provenance, confidence, and unresolved questions paired with the experiments that would settle them.

We report a preliminary negative result of exactly this form on an eight-level puzzle environment: the procedure was followed faithfully and the run failed because an encoded mechanic was wrong — a diagnosis an end-to-end policy cannot produce about itself, and one a replanner would have silently corrected around.

1. The problem is conformance, not capability

A vision-language model in an execution loop reads an observation and proposes a next action. Run it again and it proposes something else reasonable. For novel tasks that is a feature. For deployed work it is disqualifying, because the work already has a procedure and the procedure is not the model’s to choose.

Two things get conflated under “the robot should do the same thing every time”:

Trajectory determinism — identical motions on every run. Brittle, unreachable under real variation, and not what any operator actually wants.

Procedural determinism — identical decisions about what to do and in what order, with physical realization free to adapt. This is what an SOP specifies and what an audit checks.

An SOP is best read as a set of invariants over executions rather than a reference path. A run is correct if it satisfies them, not if it matches a stored trajectory. The analogy we find clarifying is compilation: a compiler guarantees semantics, not identical assembly. Same source, different target, same meaning.

Reports from industrial deployment point the same way — what unlocked scale in one documented factory-floor deployment was making the execution strategy consistent, not making the policy more capable [12].

2. Where the determinism actually lives

The architectural claim is narrow and checkable:

Given the same grounded situation, the planner returns the same plan. Always.

The reasoning engine is symbolic and has no generative capability. Variation in behaviour can therefore arise only from variation in grounded facts. Every behavioural difference between two runs is attributable to a fact that differed, and nondeterminism is confined to perception, where it can be measured, rather than distributed through procedural decision-making, where it cannot.

The operating rule that keeps this true:

The reasoning engine may ask a model anything whose answer is a fact about the world, and nothing whose answer is a decision about the procedure.

Every model interaction is logged with that classification, so the separation is an auditable property of a run rather than an assurance in a paper.

3. Architecture

Figure 1. The loop. Persistent knowledge about the skill and typed facts about the current scene both enter the reasoning engine, which issues a single action to a controller that knows nothing about the plan. The world is then checked independently, and the engine — not the controller, and not a model — decides what happens next.

The SRE is the reasoning authority, not a stage in a pipeline. It owns planning (the Chainer and Composer), Working Memory Graph (WMG is the runtime plan graph), traversal (the Walker), and repair (the Mutator). Perception and grounding sit outside it and supply facts; System 1 sits outside it and supplies motion. Figure 2 shows that decomposition.

Figure 2. The runtime. Blue components are exact and symbolic; amber components involve a model under a bounded remit; green are data. The dashed boundary marks where free text stops — nothing beyond it sees prose. Divergence at each step is judged against three independent sources: the plan’s predicted effect ①, System 1’s execution report ②, and the Grounder’s own verdict ③. Stage 1 and Stage 3 are specified but not built; removing them leaves the executing system unchanged.

PMG is persistent and reusable: mechanics, procedures with preconditions and effects, mandatory sequencing, prohibitions. Generalized, not a step map. Loaded once per episode and hashed; read-only for the duration of a run.

Situation is grounded scene fact. A large VLM describes the scene freely; a small local model coerces that description into the PMG’s declared fields and enumerations; a schema check accepts or rejects. Anything unmappable is logged rather than guessed into the pipeline. Prose stops at this boundary — nothing downstream sees free text.

Chainer plans by backward chaining with exact unification — no embeddings, no similarity scoring — asking what would make the goal true, repeatedly, until it reaches something doable now.

Composer cuts the chain into a WMG, the execution graph for this attempt, one milestone per real goal, every node tagged with its PMG origin. Built fresh, discarded at the end.

Walker has two faces. The engine-facing side owns the graph, decides the current node and evaluates exits. The System-1-facing side strips a node to its action alone — verb, actor, target, parameters, motion — through a fixed template, never a model call. System 1 never sees a milestone, an exit, a guard, or the remainder of the plan.

Grounder evaluates independently whether expected predicates hold. System 1 reporting “completed” is an execution signal, not world truth; disagreement between the two is itself a finding, and identifies silent execution failure.

Conditions and contracts. Conditions are symbolic; contracts declare how each is verified (via: detector, via: vlm). The evaluator sees normalised booleans and does not know the source. This seam absorbs rig differences — contact_made grounds via joint torque on one arm and visually on another, same condition, different contract — and it is also the edge-deployment lever, since detector grounding costs nothing while VLM grounding does.

Action parameters split by producer. Task-frame parameters come from the PMG and hold regardless of who executes, so they are safe to bake in at compose time. Scene-frame parameters — position, approach, grip — are bound late, near execution, because a coordinate fixed at compose time and used minutes later is the characteristic failure.

4. Representational scope

The PMG covers the structured-programming control constructs plus real-time constraints: sequence via states and exits; choice via exit criteria over conditions; iteration as milestone loops with collective exit conditions, where cardinality is deliberately not authored and per-iteration binding is handled by re-grounding the world each pass — the world holds the iterator; timing, where time-only conditions give wait states and time conditions on actions give deadlines; and maintenance as an action group re-entered under a guard.

Two deliberate consequences. A rig too slow to meet a deadline produces a capability report, not a representation gap. And failure in a loop attributes to the loop rather than to a particular iteration, which is the price of not authoring cardinality.

The boundary. This holds for articulable procedures — those an expert can demonstrate and narrate. Tacit, continuous, feel-based skill does not compile into a graph, and we state that limit rather than leaving it to be found.

5. Epistemic status as a first-class property

Every mechanic in a PMG carries provenance and a confidence tier. In our instantiation tier 1 is direct frame-verified observation and tier 2 is a mechanism inferred from a written account. This is not bookkeeping: during authoring, tier-2 sources asserted that contact between manipulator and element causes attachment, and frame-by-frame analysis of a 76-second recording falsified it.

Unresolved questions are carried as competing hypotheses with the experiment that would separate them, its cost, and the consequence of each answer. One example from our instantiation: an element may attach when free (H1), or when free and occupying the next unfilled slot (H2). The available footage cannot separate them, because every out-of-order element contacted was also anchored. The test — contact an out-of-order loose element, cost one undo — is carried in the graph alongside the note that the answer changes routing on every later level.

Provenance and bounded model use are the same design commitment seen from two sides. Recording where knowledge came from is only meaningful if you also control what is allowed to enter: a model permitted to make procedural decisions introduces unattributed content continuously, and no tier survives contact with it.

6. Preliminary result: a faithful failure

(Preliminary. Phase 1 — simulator correction, trace reproduction, resolution of an action ambiguity, solver re-run — gates this result and is not complete.)

We instantiated a PMG by hand for sk48, an eight-level puzzle environment from the ARC-AGI-3 public demo set, deriving mechanics from one expert recording examined frame by frame plus written accounts of three further levels.

In the run we report, the procedure was followed faithfully and the attempt failed, because an encoded mechanic was wrong. The system did not deviate from the graph, did not select an inapplicable action, and did not misexecute. It reached a state its own mechanics said was impossible, and the divergence between predicted effect and grounded observation localized the error to a specific mechanic rather than to the run as a whole.

This is the capability the architecture exists to provide. An end-to-end policy that fails produces a failed episode; it cannot say whether its world model, its perception, its ordering or its control was at fault. The decomposition here is:

The last row has a specific detection signature already specified: a grounded verdict that the world did not change, System 1 reporting success, and identical repetition at the same node, terminates with an attributed cause rather than splicing recovery indefinitely.

We regard a negative result of this shape as more informative than a success rate, and note that it is only available because the procedure is explicit and the model is out of the decision loop.

7. Relation to prior work

Symbolic planning and execution monitoring. With the reasoning engine non-generative and preconditions and effects symbolic, the Chainer’s nearest neighbour is classical planning, and we claim no novelty for it. Nor do we claim closed-loop execution as new: monitored execution of STRIPS plans dates to 1971 [16], execution monitoring is a surveyed subfield [17], replanning on divergence is standard practice [18], and contingent, conformant and fully-observable non-deterministic planning exist precisely because modelled effects are not reliable.

Three things distinguish the runtime described here, and only the third is a claim of substance.

Three sources of truth rather than two. Classical monitoring compares a predicted state against an observed one. Here a step yields a predicted effect, an execution report from System 1, and an independent grounded verdict. The disagreement pattern “System 1 reports completion, the world does not show it” identifies silent execution failure, and a two-source monitor cannot emit it.

Verification is part of the condition. Monitoring normally assumes a predicate is observable and that some perception layer supplies it. Our conditions declare their verification mechanism in a contract — torque on one rig, vision on another — which is what carries one procedure across rigs, and which makes the detector-versus-model ratio a measurable property of a run rather than an implementation detail.

Attribution rather than recovery. A replanner replans; it does not report which part of its domain was wrong. Divergence here is classified — mechanics, grounding, verification, transition, selection, execution, capability — and that classification is what a refinement layer consumes. This is also where provenance earns its place: an attributed failure needs a target that carries confidence, or there is nothing for the conclusion to update. The verification loop produces the signal; the epistemic annotation of Section 5 is what it writes to. Neither is useful alone.

Neurosymbolic systems pairing authored PDDL with VLM state acquisition [8] are closest in spirit on the grounding side.

Behavior trees. VLM-driven behavior trees evaluate free-text conditions against live images and report failures traced to those guards [7]; CABTO addresses context-aware grounding [6]; LLM-based expansion automates construction [10]. Our conditions are symbolic and our guard evaluation does not call a model, so guard misclassification is not our failure mode — but contract-level grounding can still be wrong, which is why it appears as its own row in Section 6.

Memory for embodied agents. HiMe tiers memory and retrieves by vector similarity [1]; ViReSkill stores and repairs successful plans [2]; RoboMemory [3] and MemoryVLA [4] pursue related designs. All treat stored content as uniformly reliable. None represents provenance, confidence, or open questions — which is what a refinement loop needs something to write to.

Procedure extraction from video. Task graph learning from egocentric video recovers partial orderings over keysteps and applies them to mistake detection [5, 11]. This is the closest prior work to our intended acquisition stage; the difference is what is emitted — orderings alone versus mechanics, prohibitions, alternatives, recovery and provenance. Surveys of learning from human video name as open that extracted task structures remain “loosely coupled with robot action planning” [9]; the execution path above is an attempt at that coupling.

8. Evaluation, registered

Identical models, sampling parameters and action budgets across arms.

Contamination disclosure. The PMG was authored with access to levels 1–4. Results there measure coverage of authored configurations, not generalization. Levels 5–8 are unexamined; the PMG is frozen and hashed before any level 5–8 run, and no revision in response to those outcomes precedes reporting.

Headline metric — conformance: the fraction of runs whose trace satisfies the procedure’s ordering constraints, required checkpoints and prohibitions, checked after the fact against the graph. Conformance is separable from success: a run can complete the task while skipping a required check, and under our claim that is a failure.

Second metric — attribution: for every failed run, the class assigned by Section 6’s decomposition, and whether a human reviewing the trace agrees. An attribution scheme nobody can confirm is not a capability.

Third — planner determinism: replaying a recorded Situation must reproduce the plan exactly. This is the direct test of Section 2 and it costs almost nothing to run.

Supporting measures: success rate; action-count distribution; distinct routes by action-sequence edit distance; detector-versus-VLM verification ratio and latency per condition, which doubles as an edge-feasibility estimate.

9. What is design, not result

Stated plainly, because three of these are easy to misread.

Task-universality is design. The runtime carries no task-type assumptions and we believe any PMG-expressible task runs on it, but we have evidence from one task in one environment. The measurement that would make it a demonstration — one PMG compiled from one demonstration, executed on two structurally different rigs with only the contract file differing — has not been performed.

The port is design. Contract-level portability is absorbed as specified; capability-level portability is not solved. A 5-DOF arm may need a regrasp a 7-DOF arm does not, and a human demonstrator never performed it, so a compiled PMG will not contain it. Whether capability checking lands as a compose-time gate or as runtime failure attribution is open.

Stages 1 and 3 are design. The compiler from expert video and the refinement layer are specified, not built. Stage 3’s structure is deliberately additive — logging is write-only during an episode, learned changes land in a separate overlay file with its own hash and are loaded only at the next reset, never merged into the authored PMG and never entering a running episode. Deleting the entire refinement layer leaves the executing system unchanged, which is the test that it is not load-bearing.

System 1 is not a VLA here. It converts an instruction into a discrete tool call in an environment with no continuous control. The dual-rate interface is designed so the discrete/continuous distinction does not reach the graph, but that has not been exercised against a real policy.

Further limitations: sk48’s procedure was written by us, so conformance is measurable but the standard is our own, and demonstrating conformance to an externally imposed SOP is the setting where the claim actually matters; the environment is deterministic, discrete and free of physical irreversibility; staleness policy under dual-rate execution is unresolved, sharpest inside maintenance loops; and whether a running action can be retracted mid-execution is open and belongs in the capability declaration either way.

References

(Placeholders — verify against canonical BibTeX before submission.)

[1] HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control. arXiv:2607.03449. [2] ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learning. arXiv:2509.24219. [3] RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Lifelong Learning in Physical Embodied Systems. arXiv:2508.01415. [4] MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation. arXiv:2508.19236. [5] Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos. arXiv:2406.01486. [6] CABTO: Context-Aware Behavior Tree Grounding for Robot Manipulation. AAAI. [7] VLM-driven Behavior Tree for Context-aware Task Planning. arXiv:2501.03968. [8] Neurosymbolic Embodied Agents. arXiv:2608.16794. [9] Robot Learning from Human Videos: A Survey. arXiv:2604.27621. [10] Automatic Behavior Tree Expansion with LLMs for Robotic Manipulation. arXiv:2409.13356. [11] Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videos. arXiv:2502.17753. [12] A Factory-Floor Deployment Case Study of VLA Pipelines for Industrial Packaging Task. arXiv:2605.27461. [13] Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agents. arXiv:2601.22311. [14] The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break. arXiv:2604.11978. [15] ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence. arXiv:2603.24621. [16] Fikes, R. Monitored execution of robot plans produced by STRIPS (PLANEX), 1971. [17] Fritz, C. Execution Monitoring — A Survey. University of Toronto, 2005. [18] Recent Trends in Task and Motion Planning for Robotics: A Survey. ACM Computing Surveys, 2023. doi:10.1145/3583136.

Appendix A — extended abstract / 200-word version

Deployed robot work usually already has a standard operating procedure: an order that must be held, checkpoints that must be observed, actions that must never be taken. A system that finds a better route has still failed to follow it. The requirement is not identical motion, since conditions vary, but that every execution satisfies the procedure’s invariants and that satisfaction be checkable. Determinism is needed in conformance, not in trajectory.

We place that determinism in the decision function. A Procedural Memory Graph holds persistent procedural knowledge; a symbolic engine plans over it by backward chaining against grounded facts and walks the resulting per-episode graph, handing one action at a time to an arbitrary controller. Vision-language models supply facts about the world, never decisions about the procedure, behind a boundary where prose stops. The same grounded situation yields the same plan, so behavioural differences are attributable to specific facts.

Knowledge carries provenance, confidence and unresolved hypotheses with the experiments that settle them. Failures decompose into mechanics, grounding, verification, transition, selection and execution error. We report a preliminary negative result of that form: a procedure followed faithfully that failed because an encoded mechanic was wrong.

Also on this site: The 5 Levels of Task Execution, the framework this work sits inside.