VISTA: Visual Intelligence State-Tree Architecture
A ROS 2 framework that connects on-device Vision-Language Model perception with deterministic Behavior Tree control, for local perception and reactive robot behavior.
Project: ideated and maintained by Andrea Ricciardi.
My contribution: limited to the Behavior Tree side of the system: the decision structure, custom BT nodes, blackboard integration, and the runtime link between VLM state and BT control logic.
1 Overview
VISTA uses a decoupled two-loop architecture. A slower perception loop analyzes camera frames through a local Gemma VLM served by llama.cpp; a faster Behavior Tree loop consumes structured state and selects robot behaviors without blocking on model inference.
2 System architecture
VISTA runs two ROS 2 composable nodes in a multi-threaded component container. GemmaPerceptionNode subscribes to a camera image topic, encodes frames as JPEG, calls a local OpenAI-compatible llama.cpp chat-completions endpoint, parses the JSON perception output and publishes a custom /vlm_data message. BehaviorTreeNode subscribes to that topic, updates a BehaviorTree.CPP blackboard and ticks the tree at the configured control rate.
- Camera imageConfigurable ROS 2 image topic
- GemmaPerceptionNodeJPEG encoding, prompt request, JSON parsing
- /vlm_dataScene state, hazard level, target visibility, bbox, confidence
- BT blackboardLatest VLM state copied into BehaviorTree.CPP memory
- BehaviorTreeNodeFast periodic ticks of the configured BT XML
- Robot behaviorsRecovery, target pursuit or default behavior
Runtime parameters live in one YAML file, which configures the VLM host and timeouts, the camera and output topics, JPEG quality, the system prompt, the BT tick rate and the BT XML used by the control node.
3 Behavior Tree contribution
The current tree uses a root Fallback node to encode priority-based decisions. The high-priority branch checks whether the VLM-reported hazard level meets the configured threshold and then runs RecoveryBehavior. If no hazard branch succeeds, the target branch checks target visibility and runs PursueTarget with the VLM-provided bounding box. If neither applies, DefaultBehavior runs.
| Element | Description |
|---|---|
| Custom conditions | IsHazardous and IsTargetVisible read structured VLM state from BT input ports and decide whether higher-priority branches run |
| Custom actions | RecoveryBehavior, PursueTarget and DefaultBehavior are the current action leaves. The implementation logs the selected behavior; it does not claim a full robot actuation stack |
| Blackboard bridge | The ROS subscriber callback writes scene state, hazard level, target visibility, target bounding box, confidence and description into the BT blackboard |
| Configurable XML | The tree is loaded from YAML as XML in BehaviorTree.CPP format 4, so the decision structure can be changed from configuration |
4 Runtime pipeline
A ROS 2 launch file starts both composable nodes inside a multi-threaded component container with intra-process communication. The provided tmux setup can start the local llama.cpp server, launch VISTA and optionally play a recorded rosbag.
The VLM request uses the configured system prompt to ask for structured JSON: scene state, hazard level, target visibility, target bounding box, confidence and a natural-language description. That structured message is the contract between the slow perception loop and the fast BT loop.
5 Role and credits
My work covers the Behavior Tree part of VISTA: BT architecture and implementation support. It is not authorship of the full VISTA project or of the VLM perception framework, and I do not present myself as the project author.
VISTA (Visual Intelligence State-Tree Architecture) is ideated and maintained by Andrea Ricciardi. The repository's package metadata lists Andrea Ricciardi as maintainer, and its licence and notice files attribute copyright and software development to Andrea Ricciardi. Third-party components referenced by the repository include Gemma, BehaviorTree.CPP, llama.cpp, cpp-httplib and nlohmann/json, under their respective licences.