OneStreamer Unifies Real-Time Perception, Streaming Memory, and Proactive Response in Video LLMs

Background and the Problem

Streaming video Large Language Models (LLMs) deployed on embodied robots, smart glasses, and surveillance systems must ingest continuous video streams in real time. They face a daunting operational dilemma: the agent cannot anticipate future queries, requiring continuous factual evidence logging without choking on computational costs. Existing solutions either retain short sliding windows (causing severe historical amnesia) or store high-dimensional visual latent caches, which induces out-of-memory crashes over extended multi-hour horizons.

Architecture and How It Works

Researchers from Nanjing University and Shanghai AI Lab introduce OneStreamer (arXiv:2610.01762), unifying streaming perception, memory, and proactive response:

1. Proactive Hierarchical Caption Memory (PHCM): Replaces heavy visual feature caches with proactive, time-grounded textual captions and event summaries generated dynamically. At inference, cached text records pair seamlessly with a localized recent visual window, enabling unbounded factual recall with negligible memory footprints.
2. Proactive State Transition Learning (PSTL): Solves the imbalance where uninformative 'idle/waiting' frames dilute optimization gradients. By selectively supervising meaningful state-change transitions, PSTL achieves superior proactive response timing while supervising only 27.5% of total annotated tokens.
3. OneStreamer-1M Dataset: Releases an industrial-scale dataset comprising over 1,000,000 high-fidelity streaming video interaction traces aligned with rigorous temporal timestamps.
4. Integrated End-to-End Architecture: Concurrently handles sensory perception, self-directed memory consolidation, and proactive conversational interventions within a single unified model.

Benchmarks and Measured Results

Benchmarked across eight demanding streaming video understanding suites:

1. 4B Architecture Captures SOTA Across All 8 Benchmarks: The compact 4B parameter OneStreamer checkpoint secures top-ranking performance across all eight evaluated streaming video benchmarks, outperforming substantially larger multimodal baselines.
2. 34.2% Boost in Historical Recall: PHCM improves long-range historical query accuracy by 34.2% without degrading real-time low-latency situational perception.
3. Ultra-Efficient Supervision: PSTL outperforms dense frame supervision while utilizing only 27.5% of the supervisory budget, accelerating training convergence.

Getting Started for Developers

OneStreamer code, pretrained 4B weights, and the OneStreamer-1M corpus are publicly released. Engineering teams developing smart glasses, drone monitoring agents, and automotive cockpit assistants can deploy OneStreamer on edge hardware to deliver continuous, low-latency proactive video interaction without maintaining heavy external vector databases.