A Frontend-Backend Architecture for Tool Calls in Full-Duplex Speech Models

Abstract

Full-duplex speech-to-speech (S2S) models provide natural, low-latency conversational interaction and would benefit from the ability to use external tools and complete voice-agent tasks. We propose a frontend-backend architecture where a duplex speech-to-text frontend learns to emit a delegation token and forwards streaming ASR transcripts to a text-based backend LLM for tool calls. Tool-call results from the backend are injected back into the frontend through a lightweight prefill-and-repeat mechanism and then synthesized using streaming TTS to the user. Our approach preserves regular duplex turn-taking, interruption handling, and low-latency interaction as it requires minimal modifications to the frontend model. In tool-call evaluation, our frontend achieves 92-97% tool-call recall, and with the backend shows competitive tool-call performance, and 81.2% rejection accuracy for irrelevant calls. When equipped with a larger backend (e.g., Qwen3-235B-A22B), our system achieves competitive results on Full-Duplex-Bench-V3 compared to open and closed source models, and significantly outperforms GPT-realtime-mini and Qwen3-Omni-30B-A3B on EVA-Bench. These results demonstrate that backend delegation is an effective and modular approach for combining natural duplex speech interaction with strong agentic tool-call capabilities.

This demo uses the frontend-backend architecture described in the paper to plan a trip to Tokyo. The duplex speech-to-text frontend delegates tool calls to the LangGraph backend.

Open video directly

This demo extends the frontend-backend architecture by using OpenClaw instead of LangGraph as the backend to plan a trip to Seattle. It includes parallel tool calls, searches based on user-specified conditions, and planning a task that requires multiple tool calls.

Open video directly

Paper reference

Ke Hu, Slyne Deng, Chen Chen, Elena Rastorgueva, Edresson Casanova, Punit Kumar, Dharmendra Choudhary, Nikhil Srihari, Ameya Sunil Mahabaleshwarkar, Viet Anh Trinh, Slim Essid, Oluwatobi Olabiyi, and Zhehuai Chen. A Frontend-Backend Architecture for Tool Calls in Full-Duplex Speech Models. Submitted to ICASSP 2027.