VoxBento

ON AIR

Every language,
one room.

Open-source simultaneous interpretation in the browser. Human interpreters or an AI pipeline, streamed to every listener's phone over WebRTC.

Arnav AngarkarIIIT Dharwad · GSoC 2026 with FOSSASIAOpen Tech Summit 2026

The problem

Multilingual events are hard to run.

Equipment and setupbooths, receivers, audio routing per room

Coordinationinterpreters typically hand over every 20–30 min

Costscales with every language and every room

Reachremote and smaller events often left out

What if a listener only needed a browser?

Illustration of a traditional on-site interpretation setup: a large soundproof booth, rented headsets and receivers, cable setup, and audio only. — A traditional on-site setup, drawn for an earlier talk. Many events use some of this, not all of it.

The solution

VoxBento is an open-source, browser-first interpretation platform: human or AI-assisted interpretation, on every listener's phone.

In the room

The speaker

Floor audio from a meeting feed or the venue's mixer.

Human

Interpreter in a browser booth

AI-assisted

Speech → text → translation → voice

Listener

Their own phone

Picks a language. Gets audio, and captions where enabled.

Interpretersgo live from a browser tab

Coordinatorssee every booth and choose who is on air

Listenersopen a link, no app to install

The real product

Two screens do most of the work.

Real VoxBento interpreter booth: live status, pass mic and unmute controls, the Jitsi floor feed, participants, booth audio volume and booth chat. — Interpreter booth · real screenshot
1 live status 2 pass mic 3 floor feed (Jitsi) 4 booth chat

Real VoxBento listener page on a phone for OpenTech Summit 2026: room select set to Main Stage, a source audio language select, a waiting status, and a captions mode select. — Listener · real, ots26
5 room 6 language 7 captions

TODOCapture in rehearsal: the listener page with Translated captions arriving during a live session.Replace this card before the talk.

Live captions · to capture

voxbento.com/listener/ots26

Live demo · scan once I'm live

Scan it.
Pick a language.

Open the prepared boothots26 · Main Stage
Go live as the interpreterfloor feed playing
Scan and join as a listenersecond device
Pick a languageGerman · English · Hindi
Hear the live interpretationWebRTC playback
Show captions and translationif running

~3:30. Before the talk: booth open, floor source running, floor transcription and translation started, second device on the listener page.
1 Open the ots26 booth. 2 Go live FIRST: listeners who join before the interpreter is live get no audio until they reload (issue #692). 3 Then invite the room to scan. 4 Pick German, English or Hindi. 5 Speak; point at the phone. 6 Switch captions to Translated. 7 Back to slides: "So how does that work?"
Fallback, no audio: tell people to reload; show the booth's live status and Mission Control instead.
Fallback, AI not producing output: say so plainly, skip step 6, continue.
Fallback, QR fails: read out voxbento.com/listener/ots26 (code 8GC9CF).
Never present a recording as live. If you play one, say "recorded in rehearsal".

How it works

Media, AI and control take different paths.

media (WebRTC)AI pathcontrol

Engineering challenge 1 of 2

Getting Python out of the media path.

Problem

v1 relayed interpreter audio through Python (aiortc), and listeners played HLS, which buffers whole segments before playing.

Decision

Publish WHIP straight to MediaMTX, play back over WHEP, and keep FastAPI for coordination only.

Why

WebRTC plays packets as they arrive; Python stops being a bottleneck for every listener.

Trade-off

WebRTC needs a reachable UDP port for ICE, and each listener holds a connection to MediaMTX. HLS caches through a CDN more easily.

Evidence

The migration PRs below, and the old HLS config: 1 s segments with a 3-segment buffer. Delay hasn't been benchmarked yet.

MAY 14 · v1 · #20 #23Flask + Socket.IO + aiortcPython in the audio path
MAY 24 · v2 · #56 #78WHIP into MediaMTXlisteners still on HLS
MAY 28 – JUN 5 · v3 · #81 #93 #117FastAPI + WHEP playbackaiortc, Flask and the HLS fallback removed

Startup delay — time until the first audio plays Media delay — interpreter's mic to listener's ear Interpretation delay — adds the interpreter's or AI's own lag

Engineering challenge 2 of 2

Big glossaries, small prompts.

Problem

Organisers upload thousands of terms ("FOSSASIA", "pull request"). Sending all of them with every caption segment makes LLM calls slower and costlier.

Decision

Per segment and language: keep terms that appear in the text, prefer booth over room over event entries, always keep priority ≥ 90, cap at 80.

Why

A bounded prompt keeps provider latency and token cost predictable, however large the glossary.

Trade-off

Exact and phrase matching can miss inflected forms. The local NLLB model ignores glossaries and persona entirely.

Evidence

A test with 10,000 matching terms stays at 80 entries and keeps the event loop responsive. tests/test_ai_interpretation.py · PR #888

5,000max rows per upload

this language+ entries marked "all"

in this segment+ priority ≥ 90

≤ 80terms in the prompt

AI-assisted interpretation

One pipeline, your choice of engine at each stage.

1 · Listen

Audio copy

ffmpeg reads a channel over RTSP

always on your server

2 · Transcribe

Speech to text

Deepgram, OpenAI, ElevenLabs, NVIDIA

or faster-whisper on CPU

3 · Translate

LLM or NLLB

OpenAI, Anthropic, Gemini, Groq

or NLLB via CTranslate2

4 · Speak (optional)

Text to speech

Deepgram Aura

or Supertonic via ONNX

Why configurable

Cost: pick per room, not per platform
Privacy: keep transcripts on your own server
Languages: no single provider covers every language well

What "local" means here

Each stage can run on your own server
Model weights download on first use
Listeners still need a network connection
Local NLLB ignores persona and glossary

Beyond the booth · VoxBento Local, a desktop app

Your meeting never leaves your laptop.

On-device AIWhisper for transcripts, Qwen for minutes and action items
Uses your GPUMetal on Apple Silicon, CUDA on Windows and Linux, per the download page
Every desktopmacOS via Homebrew; Windows and Linux installers on GitHub (v0.1.14)

Terminal — zsh

Beyond the booth · integrations

Built to plug into event platforms.

Eventyay integrates through invite tokens and subtitle WebSocket URLs (#333). OAuth apps can create events, booths and listener links through the API.

Eventyay your ticketing your event app your website

OAuth 2.0 appsscoped tokens, e.g. events:write
REST API v1events, rooms, booths, listener links
Signed webhooksHMAC-SHA256 on every delivery
Embeddable playerdrop it into any event page

your platform → VoxBento

# create the event, then a booth per language
POST /api/v1/events/
POST /api/v1/events/ots26/rooms/1/booths/de

# a listener link for every ticket holder
POST /api/v1/events/ots26/rooms/1/listener-token

# we call you back when a booth goes live
← session.status_changed  X-VoxBento-Signature

# known gaps: transcription routes (#507),
# GET /events/{slug} (#619)

Arnav speaking at FOSSASIA Summit in Bangkok — BANGKOK

Arnav with Mario and Saksham in Germany — GERMANY

My contribution

I designed and built VoxBento from the first commit.

Arnav Angarkar · 3rd year, IIIT Dharwad · GSoC 2026 with FOSSASIA

github.com/fossasia/voxbento

The fossasia/voxbento repository on GitHub: 1.5k stars, 113 forks, 66 contributors, 657 commits

Built in the open

A GSoC project became a community.

1.5k+stars

66contributors

113forks

657commits

Apache 2.0. Anyone can run it at their event, fork it, or send the next PR. That's the whole point.

What next

Three priorities. One invitation.

PLANNED
Organiser-friendly program audioFirst-party, guided setup for venue and broadcast feeds. Issues #746, #751.
PLANNED
Hardened security and reliabilityClose the open [SECURITY] and listener-reconnect issues, e.g. #809, #925, #692.
PLANNED
Better local AIShared model cache and a robust NLLB lifecycle. Issue #299.

Try itvoxbento.com

Contributegithub.com/fossasia/voxbento · good first issues labelled

Run it at your eventself-host with Docker Compose, or talk to me after

STILL ON AIR

Thank you.
Let's talk.

voxbento.com github.com/fossasia/voxbento arnavballincode.github.io/fossasia-journey

GitHub

github.com/arnavballincode

LinkedIn

linkedin.com/in/arnav-angarkar

Backup slides

Appendix

Full feature list
Engines and models
Technology stack
Going live safely
Lessons

Backup · full feature list

The whole platform, on one slide.

Booth

Go live from a browserWHIP, no plugins
Mic test + level meter
Active / backup handoff
Booth chat
Floor monitorself-hosted Jitsi

Listeners

QR link, pick a language
WebRTC playbackWHEP, auto-reconnect
Live + translated captions
Embeddable playerfor event sites
Join codes

AI interpretation

Floor transcription5 speech engines
LLM translationpersona + style
Glossary CSVkeeps "FOSSASIA" as FOSSASIA
AI voicecloud or local model
Transcript export

Organisers

Mission Controlall rooms, all booths
Setup wizardevent → rooms → booths
Invite links by role
OBS / ATEM / vMix ingest
YouTube relayRTMPS

Platform

REST API + OAuthEventyay integration
Webhooks
Developer apps
VoxBento Localoffline desktop app
Self-hostableDocker Compose

shown in the demonot demoed today

Backup · engines and models

Supported engines. "Local" runs on your server.

Hear

SPEECH → TEXT

Faster-Whisper localruns on CPU

OpenAIgpt-4o-transcribe · whisper-1 · realtime

Deepgramnova-2

NVIDIA ParakeetCTC · RNNT

ElevenLabsscribe v2 realtime

Translate

TEXT → TEXT

NLLB localCTranslate2 · no prompt, so no glossary

OpenAIGPT-4o · GPT-4o mini

AnthropicClaude Sonnet · Haiku

Google Gemini2.5 Flash · 2.5 Pro

GroqLlama 3.3 70B · OpenAI-compatible

Speak

TEXT → VOICE

Supertonic localONNX, on your machine

Deepgram Aura-2voices in en · de · it · nl · ja

room persona + style glossary: only matching terms reach the prompt captions and voice share one translation

Backup · technology stack

Boring, proven, open parts.

Media

WHIP · WHEP · RTSP

WebRTC

Jitsi Meet

FFmpeg

Backend

FastAPI

Python 3.13

uv

PostgreSQL

SQLite

Docker Compose

Plain ES modules on the front end. No framework, no build step for the UI.

Backup · going live safely

Fast isn't enough. Live has to be safe.

1

One voice per channel

Two interpreters live at once means listeners hear both. MediaMTX replaces the old publisher, and the booth registry decides who's active.

30 s

Tickets, not open mics

Going live redeems a single-use ticket. It expires in 30 seconds and only opens that booth's path, for the active interpreter. Tickets live in memory, so this assumes one portal replica. Built by Sanket Kokate (#796).

0

Roles from the client

The browser never says who it is. Roles come from the server session, so a modified client can't promote itself.

Backup · what I learned

Latency is an architecture decision.

No amount of frontend tuning fixes audio that takes the long way round.

Keep Python out of the interpreter's audio path.

Python coordinates. A media server moves the human audio; AI workers read a copy.

Small PRs, reviewed in the open.

621 merged across FOSSASIA. Each one taught me something about the code or the people.