Google Summer of Code 2026 · FOSSASIA
Interpretation without the distance.
Traditional interpretation needs booths, specialist hardware and tricky media routing. VoxBento moves the interpreter booth and the listener into the browser, while keeping a direct media path built for responsive audio.
Architecture
The one rule: the live audio path stays out of Python. MediaMTX transports audio, FastAPI coordinates people, state and processing, and Jitsi handles floor monitoring.
Live audio path
Direct media path: Python coordinates it but never touches audio bytes.
Interpretation pipeline
Captions, translations and synthesized speech are pushed to clients over WebSocket.
Interpreter to listener
01 · Publish
Interpreter microphone
The interpreter booth opens a browser RTCPeerConnection, adds the microphone track and publishes over WebRTC/WHIP.
02 · Transport
MediaMTX
MediaMTX receives the WHIP stream, serves it to listeners over WHEP, and exposes an RTSP feed for the processing workers.
03 · Listen
Listener browser
Listeners pick a language and open a WHEP session, with an HLS fallback when WebRTC cannot connect.
Deep dive
Speech-to-text: a worker per active booth
Each enabled booth gets a worker that reads the MediaMTX RTSP stream through FFmpeg. It supports local faster-whisper and cloud providers (OpenAI, Deepgram, NVIDIA Riva, ElevenLabs): Deepgram streams, the rest process short chunks. Workers keep per-booth session state, track gaps and latency, retry with exponential backoff, and broadcast partial and final captions over WebSocket.
Translation: only when someone is listening
Partial transcripts feed live captions; final segments become durable records and trigger translation. Translation is lazy, running only for languages with interested listeners, and skipped when source and target match. Backends include OpenAI, Anthropic, Gemini, OpenRouter and a local fallback, with per-language semaphores so a busy room cannot overwhelm a provider.
Text-to-speech: optional spoken interpretation
Translated segments can be synthesized server-side with Deepgram Aura or a local Supertonic ONNX model, normalized to 24 kHz mono PCM, and sent with their text in a binary WebSocket bundle for scheduled playback in the browser.
State: fast when transient, durable when it matters
Active interpreter status, chat and booth presence live in an in-memory BoothRegistry guarded by per-booth asyncio locks. Events, rooms, booths, memberships, transcripts and webhook deliveries persist through SQLAlchemy and Alembic (SQLite in development, PostgreSQL in production).
Interpreter handoff without dropping listeners
A coordinator switches the active interpreter over WebSocket. The old WHIP publisher disconnects, MediaMTX accepts the new one, and listeners keep the same endpoint.
Access and deployment
JWT-backed roles separate super admins, event admins, coordinators, interpreters and listeners, checked in both routes and WebSockets; single-use invite tokens give time-bounded booth access. Docker Compose runs the FastAPI portal, MediaMTX, an optional Chromium floor bot and self-hosted Jitsi, with API keys encrypted at rest using Fernet.
Work across FOSSASIA
VoxBento
Real-time interpretation infrastructure
WHIP publishing, WHEP delivery, RTSP worker feeds, FastAPI coordination, live captions, translation, TTS, auth and role-aware booths.
Eventyay Business
Pricing, entitlements and billing foundations
Built the plugin foundation, versioned tiers and pricing, capability entitlements, subscriptions, and idempotent usage records; extended it with invoice calculation, Stripe collection, reconciliation and guarded billing rollout. See the final-report draft for upstream evidence and remaining validation.
Eventyay core
Images, imports and exports
Image uploads validated, EXIF-transposed, stripped of metadata and resized with LANCZOS resampling; schedule and speaker import paths and export reliability.
Star-Me
Repository discovery automation
Fixed the hanging behaviour and broken selectors in the browser script that walks FOSSASIA repositories and helps users star them.