Speech Event Detector: Surround Sound

Core Concept

Beyond Plain Words: Rich Speech Transcription

Standard ASR produces words but strips away non-verbal speaker signals and environmental soundscapes. Surround Sound captures the missing acoustic context and weaves it directly into the transcript with exact word-level alignment.

Standard ASR (Whisper)

"Hey there, can you tell me where I am going? So I want to understand the road was blocked okay."

× Non-verbal speaker vocalizations (coughs, breaths) and surrounding environment (horns, traffic) are completely lost.

✨ Surround Sound (Rich Transcript)

"Hey [cough 0.68-2.00] there, can you tell me where I am going? [breath] So I want to understand <horn> the road was blocked </horn> okay."

✓ [tag]: Speaker sounds inserted inline at start timestamp.
✓ <tag>...</tag>: Ambient background sounds wrapping the exact words they overlap.

Real Alignment in Action (Waveform, Mel-Spectrogram & Merged Tags)

An illustrative recording showing how speaker vocalizations and ambient environmental noise are detected and aligned around the Whisper transcript words without disrupting recognizer word order.

Rich Speech Event Detection Alignment Examples

Tap to enlarge

Sample Inference

Audio Samples & Event Detections

Listen to sample audio clips and inspect the exact timestamped sound events, classification categories, and probability scores detected by Surround Sound (without ASR).

Speech Event Outputs 4 events detected

0.00s – 4.20s

animal

p=0.32

0.00s – 5.68s

music

p=0.93

0.00s – 5.68s

tv_speaker

p=0.63

2.64s – 4.12s

breath

p=0.65

Speech Event Outputs 2 events detected

0.00s – 5.76s

bird

p=0.94

0.76s – 2.88s

breath

p=0.76

Speech Event Outputs 2 events detected

0.00s – 2.16s

bird

p=0.95

0.00s – 2.16s

child_voice

p=0.73

Speech Event Outputs 2 events detected

0.00s – 2.32s

baby_cry

p=0.98

0.00s – 2.32s

child_voice

p=0.86

Speech Event Outputs 5 events detected

0.00s – 6.72s

fan

p=0.74

0.00s – 6.72s

music

p=0.72

1.04s – 1.76s

breath

p=0.71

3.16s – 3.72s

breath

p=0.71

4.84s – 5.56s

breath

p=0.76

Speech Event Outputs 6 events detected

0.00s – 4.68s

animal

p=0.56

0.00s – 9.96s

vehicle

p=0.91

0.00s – 9.96s

wind

p=0.90

2.00s – 9.96s

water

p=0.50

4.92s – 7.16s

breath

p=0.69

6.72s – 8.28s

animal

p=0.50

Speech Event Outputs 1 event detected

0.00s – 10.04s

music

p=1.00

Speech Event Outputs 2 events detected

0.00s – 5.08s

horn

p=0.93

0.00s – 5.08s

insect

p=0.92

Model Architecture

Network Architecture (Figure 1 from Paper)

Architecture with tensor shapes measured with forward hooks on the released model. A 64-band log-mel spectrogram is downsampled by five convolutional blocks into 201 time steps (40 ms stride), contextualized by six Transformer encoder layers, and mapped to 54 independent class probabilities.

End-to-End Network Pipeline & Tensor Shapes

Figure 1 (Main)

Audio → Spectrogram: 8s waveform (128,000 samples at 16 kHz) converted via STFT 512 (10 ms hop) into a 64-band log-mel spectrogram (1 × 64 × 801).
5 Conv Blocks: Proportional 3D volumes shrink frequency (64 → 32 → 16 → 8 → 4) and time (801 → 801 → 801 → 401 → 201), producing 192 × 4 = 768 features per 40 ms step.
Transformer & Output: Linear projection to 256 dimensions with sinusoidal position encodings → six Transformer encoder layers (8 heads, d_ff=1024, GELU) → linear sigmoid layer outputting 201 × 54 class probabilities.

Speech Event Detector: End-to-End Network Architecture and Tensor Shapes

Tap to enlarge

1. Front-End Spectrogram

64 Log-Mel Bins • 801 Steps

STFT 512, 25 ms window, 10 ms hop, 50–8000 Hz, compressed as 10log10(x+1e-9) and scaled to [-1, 3].

2. Five Conv Blocks

866K Params (15% of model)

Channels: 48 → 48 → 96 → 96 → 192. Strides shrink frequency 64→4 and time 801→201. Each step draws on ~0.33s receptive field.

3. Transformer Encoder

6 Pre-Norm Layers (82%)

d_model=256, 8-head self-attention, d_ff=1024, GELU, dropout 0.1. Full 8s global acoustic context per frame.

4. Classification Head

54 Sigmoid Binary Heads (<1%)

Linear projection from 256 → 54 classes. Produces an independent frame-level probability track on a 40 ms grid.

(a) Training Pipeline, (b) Streaming Inference with ASR & (c) Inside the Blocks

Figure 1 (Modules)

(a) Training: Multi-task loss combining frame-level BCE and clip-level BCE (0.25×) over real speech windows, synthetic mixtures, and acoustic augmentations (noise, reverb, telephony bandpass).
(b) Inference: Streaming audio of any length split into 8s windows (2s hop), passed through per-class hysteresis and median thresholding, and integrated with timestamped Whisper ASR tokens for rich transcription.
(c) Inside the Blocks: Detailed schematics of the 5 Conv blocks (2 × [conv 3×3, BatchNorm, ReLU] with residual shortcuts and frequency/time strides) and the 6 pre-norm Transformer encoder layers (LayerNorm, 8-head self-attention, LayerNorm, GELU feed-forward 256→1024→256).

Speech Event Detector: Training Pipeline, Inference Flow, and Block Schematics

Tap to enlarge

Layer-by-Layer Network Specifications (Table 2 from Paper)

Tensor shapes measured with forward hooks on the released model for one 8s window. Parameter counts are exact.

Layer	Operation	Output Tensor Shape	Parameters
`input`	waveform, 16 kHz, 8 s window	`128,000`	—
`log-mel`	STFT 512 / win 400 / hop 160, 64 mel bands, log scale	`64 × 801`	—
`conv block 1`	2 × (conv 3×3, BN, ReLU); 1→48 ch; stride (f,t)=(1, 1)	`48 × 64 × 801`	21,456
`conv block 2`	2 × (conv 3×3, BN, ReLU); 48→48 ch; stride (f,t)=(2, 1)	`48 × 32 × 801`	41,760
`conv block 3`	2 × (conv 3×3, BN, ReLU); 48→96 ch; stride (f,t)=(2, 1)	`96 × 16 × 801`	124,992
`conv block 4`	2 × (conv 3×3, BN, ReLU); 96→96 ch; stride (f,t)=(2, 2)	`96 × 8 × 401`	166,464
`conv block 5`	2 × (conv 3×3, BN, ReLU); 96→192 ch; stride (f,t)=(2, 2)	`192 × 4 × 201`	498,816
`reshape`	(C × F) per 40 ms step: 192 × 4 = 768	`201 × 768`	—
`projection`	linear 768 → 256, + sinusoidal positions	`201 × 256`	196,864
`encoder layers 1–6`	6 pre-norm Transformer layers (8 heads, FF 1024, GELU)	`201 × 256`	4,738,560
`head`	linear 256 → 54, sigmoid	`201 × 54`	13,878
Total Trainable Parameters (Conv: 853,488 • Proj: 196,864 • Transformer: 4,738,560 • Head: 13,878)			5,802,790

Layer Activation Maps & Signal Flow

A held-out clip passing through the network: convolutional activations (mean absolute activation over channels), standardized 256-d vector activations across Transformer layers, and the final 54-class probability output tracks.

Signal flow through convolutional blocks and Transformer encoder

Tap to enlarge

Parameter Distribution by Component

82% in 6 Transformer encoder layers (789,760 each), 15% in convolutional blocks, 3% in feature projection, <1% in output head.

Training Dynamics & Validation AP

Trained in ~3 hours on a single NVIDIA T4 GPU (~1,095s/epoch). Selected checkpoint from Epoch 6 achieved highest validation AP (0.287).

Interactive Sound Taxonomy

Explore All 54 Sound Classes

Surround Sound recognizes 20 speaker vocalizations and 34 background ambient noises. Filter by category, reliability, or search for any sound to view its cross-validated performance metrics.

Data Engine

Multi-Dataset Unification & CTC Forced Alignment

Trained on material drawn from seven public corpora whose labels arrived in incompatible formats: bounding boxes, free-text tags, transcript symbols, and environmental noise banks.

Dataset Source	Size / Duration	Label Format & Role	Held-Out Evaluation Split	License
NonVerbalSpeech-38K	38,718 clips • 130.6h	Region-level non-verbal events; crops, beds, real windows	10% by stable clip hash (3,828 clips)	CC BY-NC 4.0
Vaani Noise Events	90,637 clips • 154.6h	319 free-text noise tags with timestamps (58 Indian langs)	14 districts (verified clips only)	CC BY 4.0
NonverbalTTS	6,256 clips • 17.6h	8,620 transcript-embedded tags → aligned via CTC	Official dev + test (405 clips)	CC BY-NC-SA 4.0
AudioSet-Strong	93,609 clips • 260h	Strong ontology labels with millisecond start/end	Official evaluation split	Public Domain / Mirror
OpenSLR 99	727 clips • 410 spkrs	Isolated phone-recorded vocalizations (trimmed crops)	15% of speakers	CC BY-NC-ND 4.0
DNC Home Recordings	4,377 recordings	Ambient noise crops (TV, music, appliances)	10% of sessions	MIT
DEMAND	18 environments	Multi-channel real ambient sound pool	Last 20% of each recording	CC BY 4.0

Recovering Timestamps via CTC Forced Alignment

Many valuable speech datasets (like NonverbalTTS) mark non-verbal sounds inside written transcripts (e.g., "I wanted to [laughter] tell you...") without giving timestamps. We used a Wav2Vec2 CTC forced aligner to locate the boundary of surrounding words, recovering exact timestamps for 6,360 out of 8,620 sounds.

Distribution of event crops across training sources

Evaluation

Held-Out Evaluation & Operating Profiles

Evaluated at the event level on held-out recordings across four datasets. Surround Sound provides two operating threshold profiles: Sensitive (S) tuned for F2 (prioritizing recall for human review) and Balanced (B) tuned for F1.

Classes	Count	Held-Out Events	Macro Recall		Macro Precision		Macro F1		Micro Recall (Event-Weighted)
Classes	Count	Held-Out Events	Sensitive	Balanced	Sensitive	Balanced	Sensitive	Balanced	Sensitive	Balanced
Speaker Sounds	16	3,271	0.455	0.327	0.383	0.395	0.378	0.338	70.3%	54.2%
Background Sounds	32	6,096	0.510	0.390	0.261	0.345	0.294	0.325	65.6%	52.8%
All Evaluated Classes	48	9,367	0.491	0.369	0.302	0.361	0.322	0.329	67.2%	53.3%

Event Evaluation by Dataset

Threshold Profile Tuning (F1 vs F2)

Per-Class Average Precision (AP)

Quickstart

Get Started in 3 Lines of Code

Surround Sound works out of the box with PyTorch and any Whisper backend (OpenAI Whisper or Hugging Face Transformers).

from sed import Detector, load_audio, rich_transcript, clean

# Load model weights & threshold profile directly from Hugging Face Hub
det = Detector.from_pretrained("theboringai-work/surround-sound", profile="sensitive")

# 1. Detect timestamped events in any audio file (ffmpeg supported)
events = det.detect("call.mp3")
print("Detected Events:", events)
# [{'label': 'cough', 'start': 0.68, 'end': 2.0, 'prob': 0.92}, ...]

# 2. Extract raw 40 ms probability tracks
times, probs = det.frame_probs(load_audio("call.mp3"))

# 3. Seamlessly merge with word timestamps from any Whisper ASR
# words = [(word, start_sec, end_sec), ...]
text = rich_transcript(words, clean(words, events))
print("Rich Transcript:", text)
# 'Hey [cough] there, can you tell me where I am going? <horn> the road was blocked </horn> okay.'

# Transcript with inline speaker tags + environmental noise + timestamps
sed-infer call.mp3

# Run events only on an entire directory without ASR (JSON Lines output)
sed-infer recordings/ --asr none --json events.jsonl

# Customize operating profile, filter unwanted tags, or override thresholds
sed-infer call.wav --profile balanced --hide breath,lip_smack --times

# Install prerequisites
brew install ffmpeg      # macOS (or: sudo apt install ffmpeg on Ubuntu)

# Install surround-sound and inference dependencies
pip install git+https://github.com/theboringai-work/surround-sound.git

# Optional: install whisper for transcript generation
pip install openai-whisper

Citation

BibTeX

BibTeX Citation

@article{bar2026speecheventdetector,
  title   = {Speech Event Detector: Timestamped Detection of Non-Verbal Vocalizations and Background Sounds for Rich Speech Transcription},
  author  = {Bar, Ayush Kumar and Ghosh, Promila},
  journal = {arXiv preprint arXiv:2610.14232},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.14232}
}