A media company needed to verify whether television commercials aired as scheduled and played in full. Similar creative versions, incomplete playback and broadcast interruptions made identifying an appearance only part of the problem.
The solution uses multimodal AI, audio fingerprinting and temporal analysis to identify commercials, verify playback and check schedules, with supporting broadcast evidence for each finding.
Advertising verification requires identifying the correct commercial, checking when it aired and establishing whether it played in full.
Near-identical creatives make this difficult. Two advertisements can share footage and music while changing the closing offer, language or duration. Compression, channel overlays and interrupted recordings introduce further uncertainty.
The engineering scope covers media ingestion, feature extraction, candidate retrieval, exact-version verification and booking reconciliation.
The architecture separates fast recognition from detailed verification. Audio fingerprints and neural embeddings locate likely appearances. Temporal and localized checks establish identity and completeness. Structured rules determine booking outcomes, while an evidence-grounded language model supports investigation.
(opens full size in a new tab)FFmpeg decodes reference commercials and broadcast recordings into audio samples and video frames, preserving presentation timestamps.
Each reference has a creative-version ID, checksum and expected duration. Related campaign edits remain separate assets.
Overlapping processing chunks preserve appearances across chunk boundaries. Decode errors, missing footage and timestamp discontinuities are recorded so capture problems remain distinguishable from broadcast defects.
The audio pipeline applies resampling, overlapping windows, Hann windowing and Fast Fourier Transform analysis.
Spectral landmarks capture frequency relationships and relative timing. Matching fingerprints vote for likely offsets between the reference soundtrack and broadcast recording. Agreement across multiple landmarks identifies candidate appearances.
This provides an efficient search path before more expensive visual verification.
The visual pipeline uses DINOv3 through PyTorch to extract timestamped features from reference and broadcast frames.
Global frame embeddings support candidate retrieval. Local patch features support closer comparison of regions containing product details, branding or closing offers.
Preserving features across the timeline allows the verifier to examine content order and coverage. A single embedding for the whole commercial would lose information needed to detect missing sections and distinguish similar edits.
Broadcast embeddings retrieve similar reference frames from Qdrant. Matches are grouped by creative ID and implied appearance start time:
Appearance offset = broadcast timestamp − reference timestamp
Several matching frames supporting the same offset provide stronger evidence than isolated visual similarities.
Exact vector search provides a baseline for smaller libraries. HNSW indexing supports larger reference collections where approximate retrieval offers a useful latency–recall trade-off.
New commercials enter through feature extraction and indexing, without requiring a new classifier for every creative.
Candidate intervals receive denser analysis through constrained dynamic time warping.
The alignment checks whether reference content appears in the correct order while allowing bounded timing variation. Missing sections remain explicit gaps; the algorithm cannot freely stretch the sequence to conceal incomplete playback.
Verification measures audio agreement, visual correspondence, reference coverage, alignment continuity and start/end boundaries.
Closely related creatives receive localized checks around their distinguishing content.
(opens full size in a new tab)
(opens full size in a new tab)OpenCV supports frame and region comparisons. OCR provides additional evidence for offer text, prices or dates. These signals are evaluated alongside temporal and visual agreement rather than becoming independent proof of identity.
Black-frame, freeze and silence checks identify possible playback defects. Comparison with the reference prevents intentional fades or silent scenes from being reported as faults.
When the distinguishing content is missing or obscured, the system retains an unresolved version for review.
The decision layer combines complementary evidence rather than relying on a single embedding score.
Automatic confirmation requires sufficient temporal agreement, reference coverage and support for the selected creative over competing versions. Ambiguous appearances enter a review workflow with synchronized clips and highlighted discrepancies.
Reviewer corrections become versioned evaluation data. Model or threshold changes pass validation before affecting detection decisions.
Verified appearances are matched against channel, creative version, scheduled window, expected duration and count.
Stable event identifiers prevent retries and overlapping chunks from creating duplicate appearances. Media timestamps are mapped to channel wall-clock time for schedule comparisons.
Results distinguish complete delivery, partial playback, unexpected appearances and insufficient recording evidence. Each outcome retains its supporting media interval and reconciliation rule.
The analyst interface combines structured PostgreSQL queries with retrieval over booking instructions and reviewer notes.
An open-weight language model explains findings and references the relevant events and evidence clips. Application logic calculates counts and booking outcomes.
This supports questions such as “Why was this appearance classified as incomplete?” without making the language model responsible for detecting the commercial or determining delivery status.
Evaluation compares audio-only, visual-only and combined retrieval, followed by temporal verification.
Test cases include related campaign versions, shared soundtracks, overlays, compression changes, incomplete playback and recording gaps. Repeated footage is separated across tuning and testing.
The key measures are:
Sparse retrieval sampling, batched neural inference and detailed processing around candidate intervals control compute cost. Every detection retains its reference, encoder, preprocessing and decision-policy versions for reproducibility.
The solution produces an evidence record for each detected appearance: creative identity, broadcast timing, reference coverage, suspected defects and booking status.
Audio fingerprints narrow the search. Neural embeddings retrieve visual candidates. Temporal and localized verification establish what played. Deterministic reconciliation connects those findings to advertising commitments, and the AI analyst makes the evidence easier to investigate.
Next case study
Assess · Healthcare scale-up
Turning a Technology Assessment into a Board-Level Plan
A four-week technology and engineering assessment across architecture, security, infrastructure, delivery and engineering capability.
View case study: Digital HealthTechnology leadership & engineering
A 30-minute conversation about the technology decision, delivery challenge, or investment question in front of you.
