# Make video analytics earn production trust > A demo proves that an analytic can work. Acceptance testing must show how the complete camera, scene, network, model, rules, operators, and response workflow behave in the conditions where the organization will rely on them. - Canonical URL: https://update.dsesecurity.com/updates/make-video-analytics-earn-production-trust/ - Publisher: Detection Systems & Engineering (DSE Security) - Author: DSE Security Editorial Team - Published: 2026-08-04T22:53:02+00:00 - Modified: 2026-08-04T22:53:02+00:00 - Last reviewed by DSE: 2026-08-04 - Resource type: Checklist - DSE priority: Advisory - Topics: Cybersecurity, IT, Video Surveillance - Reading time: 4 minutes ## What you need to know A demo proves that an analytic can work. Acceptance testing must show how the complete camera, scene, network, model, rules, operators, and response workflow behave in the conditions where the organization will rely on them. ## Potentially affected Organizations deploying motion classification, line crossing, intrusion, loitering, object, people, vehicle, face, license-plate, or other video analytics. ## DSE recommendation Define scenario-specific acceptance criteria, capture labeled field trials across real operating conditions, test the human response path, document limitations, and require retesting after material change. ## Article ## Source fact: test in the intended context The [NIST AI Risk Management Framework 1.0](https://doi.org/10.6028/NIST.AI.100-1) says AI systems should be tested before deployment and regularly in operation. Its Measure function calls for objective, repeatable, documented test, evaluation, verification, and validation processes; performance should be demonstrated under conditions similar to the deployment setting, and limits on generalizing beyond tested conditions should be documented. NIST also calls for production monitoring and management of incidents and errors. NIST’s [video analytics program](https://www.nist.gov/programs-projects/video-analytics) emphasizes defined performance metrics, real-world datasets, and systematic evaluation. These sources do not prescribe one universal pass rate for a commercial security analytic. The acceptable balance of missed events, nuisance alerts, latency, privacy, and operator workload is a risk decision for the specific use. ## DSE recommendation: write the decision before the test Define what the analytic is allowed to do. Is it an operator cue, a search aid, a maintenance signal, or an input to an automated action? Identify the protected area, relevant hours, subjects or objects, response owner, and consequence of a false positive and false negative. An analytic that helps a person search recorded video is not automatically suitable for denying entry, dispatching responders, or making an employment decision. Create a scenario matrix instead of a single accuracy target. Include expected positives, expected negatives, ambiguous cases, and deliberately out-of-scope cases. Define the unit of measurement: event, track, person, vehicle, frame, or time interval. Otherwise, two impressive percentages may describe completely different tests. ## Build ground truth the system did not create Use authorized, safety-reviewed field trials and an independent record of what actually happened. A person who did not tune the analytic should label the trial when practical. Record camera model and firmware, analytic and model version, lens and view, resolution, frame rate, compression, shutter and exposure behavior, zones, thresholds, server resources, network path, date, time, weather, and lighting. Preserve the configuration with the results. - Test dawn, daylight, dusk, darkness, artificial light, glare, headlamps, shadows, and wide-dynamic-range scenes that occur at the site. - Include rain, snow, fog, wind-driven movement, seasonal foliage, insects, dirty or wet covers, and camera vibration where relevant. - Vary distance, speed, direction, dwell time, occlusion, crowding, clothing, carried objects, vehicle types, and legitimate activity near rule boundaries. - Exercise bandwidth constraint, dropped video, recorder or analytic-server restart, lost camera, delayed notification, and recovery. ## Report errors operators can understand For every scenario, count the relevant true detections, misses, nuisance detections, duplicates, and alerts with incorrect classification. Measure time from the real event to operator presentation and then to acknowledgment or action. Report the denominator, test duration, scene, and uncertainty with every rate. Do not combine a quiet indoor test and a busy outdoor test into one number that hides failure. Review error clusters, not only the average. A low overall nuisance rate can conceal repeated alerts every time headlights sweep a gate; a good daytime result can conceal unusable night performance. If people are identified or classified, involve privacy, legal, security, and affected operational stakeholders, and test representative conditions without claiming that a small local trial proves performance for every population. ## Test the human and operational system Send alerts through the production path. Confirm the correct camera and pre-event context appear, the message arrives on the staffed console or device, the operator can distinguish live from recorded material, and the response procedure is available. Measure alert bursts and simultaneous events. Have operators explain the alert and record disposition; an accurate model that overwhelms the desk is not an accepted system. ## Commission with bounded claims The acceptance record should state the tested conditions, passed and failed scenarios, known blind spots, required camera and rule settings, operator responsibilities, privacy controls, residual risk, approver, and retest triggers. Material camera movement, lighting change, construction, foliage, firmware, model, threshold, server, or integration changes should reopen the affected tests. Monitor field outcomes and maintain a simple path for users to report misses and nuisance alerts. DSE recommends a pilot or limited-use decision when evidence is incomplete. Acceptance means the system met documented criteria in defined conditions—not that an AI analytic is infallible. ## Official sources - [NIST AI 100-1, AI Risk Management Framework 1.0](https://doi.org/10.6028/NIST.AI.100-1) - [NIST AI RMF Core: Measure and Manage](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/) - [NIST Video Analytics program](https://www.nist.gov/programs-projects/video-analytics) - [NIST: Test scenarios for AI must reflect real-world uses](https://www.nist.gov/news-events/news/2022/02/nist-led-panel-assesses-test-and-evaluation-industrial-ai-risk-awareness) ## Primary reference - Name: NIST AI Risk Management Framework 1.0 - Authority: doi.org - URL: https://doi.org/10.6028/NIST.AI.100-1 - Source publication date: 2023-01-26 ## Citation and use Preferred citation: “Make video analytics earn production trust,” DSE Security, https://update.dsesecurity.com/updates/make-video-analytics-earn-production-trust/ Publishing principles: https://update.dsesecurity.com/updates/dse-updates-editorial-methodology/ Usage and citation policy: https://update.dsesecurity.com/usage/ Copyright © 2026 Detection Systems & Engineering. All rights reserved.