The NIST AI Risk Management Framework 1.0 says AI systems should be tested before deployment and regularly in operation. Its Measure function calls for objective, repeatable, documented test, evaluation, verification, and validation processes; performance should be demonstrated under conditions similar to the deployment setting, and limits on generalizing beyond tested conditions should be documented. NIST also calls for production monitoring and management of incidents and errors.
\r\nNIST’s video analytics program emphasizes defined performance metrics, real-world datasets, and systematic evaluation. These sources do not prescribe one universal pass rate for a commercial security analytic. The acceptable balance of missed events, nuisance alerts, latency, privacy, and operator workload is a risk decision for the specific use.
\r\nDefine what the analytic is allowed to do. Is it an operator cue, a search aid, a maintenance signal, or an input to an automated action? Identify the protected area, relevant hours, subjects or objects, response owner, and consequence of a false positive and false negative. An analytic that helps a person search recorded video is not automatically suitable for denying entry, dispatching responders, or making an employment decision.
\r\nCreate a scenario matrix instead of a single accuracy target. Include expected positives, expected negatives, ambiguous cases, and deliberately out-of-scope cases. Define the unit of measurement: event, track, person, vehicle, frame, or time interval. Otherwise, two impressive percentages may describe completely different tests.
\r\nUse authorized, safety-reviewed field trials and an independent record of what actually happened. A person who did not tune the analytic should label the trial when practical. Record camera model and firmware, analytic and model version, lens and view, resolution, frame rate, compression, shutter and exposure behavior, zones, thresholds, server resources, network path, date, time, weather, and lighting. Preserve the configuration with the results.
\r\nFor every scenario, count the relevant true detections, misses, nuisance detections, duplicates, and alerts with incorrect classification. Measure time from the real event to operator presentation and then to acknowledgment or action. Report the denominator, test duration, scene, and uncertainty with every rate. Do not combine a quiet indoor test and a busy outdoor test into one number that hides failure.
\r\nReview error clusters, not only the average. A low overall nuisance rate can conceal repeated alerts every time headlights sweep a gate; a good daytime result can conceal unusable night performance. If people are identified or classified, involve privacy, legal, security, and affected operational stakeholders, and test representative conditions without claiming that a small local trial proves performance for every population.
\r\nSend alerts through the production path. Confirm the correct camera and pre-event context appear, the message arrives on the staffed console or device, the operator can distinguish live from recorded material, and the response procedure is available. Measure alert bursts and simultaneous events. Have operators explain the alert and record disposition; an accurate model that overwhelms the desk is not an accepted system.
\r\nThe acceptance record should state the tested conditions, passed and failed scenarios, known blind spots, required camera and rule settings, operator responsibilities, privacy controls, residual risk, approver, and retest triggers. Material camera movement, lighting change, construction, foliage, firmware, model, threshold, server, or integration changes should reopen the affected tests. Monitor field outcomes and maintain a simple path for users to report misses and nuisance alerts.
\r\nDSE recommends a pilot or limited-use decision when evidence is incomplete. Acceptance means the system met documented criteria in defined conditions—not that an AI analytic is infallible.
\r\n