Why a 10 TOPS NPU Does Not Guarantee High FPS¶

TOPS is a theoretical operation rate under particular precision and workload assumptions. Product frame rate is an end-to-end property of the whole data path.

TOPS is not end-to-end FPS¶

Camera → colour conversion / resize → tensor preparation → NPU inference
       → decode / NMS → OSD, display, encoding, or reporting

Even with 10 ms inference, another 30 ms in capture, preprocessing, buffer synchronization, postprocessing, and display yields 40 ms end-to-end—about 25 FPS. Time every stage rather than recording only inference time.

A model may not run entirely on the NPU¶

Unsupported or slow operators, dynamic shapes, incompatible quantization, and pre/post-processing can fall back to the CPU. Detection backbones may execute on an NPU while decoding, coordinate transforms, filtering, and NMS remain on the CPU. Confirm operator placement and fallbacks with runtime logs or profiling.

Moving data can cost more than computing¶

Camera output such as NV12, YUYV, or MJPEG commonly needs conversion to a model's RGB/BGR format, size, and layout. A path through application, conversion, NPU, and display/encoder buffers repeatedly reads and writes DDR.

DMA-BUF, buffer reuse, and fewer format conversions reduce copies. Zero-copy still has costs: cache synchronization, stride and format alignment, fences, and buffer lifetime management.

Concurrency magnifies bottlenecks¶

In isolation, camera, NPU, GPU, VPU, and display do not compete much. In a product they contend for DDR and the SoC interconnect. CPU and NPU utilization can look modest while FPS stalls because the real wait is memory bandwidth, synchronization, or arbitration.

A higher TOPS number is not always the answer. Fewer copies, hardware-friendly formats and quantization, smaller input, lower CPU postprocessing, and deliberate buffer reuse often improve the actual bottleneck.