Benchmark Your Model

Field	Value
Category	Models & Inference
Difficulty	Beginner
Estimated Read Time	5-10 minutes
Labels	benchmark, synthetic, latency, throughput, power

Chapters 001 and 002 showed how to run a model once and then how to drive it asynchronously. This chapter answers the next practical question: "How fast does this model run on the device?" The benchmark API is intentionally small. You load the model, choose how many samples to measure, call benchmark(...), and read the returned BenchmarkReport.

The benchmark uses the model's input_specs() to create deterministic synthetic inputs. That makes it useful for a quick model smoke benchmark and for comparing compiled model variants, but it is not a camera benchmark. It does not include camera decode, real preprocessing variability, dynamic input sizes, or data-dependent postprocessing behavior.

Walkthrough

Load the model

Start with the same compiled .tar.gz archive used by the earlier model tutorials. No image is needed because the benchmark creates synthetic tensors from the model's declared input specs.

Construct simaai::neat::Model from the archive path.

tutorials/003_benchmark_your_model/benchmark_your_model.cpp

simaai::neat::Model model(model_path);

Run the benchmark

Call benchmark(samples). The API warms up the model, measures single-flight latency, then measures throughput in a separate async push/pull pass, prints a summary to stdout, and returns the same headline values in a BenchmarkReport.

Each pass runs until it has measured at least samples inputs and 10 seconds have passed, so the throughput pass's power and temperature readings cover several samples. Set BenchmarkOptions::min_duration_ms to change that floor, or to 0 for a quick smoke check that stops each pass after samples inputs.

Detection models whose route ends in BoxDecode can also use BenchmarkOptions. Set original_width, original_height, and resize_mode to describe the source-image geometry that BoxDecode uses when it maps detections from model coordinates. The synthetic tensor remains model-shaped:

simaai::neat::BenchmarkOptions options;
options.num_samples = 100;
options.original_width = 1920;
options.original_height = 1080;
options.resize_mode = simaai::neat::ResizeMode::Letterbox;
auto report = model.benchmark(options);

Python exposes the same fields through pyneat.BenchmarkOptions. Set both original dimensions or omit both; when omitted, the benchmark infers geometry from the resolved model route. Per-run benchmark geometry takes precedence over deprecated BoxDecode geometry in ModelOptions.

tutorials/003_benchmark_your_model/benchmark_your_model.cpp

simaai::neat::BenchmarkOptions options;
options.num_samples = samples;
options.include_mla_layer_stats = has_flag(argc, argv, "--include-mla-layer-stats");
options.include_mla_stage_stats = has_flag(argc, argv, "--include-mla-stage-stats");
simaai::neat::BenchmarkReport report = model.benchmark(options);
if (report.latency_ms <= 0.0 || report.fps <= 0.0)
  throw std::runtime_error("benchmark produced no measured latency/fps");

Read the report

The returned report starts with the headline fields most users need: average end-to-end latency in milliseconds, throughput in frames per second, average and peak SoM power in watts, and the energy in joules needed to process samples inputs. throughput_duration_ms reports how long the throughput pass ran.

Power comes from Sentinel, the board telemetry daemon that the Neat installer sets up. report.telemetry also carries rail power, die and board temperatures, CPU utilization, and MLA memory for the throughput pass; see Measure model execution for each field. If Sentinel is not running, the benchmark still reports latency and throughput, leaves the power fields at zero, and sets report.telemetry.status to the reason.

tutorials/003_benchmark_your_model/benchmark_your_model.cpp

std::cout << "report_latency_ms=" << report.latency_ms << "\n";
std::cout << "report_fps=" << report.fps << "\n";
std::cout << "report_avg_power_watts=" << report.avg_power_watts << "\n";
std::cout << "report_energy_joules=" << report.energy_joules << "\n";
std::cout << "report_peak_power_watts=" << report.peak_power_watts << "\n";
std::cout << "report_throughput_duration_ms=" << report.throughput_duration_ms << "\n";
std::cout << "report_telemetry_status=" << report.telemetry.status << "\n";

Advanced: MLA layer and stage stats

Two opt-in options run mla-rt after the benchmark to time the compiled MLA code. Both are disabled by default and need an idle devkit with no other MLA workloads. Add --include-mla-layer-stats or --include-mla-stage-stats to either tutorial command, or enable them through the API:

simaai::neat::BenchmarkOptions options;
options.include_mla_layer_stats = true;
options.include_mla_stage_stats = true;
auto report = model.benchmark(options);

options = pyneat.BenchmarkOptions()
options.include_mla_layer_stats = True
options.include_mla_stage_stats = True
report = model.benchmark(options)

Read report.mla_layer_stats.layers for each compiled MLA layer's zero-based stage_index, name, and latency_us: the hardware run time from one profiling pass that runs that layer on its own. Zero-time no-op entries are retained. Layer latencies do not add up to the stage latency or the headline latency.

Read report.mla_stage_stats.stages for each MLA stage's stage_index and latency_us: the mean hardware time of 200 uninterrupted runs of that stage. Stage latency includes MLA computation and the program's DRAM reads and writes. It excludes model loading, host and EV74 processing, and pipeline overhead.

If mla-rt is missing, fails, or times out, benchmark() still returns the headline measurements, leaves the list empty, and sets report.mla_layer_stats.status or report.mla_stage_stats.status to the reason. Each status is "not requested" or "ok" otherwise. After an mla-rt run fails, times out, or crashes, MLA may still hold unfinished work, so later profiling in the same process is skipped with that reason. Keep the devkit idle until MLA recovery is verified.

MLA layer stats follows the selected terminal policy: it measures only the MLA stages in the selected inference route, and each keeps its stage_index in the full model.

tutorials/003_benchmark_your_model/benchmark_your_model.cpp

if (options.include_mla_layer_stats) {
  std::cout << "mla_layer_stats.status=" << report.mla_layer_stats.status << "\n";
  for (const auto& layer : report.mla_layer_stats.layers) {
    std::cout << "  stage_index=" << layer.stage_index << " name=" << layer.name
              << " latency_us=" << layer.latency_us << "\n";
  }
}
if (options.include_mla_stage_stats) {
  std::cout << "mla_stage_stats.status=" << report.mla_stage_stats.status << "\n";
  for (const auto& stage : report.mla_stage_stats.stages) {
    std::cout << "  stage_index=" << stage.stage_index << " latency_us=" << stage.latency_us
              << "\n";
  }
}

Save the report

Call report.save(path) to write the complete report as YAML or JSON, including MLA layer and stage stats when requested:

auto path = report.save("results/model/yolo_benchmark.yaml");
auto json_path = report.save("results/model/yolo_benchmark.json");

path = report.save("results/model/yolo_benchmark.yaml")
json_path = report.save("results/model/yolo_benchmark.json")

The .json extension selects JSON; .yaml and all other endings use YAML. The filename is preserved.

Give a file path including the filename. Missing parent directories are created, an existing file is replaced, and the written path is returned. Saving is explicit; benchmark() does not create a report file automatically.

For either tutorial command, append --report results/model/yolo_benchmark.yaml. Combine it with --include-mla-layer-stats or --include-mla-stage-stats to save the headline measurements together with the MLA timings.

tutorials/003_benchmark_your_model/benchmark_your_model.cpp

std::string report_path;
if (get_arg(argc, argv, "--report", report_path)) {
  std::cout << "report_path=" << report.save(report_path).string() << "\n";
}

Run

Run it and you should see the benchmark summary printed by benchmark(), followed by the same values printed from the returned report. Run the Python and C++ (prebuilt) commands from the Neat install root (the directory that contains share/ and lib/); run the build from source commands from the repo root.

C++ (prebuilt):

./lib/sima-neat/tutorials/tutorial_003_benchmark_your_model \
  --model /tmp/resnet_50.tar.gz --samples 100

C++ (build from source):

./build.sh --all --no-doc
./build/tutorials/tutorial_003_benchmark_your_model \
  --model /tmp/resnet_50.tar.gz --samples 100

Expected output (exact numbers depend on the model, board, and current load; the C++ build also prints the trailing [OK] line):

NEAT Benchmark
Input: synthetic
Samples: 100
Compiled batch: 1
Measurement detail: e2e+throughput-only
Latency: 1.97 ms
FPS:     1457.04 inferences/s
Batch throughput: 1457.04 batches/s
Throughput window: 30.09 s, 43849 inputs
Power:         13.19 W avg, 13.81 W peak (SoM)
MLA power:     3.05 W avg, 3.25 W max
Energy:        0.91 J for 100 inputs
Die temp:      46.34 °C avg, 47.56 °C max
CPU usage:     12.38 % avg, 13.22 % max
MLA memory:    29.65 MiB
Linux memory:  682.88 MiB avg, 697.41 MiB max
Telemetry:     Sentinel main:766066b74f49, 15 samples
report_latency_ms=1.96852
report_fps=1457.04
report_avg_power_watts=13.1875
report_energy_joules=0.905085
report_peak_power_watts=13.8125
report_throughput_duration_ms=30094.5
report_telemetry_status=ok
[OK] 003_benchmark_your_model

To integrate this chapter's C++ source into your own project with a custom CMakeLists.txt (no extras folder required), see How to Run Tutorials on the landing page.

In Practice

Use this benchmark when you want a quick answer for a compiled model archive: does it run, what is the measured async throughput, and what are the headline board power numbers on this target?

For application performance, benchmark the real pipeline too. Synthetic model input is deliberately stable, so it does not represent camera jitter, codec cost, real preprocessing, host scheduling under load, or downstream application logic. For queue-depth and backpressure tuning with a hand-built async run, see Tune Throughput and Queue Depth.

Model::benchmark() requires concrete input_specs() dimensions. If an input shape is dynamic or non-concrete, the benchmark fails clearly instead of guessing a shape.

Full source

Show the complete source programs

tutorials/003_benchmark_your_model/benchmark_your_model.cpp

// Benchmark a compiled model with deterministic synthetic inputs.
//
// Usage:
//   tutorial_003_benchmark_your_model --model /path/to/model.tar.gz [--samples 100]
//   [--report results/model/yolo_benchmark.yaml] [--include-mla-layer-stats]
//   [--include-mla-stage-stats]

#include "neat.h"

#include <iostream>
#include <stdexcept>
#include <string>

namespace {

bool has_flag(int argc, char** argv, const std::string& key) {
  for (int i = 1; i < argc; ++i) {
    if (key == argv[i])
      return true;
  }
  return false;
}

bool get_arg(int argc, char** argv, const std::string& key, std::string& out) {
  for (int i = 1; i + 1 < argc; ++i) {
    if (key == argv[i]) {
      out = argv[i + 1];
      return true;
    }
  }
  return false;
}

int parse_int_arg(int argc, char** argv, const std::string& key, int def) {
  std::string value;
  if (!get_arg(argc, argv, key, value))
    return def;
  return std::stoi(value);
}

} // namespace

int main(int argc, char** argv) {
  try {
    std::string model_path;
    if (!get_arg(argc, argv, "--model", model_path)) {
      std::cerr << "Usage: tutorial_003_benchmark_your_model --model <path> [--samples <n>] "
                   "[--report <file>] [--include-mla-layer-stats] [--include-mla-stage-stats]\n";
      return 1;
    }
    const int samples = parse_int_arg(argc, argv, "--samples", 100);

    // CORE LOGIC
    simaai::neat::Model model(model_path);

    simaai::neat::BenchmarkOptions options;
    options.num_samples = samples;
    options.include_mla_layer_stats = has_flag(argc, argv, "--include-mla-layer-stats");
    options.include_mla_stage_stats = has_flag(argc, argv, "--include-mla-stage-stats");
    simaai::neat::BenchmarkReport report = model.benchmark(options);
    if (report.latency_ms <= 0.0 || report.fps <= 0.0)
      throw std::runtime_error("benchmark produced no measured latency/fps");

    std::cout << "report_latency_ms=" << report.latency_ms << "\n";
    std::cout << "report_fps=" << report.fps << "\n";
    std::cout << "report_avg_power_watts=" << report.avg_power_watts << "\n";
    std::cout << "report_energy_joules=" << report.energy_joules << "\n";
    std::cout << "report_peak_power_watts=" << report.peak_power_watts << "\n";
    std::cout << "report_throughput_duration_ms=" << report.throughput_duration_ms << "\n";
    std::cout << "report_telemetry_status=" << report.telemetry.status << "\n";
    if (options.include_mla_layer_stats) {
      std::cout << "mla_layer_stats.status=" << report.mla_layer_stats.status << "\n";
      for (const auto& layer : report.mla_layer_stats.layers) {
        std::cout << "  stage_index=" << layer.stage_index << " name=" << layer.name
                  << " latency_us=" << layer.latency_us << "\n";
      }
    }
    if (options.include_mla_stage_stats) {
      std::cout << "mla_stage_stats.status=" << report.mla_stage_stats.status << "\n";
      for (const auto& stage : report.mla_stage_stats.stages) {
        std::cout << "  stage_index=" << stage.stage_index << " latency_us=" << stage.latency_us
                  << "\n";
      }
    }

    std::string report_path;
    if (get_arg(argc, argv, "--report", report_path)) {
      std::cout << "report_path=" << report.save(report_path).string() << "\n";
    }

    std::cout << "[OK] 003_benchmark_your_model\n";
    return 0;
  } catch (const std::exception& e) {
    std::cerr << "[FAIL] " << e.what() << "\n";
    return 1;
  }
}

Source

• C++
• Python
• README