執行 MIPI 相機模型

欄位	值
類別	相機與串流
難度	中級
預估閱讀時間	10-15 minutes
標籤	mipi, camera, live-input, model, ev74

本章假設相機已透過板載疊加層和 libcamera 正常運作。Neat 不會選擇 .dtbo 檔案或調整 ISP；它會在 libcamerasrc 能夠產生影像時，開始處理這些影像。在執行教學之前，請使用 硬體 MIPI 指南 和 GStreamer caps 檢查來驗證相機。

將本教學視為第二個關卡。第一個關卡是相機啟動：疊加層、驅動程式、libcamera、ISP 和精確的 caps 設定。第二個關卡是 Neat 圖：將相機影像輸入到 CVU 預處理、MLA 推論、可選的 EV74 BoxDecode，以及輸出提取。

操作指南

設定相機來源

CameraInputOptions 描述了 Neat 從 libcamerasrc 請求的來源 caps：解析度、幀率、格式以及一個可選的 libcamera 相機名稱。對於尚未公開 SiMaAI 零拷貝緩衝區的目前相機堆疊，請設定 allow_cpu_fallback = true。當您的 libcamerasrc 支援時，仍可透過 --strict-zero-copy 使用嚴格的零拷貝。

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

neat::CameraInputOptions camera;
camera.width = static_cast<std::uint32_t>(int_arg(argc, argv, "--width", 1920));
camera.height = static_cast<std::uint32_t>(int_arg(argc, argv, "--height", 1080));
camera.framerate_num = static_cast<std::uint32_t>(int_arg(argc, argv, "--fps", 30));
camera.framerate_den = 1;
camera.format = "NV12";
camera.buffer_name = "camera0";
camera.allow_cpu_fallback = !has_flag(argc, argv, "--strict-zero-copy");
std::string camera_name;
if (get_arg(argc, argv, "--camera-name", camera_name)) {
  camera.camera_name = camera_name;
}

設定模型路徑

模型將相機影像視為 NV12 影像。設定模型管理的預處理，用於色彩轉換、調整大小、正規化、量化和鑲嵌。範例將模型管理的 CVU 預處理固定到 EV74，以確保生產圖不會悄悄地變成 CPU 影像管線。使用 --decode none 時，路徑將在 MLA 處終止，並傳回原始模型張量。使用 YOLO --decode 標記時，BoxDecode 將作為模型管理的 EV74 後處理階段執行。

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

const neat::BoxDecodeType decode_type = decode_type_from_token(decode_token);
neat::Model model(model_path, model_options_for_camera(camera, decode_type));

neat::Model::RouteOptions route;
route.include_input = false;
route.include_output = true;
route.upstream_name = camera.buffer_name;
route.buffer_name = camera.buffer_name;
route.name_suffix = "_camera0";
route.advanced_execution.preprocess_target = "EV74";
if (decode_type != neat::BoxDecodeType::Unspecified) {
  route.advanced_execution.postprocess_target = "EV74";
}

組合來源擁有的圖

首先新增 CameraInput，然後使用 include_input = false 新增模型路徑。由於影像源自正在執行的管線內部，因此沒有公開的 Input 節點。include_output = true 會保留一個提取端點，用於檢測或張量。

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

neat::Graph graph("mipi_camera_model");
graph.add(neat::nodes::CameraInputWithCaptureBuffers(camera, 32));
graph.add(model.graph(route));

if (has_flag(argc, argv, "--print-backend")) {
  std::cout << graph.describe_backend(false) << "\n";
}

neat::Run run = graph.build();

提取輸出

建立圖並提取固定數量的輸出。逾時表示在 --pull-timeout-ms 之前，沒有模型輸出到達應用程式；相機可能已停止、caps 可能未協商成功，或者下游階段（例如 BoxDecode）可能受到反壓。列印張量計數和第一個張量的形狀，以便在新增應用程式邏輯之前，確認資料正在傳輸。

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

for (int i = 0; i < frames; ++i) {
  std::optional<neat::Sample> sample = run.pull(/*timeout_ms=*/pull_timeout_ms);
  if (!sample.has_value()) {
    std::cout << "frame=" << i << " output_timeout timeout_ms=" << pull_timeout_ms;
    const std::string last_error = run.last_error();
    if (!last_error.empty())
      std::cout << " last_error=" << last_error;
    std::cout << "\n";
    return 2;
  }
  const neat::TensorList tensors = neat::tensors_from_sample(*sample, true);
  std::cout << "frame=" << i << " tensors=" << tensors.size();
  if (!tensors.empty())
    std::cout << " first_shape=" << shape_string(tensors.front().shape);
  std::cout << "\n";
}

執行

直接在已設定 MIPI 攝影機的 Modalix DevKit 上執行此教學。從 Neat 安裝目錄執行預先建置的指令；從程式碼庫的根目錄執行從原始碼建置的指令。模型封存檔必須與您要求的預處理和選擇性 --decode 模式相符。

預設的拉取逾時時間為 15 秒。當您在全新硬體上收集首次執行診斷資訊時，請增加 --pull-timeout-ms。

C++ (prebuilt):

devkit$./lib/sima-neat/tutorials/tutorial_023_run_mipi_camera_model devkit$--model /路徑/到/model.tar.gz --frames 5 --decode none

對於支援「BoxDecode」路徑的 YOLO 樣式模型，請選擇一個解碼標記，例如 yolov8 或 yolov9seg：

devkit$``devkit$python3 share/sima-neat/tutorials/023_run_mipi_camera_model/run_mipi_camera_model.py \devkit$  --model /path/to/yolo.tar.gz --frames 5 --decode yolov8devkit$``

devkit$./lib/sima-neat/tutorials/tutorial_023_run_mipi_camera_model \devkit$  --model /path/to/yolo.tar.gz --frames 5 --decode yolov8

C++ (build from source):

devkit$./build.sh --target tutorial_023_run_mipi_camera_model

devkit$./build/tutorials-standalone/tutorial_023_run_mipi_camera_model \devkit$  --model /路徑/到/model.tar.gz --frames 5 --decode none

預期的輸出形狀取決於模型和解碼路徑。原始 MLA 輸出通常包含模型特定的張量：

frame=0 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
frame=1 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
frame=2 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
frame=3 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
frame=4 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
[OK] 023_run_mipi_camera_model

如果使用受支援的 BoxDecode 路徑，則輸出會變更為解碼後的檢測或分割張量。請使用張量數量和第一個形狀作為移動檢查，而不是作為通用的合約。

如果您看到 output_timeout，請使用 gst-launch-1.0 驗證相機，然後使用 --print-backend 檢查產生的後端。對於 BoxDecode 路徑，請確認模型封存檔、--decode 標記以及閾值是否與模型相符。

實務應用

當您需要檢查產生的 GStreamer 路徑時，請使用 --print-backend。 產生的路徑應包含 libcamerasrc、neatcamerabridge（當啟用回退時）、neatprocesscvu、neatprocessmla、可選的 EV74 後處理，以及 appsink。 除非您有意新增僅用於除錯的路徑，否則它不應包含 appsrc、ostosima、videoconvert 或 videoscale。

完整原始碼

顯示完整原始碼程式

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

// Run a model from a MIPI/libcamera camera source.
//
// Usage:
//   tutorial_023_run_mipi_camera_model --model /path/to/model.tar.gz [--frames 5]

#include <neat.h>

#include <algorithm>
#include <cctype>
#include <cstdint>
#include <filesystem>
#include <iostream>
#include <optional>
#include <stdexcept>
#include <string>
#include <vector>

namespace neat = simaai::neat;
namespace fs = std::filesystem;

namespace {

bool get_arg(int argc, char** argv, const std::string& key, std::string& out) {
  for (int i = 1; i + 1 < argc; ++i) {
    if (key == argv[i]) {
      out = argv[i + 1];
      return true;
    }
  }
  return false;
}

bool has_flag(int argc, char** argv, const std::string& key) {
  for (int i = 1; i < argc; ++i) {
    if (key == argv[i])
      return true;
  }
  return false;
}

int int_arg(int argc, char** argv, const std::string& key, int def) {
  std::string value;
  if (!get_arg(argc, argv, key, value))
    return def;
  return std::stoi(value);
}

std::string lower_copy(std::string value) {
  std::transform(value.begin(), value.end(), value.begin(),
                 [](unsigned char c) { return static_cast<char>(std::tolower(c)); });
  return value;
}

neat::BoxDecodeType decode_type_from_token(const std::string& token) {
  const std::string v = lower_copy(token);
  if (v.empty() || v == "none" || v == "raw")
    return neat::BoxDecodeType::Unspecified;
  if (v == "yolo")
    return neat::BoxDecodeType::Yolo;
  if (v == "yolov5")
    return neat::BoxDecodeType::YoloV5;
  if (v == "yolov8")
    return neat::BoxDecodeType::YoloV8;
  if (v == "yolov8seg" || v == "yolov8-seg")
    return neat::BoxDecodeType::YoloV8Seg;
  if (v == "yolov9")
    return neat::BoxDecodeType::YoloV9;
  if (v == "yolov9seg" || v == "yolov9-seg")
    return neat::BoxDecodeType::YoloV9Seg;
  throw std::runtime_error("unsupported --decode token: " + token);
}

template <typename Shape> std::string shape_string(const Shape& shape) {
  std::string out = "[";
  for (std::size_t i = 0; i < shape.size(); ++i) {
    out += std::to_string(shape[i]);
    if (i + 1 < shape.size())
      out += ",";
  }
  out += "]";
  return out;
}

neat::Model::Options model_options_for_camera(const neat::CameraInputOptions& camera,
                                              neat::BoxDecodeType decode_type) {
  neat::Model::Options options;
  options.preprocess.kind = neat::InputKind::Image;
  options.preprocess.input_max_width = static_cast<int>(camera.width);
  options.preprocess.input_max_height = static_cast<int>(camera.height);
  options.preprocess.input_max_depth = 3;
  options.preprocess.color_convert.input_format = neat::PreprocessColorFormat::NV12;
  options.preprocess.color_convert.output_format = neat::PreprocessColorFormat::RGB;
  options.preprocess.resize.enable = neat::AutoFlag::On;
  options.preprocess.resize.width = 640;
  options.preprocess.resize.height = 640;
  options.preprocess.resize.mode = neat::ResizeMode::Letterbox;
  options.preprocess.resize.pad_value = 114;
  options.preprocess.preset = neat::NormalizePreset::COCO_YOLO;
  options.advanced_execution.preprocess_target = "EV74";
  options.decode_type = decode_type;
  if (decode_type == neat::BoxDecodeType::Unspecified) {
    options.inference_terminal.mla_only = true;
  } else {
    options.advanced_execution.postprocess_target = "EV74";
    options.score_threshold = 0.25f;
    options.nms_iou_threshold = 0.45f;
    options.top_k = 100;
  }
  return options;
}

void usage(const char* argv0) {
  std::cerr << "Usage: " << argv0
            << " --model <model.tar.gz> [--frames 5] [--width 1920] [--height 1080] "
               "[--fps 30] [--camera-name NAME] [--decode none|yolov8|yolov9seg] "
               "[--pull-timeout-ms 15000] [--strict-zero-copy] [--print-backend]\n";
}

} // namespace

int main(int argc, char** argv) {
  try {
    std::string model_path;
    if (!get_arg(argc, argv, "--model", model_path)) {
      usage(argv[0]);
      return 1;
    }
    if (!fs::exists(model_path))
      throw std::runtime_error("model archive not found: " + model_path);

    const int frames = int_arg(argc, argv, "--frames", 5);
    if (frames <= 0)
      throw std::runtime_error("--frames must be positive");
    const int pull_timeout_ms = int_arg(argc, argv, "--pull-timeout-ms", 15000);
    if (pull_timeout_ms <= 0)
      throw std::runtime_error("--pull-timeout-ms must be positive");

    std::string decode_token = "none";
    get_arg(argc, argv, "--decode", decode_token);

    // CORE LOGIC
    neat::CameraInputOptions camera;
    camera.width = static_cast<std::uint32_t>(int_arg(argc, argv, "--width", 1920));
    camera.height = static_cast<std::uint32_t>(int_arg(argc, argv, "--height", 1080));
    camera.framerate_num = static_cast<std::uint32_t>(int_arg(argc, argv, "--fps", 30));
    camera.framerate_den = 1;
    camera.format = "NV12";
    camera.buffer_name = "camera0";
    camera.allow_cpu_fallback = !has_flag(argc, argv, "--strict-zero-copy");
    std::string camera_name;
    if (get_arg(argc, argv, "--camera-name", camera_name)) {
      camera.camera_name = camera_name;
    }

    const neat::BoxDecodeType decode_type = decode_type_from_token(decode_token);
    neat::Model model(model_path, model_options_for_camera(camera, decode_type));

    neat::Model::RouteOptions route;
    route.include_input = false;
    route.include_output = true;
    route.upstream_name = camera.buffer_name;
    route.buffer_name = camera.buffer_name;
    route.name_suffix = "_camera0";
    route.advanced_execution.preprocess_target = "EV74";
    if (decode_type != neat::BoxDecodeType::Unspecified) {
      route.advanced_execution.postprocess_target = "EV74";
    }

    neat::Graph graph("mipi_camera_model");
    graph.add(neat::nodes::CameraInputWithCaptureBuffers(camera, 32));
    graph.add(model.graph(route));

    if (has_flag(argc, argv, "--print-backend")) {
      std::cout << graph.describe_backend(false) << "\n";
    }

    neat::Run run = graph.build();

    for (int i = 0; i < frames; ++i) {
      std::optional<neat::Sample> sample = run.pull(/*timeout_ms=*/pull_timeout_ms);
      if (!sample.has_value()) {
        std::cout << "frame=" << i << " output_timeout timeout_ms=" << pull_timeout_ms;
        const std::string last_error = run.last_error();
        if (!last_error.empty())
          std::cout << " last_error=" << last_error;
        std::cout << "\n";
        return 2;
      }
      const neat::TensorList tensors = neat::tensors_from_sample(*sample, true);
      std::cout << "frame=" << i << " tensors=" << tensors.size();
      if (!tensors.empty())
        std::cout << " first_shape=" << shape_string(tensors.front().shape);
      std::cout << "\n";
    }

    std::cout << "[OK] 023_run_mipi_camera_model\n";
    return 0;
  } catch (const std::exception& e) {
    std::cerr << "[FAIL] " << e.what() << "\n";
    return 1;
  }
}

來源

• C++
• Python
• README