Consume a Live RTSP Stream

Field	Value
Category	Cameras & Streaming
Difficulty	Intermediate
Estimated Read Time	5-10 minutes
Labels	rtsp, h264, h265, streaming, input-group, live-input

This is the first chapter where input originates outside the program. Earlier chapters manufactured test images or read files from disk; here, frames arrive continuously from a network stream and you consume them as fast as you pull. The mechanism is a reusable Graph fragment, RtspDecodedInput, that bundles the whole RTSP-to-raw-frames front end behind one interface.

The chapter deliberately stops at "pull the decoded frames." Feeding them into a Model is covered elsewhere (001 for a single model run, 007 for plugging a model into a pipeline, 015 for embedding a model inside a graph). By the end you will have connected to an RTSP URL and printed the tensor shape of each decoded frame — proof the stream is flowing.

This is a consumer only. To publish a stream, run a separate RTSP server (e.g. mediamtx) and point --url at it.

Walkthrough

Configure the RTSP client

RtspDecodedInputOptions configures the source and decoder. url selects the rtsp://... source. codec selects the encoded format. H.264 is the default; the tutorial also accepts avc, h265, and hevc, where AVC equals H.264 and HEVC equals H.265.

Set source_fps when you already know the source cadence. If you omit it, this tutorial opens the RTSP source with OpenCV, reads its reported FPS, and supplies the detected value to RtspDecodedInput. The group itself does not probe the URL. Only the probing path needs OpenCV, and the Python version imports it on demand, so supplying --source-fps runs without it; to probe, install it with pip install opencv-python. For H.265, Neat propagates this value into the parsed stream caps and decoder configuration. It does not change the frame rate. The H.265 stream must use HEVC Main profile, 8-bit, 4:2:0 input.

Setting tcp = true carries RTP over TCP. TCP preserves order and retransmits lost segments, which can reduce visible loss compared with UDP but can increase latency while recovering lost data.

tutorials/018_consume_rtsp_stream/consume_rtsp_stream.cpp

// Configure the URL, codec, source cadence, and RTSP transport.
simaai::neat::nodes::groups::RtspDecodedInputOptions rtsp_opt;
rtsp_opt.url = url;
rtsp_opt.codec = parse_codec(codec_name);
rtsp_opt.source_fps = source_fps;
rtsp_opt.tcp = true;

Compose the graph

Build a Graph with just two stages: the RtspDecodedInput fragment (the source) and a bare Output node (the pull endpoint). Adding the fragment is a single add(...) — it expands internally into the connect/depacketize/decode elements, so your composition stays at the level of intent. Because the input originates inside the pipeline, we call the build(RunOptions{}) overload that takes no priming sample: there is no frame to hand build() up front, since the stream produces them.

tutorials/018_consume_rtsp_stream/consume_rtsp_stream.cpp

// Build a Graph whose only stages are the RTSP group and an Output node.
simaai::neat::Graph graph;
graph.add(simaai::neat::nodes::groups::RtspDecodedInput(rtsp_opt));
graph.add(simaai::neat::nodes::Output());
auto run = graph.build(simaai::neat::RunOptions{});

Pull decoded frames

With the run live, loop and pull(...) with a timeout. Each successful pull yields a Sample whose tensor is one decoded frame. The tutorial uses the default NV12 output, represented as a logical [H, W] tensor with Y and UV plane metadata. A pull that returns nothing (or an empty tensor) prints frame=N rtsp_timeout and breaks the loop — that usually means the URL is wrong or the stream is not delivering. The timeout is what keeps a dead stream from hanging the program.

A frame is extracted with tensors_from_sample(*sample, true); the loop checks for an empty list before reading shape.

tutorials/018_consume_rtsp_stream/consume_rtsp_stream.cpp

for (int i = 0; i < frames; ++i) {
  auto sample = run.pull(/*timeout_ms=*/5000);
  if (!sample.has_value() || simaai::neat::tensors_from_sample(*sample, true).empty()) {
    std::cout << "frame=" << i << " rtsp_timeout\n";
    break;
  }
  const auto tensors = simaai::neat::tensors_from_sample(*sample, true);
  const auto& shape = tensors.front().shape;
  std::cout << "frame=" << i << " shape=[";
  for (std::size_t d = 0; d < shape.size(); ++d) {
    std::cout << shape[d] << (d + 1 < shape.size() ? ", " : "");
  }
  std::cout << "]\n";
}

Run

This chapter consumes a live RTSP stream, so you must supply a reachable --url. If you do not have a camera, publish a compatible video through an RTSP server and point --url at it. Run the Python and C++ (prebuilt) commands from the Neat install root (the directory that contains share/ and lib/); run the build from source commands from the repo root.

The automated tutorial regression runs both codecs. It reads the first usable URL from SIMANEAT_TEST_RTSP_H264_URL or SIMANEAT_TEST_RTSP_H264_URLS, and from SIMANEAT_TEST_RTSP_H265_URL or SIMANEAT_TEST_RTSP_H265_URLS. The test probes each source and supplies its detected FPS to the RTSP group.

C++ (prebuilt):

./lib/sima-neat/tutorials/tutorial_018_consume_rtsp_stream \
  --url rtsp://host:port/stream --codec h265 --source-fps 30 --frames 5

C++ (build from source):

./build.sh --target tutorial_018_consume_rtsp_stream
./build/tutorials-standalone/tutorial_018_consume_rtsp_stream \
  --url rtsp://host:port/stream --codec h265 --source-fps 30 --frames 5

Expected output (shape depends on the stream's resolution and decoder format):

frame=0 shape=[720, 1280]
frame=1 shape=[720, 1280]
frame=2 shape=[720, 1280]
frame=3 shape=[720, 1280]
frame=4 shape=[720, 1280]

If the stream is unreachable you will instead see frame=0 rtsp_timeout. To integrate this chapter's C++ source into your own project with a custom CMakeLists.txt (no extras folder required), see How to Run Tutorials on the landing page.

Full source

Show the complete source programs

tutorials/018_consume_rtsp_stream/consume_rtsp_stream.cpp

// Consume a live H.264 or H.265 RTSP stream via RtspDecodedInput.
//
// The fragment handles RTSP connect, codec-specific depacketize/parse, and
// hardware decode. This chapter is about the input fragment only.
//
// Usage:
//   tutorial_018_consume_rtsp_stream --url rtsp://host/path
//     [--codec h264|avc|h265|hevc] [--source-fps 30] [--frames 5]

#include "neat.h"
#include "nodes/groups/RtspDecodedInput.h"

#include <opencv2/videoio.hpp>

#include <cmath>
#include <exception>
#include <iostream>
#include <stdexcept>
#include <string>

namespace {

bool get_arg(int argc, char** argv, const std::string& key, std::string& out) {
  for (int i = 1; i + 1 < argc; ++i) {
    if (key == argv[i]) {
      out = argv[i + 1];
      return true;
    }
  }
  return false;
}

int parse_int_arg(int argc, char** argv, const std::string& key, int def) {
  std::string v;
  if (!get_arg(argc, argv, key, v))
    return def;
  return std::stoi(v);
}

simaai::neat::nodes::groups::RtspCodec parse_codec(const std::string& value) {
  using simaai::neat::nodes::groups::RtspCodec;
  if (value == "h264" || value == "avc")
    return RtspCodec::H264;
  if (value == "h265" || value == "hevc")
    return RtspCodec::H265;
  throw std::invalid_argument("--codec must be h264, avc, h265, or hevc");
}

int probe_source_fps(const std::string& url) {
  cv::VideoCapture capture(url);
  if (!capture.isOpened()) {
    throw std::runtime_error("failed to open RTSP source for FPS probe");
  }
  const int fps = static_cast<int>(std::lround(capture.get(cv::CAP_PROP_FPS)));
  capture.release();
  if (fps <= 0) {
    throw std::runtime_error("failed to probe a positive RTSP source FPS");
  }
  return fps;
}

} // namespace

int main(int argc, char** argv) {
  try {
    std::string url;
    if (!get_arg(argc, argv, "--url", url)) {
      std::cerr << "Usage: tutorial_018_consume_rtsp_stream --url <rtsp://...> "
                   "[--codec h264|avc|h265|hevc] [--source-fps <n>] [--frames <n>]\n";
      return 1;
    }
    std::string codec_name = "h264";
    get_arg(argc, argv, "--codec", codec_name);
    const int frames = parse_int_arg(argc, argv, "--frames", 5);
    int source_fps = parse_int_arg(argc, argv, "--source-fps", -1);
    if (source_fps != -1 && source_fps <= 0) {
      throw std::invalid_argument("--source-fps must be positive");
    }
    if (source_fps == -1) {
      source_fps = probe_source_fps(url);
    }

    // CORE LOGIC
    // Configure the URL, codec, source cadence, and RTSP transport.
    simaai::neat::nodes::groups::RtspDecodedInputOptions rtsp_opt;
    rtsp_opt.url = url;
    rtsp_opt.codec = parse_codec(codec_name);
    rtsp_opt.source_fps = source_fps;
    rtsp_opt.tcp = true;

    // Build a Graph whose only stages are the RTSP group and an Output node.
    simaai::neat::Graph graph;
    graph.add(simaai::neat::nodes::groups::RtspDecodedInput(rtsp_opt));
    graph.add(simaai::neat::nodes::Output());
    auto run = graph.build(simaai::neat::RunOptions{});

    for (int i = 0; i < frames; ++i) {
      auto sample = run.pull(/*timeout_ms=*/5000);
      if (!sample.has_value() || simaai::neat::tensors_from_sample(*sample, true).empty()) {
        std::cout << "frame=" << i << " rtsp_timeout\n";
        break;
      }
      const auto tensors = simaai::neat::tensors_from_sample(*sample, true);
      const auto& shape = tensors.front().shape;
      std::cout << "frame=" << i << " shape=[";
      for (std::size_t d = 0; d < shape.size(); ++d) {
        std::cout << shape[d] << (d + 1 < shape.size() ? ", " : "");
      }
      std::cout << "]\n";
    }

    return 0;
  } catch (const std::exception& e) {
    std::cerr << "[FAIL] " << e.what() << "\n";
    return 1;
  }
}

Source

• C++
• Python
• README