Run GenAI Models over PCIe

Field	Value
Category	PCIe Co-Processing
Difficulty	Intermediate
Estimated Read Time	15 minutes
Labels	`PCIe`, `GenAI`, `LLM`, `VLM`, `Whisper`

Each example loads one prepared LLiMa model directory on the host, runs a bounded request, and closes the model. Run the examples separately first. They use the same connection settings but different request inputs.

Walkthrough

Load an LLM

Construct pcie::genai::GenAIModel in C++ or pyneatpcie.genai.GenAIModel in Python. Construction loads the model on the card; there is no build() or queue selection. Set connection.card_id to select the card. Absolute model paths and paths relative to the current working directory are accepted.

pcie_host/tutorials/029_run_genai_over_pcie/run_llm.cpp
genai::GenAIModel model(args.model, args.connection);
if (!model.accepts_text() || model.task() == genai::GenAITask::ASR)
  throw std::invalid_argument("choose a text-generation model directory");

Send a text request

Set prompt and max_new_tokens, then call run(). The result contains answer text and generation metrics. The examples default to 128 generated tokens; use --max-tokens to change that budget.

pcie_host/tutorials/029_run_genai_over_pcie/run_llm.cpp
genai::GenerationRequest request;
request.prompt = args.prompt;
request.max_new_tokens = args.max_tokens;
if (!args.stream) {
  const auto result = model.run(request);
  std::cout << result.text << '\n';
  std::cout << "Generated tokens: " << result.metrics.generated_tokens << '\n';
}

Stream the answer

Pass --stream to the LLM example to iterate TokenSample values instead. Print each text fragment as it arrives; the final sample carries the completed generation metrics. Only one request may be active per model.

pcie_host/tutorials/029_run_genai_over_pcie/run_llm.cpp
auto stream = model.stream(request);
for (const auto& sample : stream) {
  std::cout << sample.text << std::flush;
  if (sample.is_final)
    std::cout << "\nGenerated tokens: " << sample.metrics.generated_tokens << '\n';
}

Close the model

C++ calls close() explicitly after inference and uses destructor cleanup if an exception interrupts the example. Python uses a context manager, which closes the model when the block exits, including on errors. Both connection timeouts are finite: 15 minutes for startup and 5 minutes for a request.

pcie_host/tutorials/029_run_genai_over_pcie/run_llm.cpp
model.close();

Prepare an RGB image

The VLM example reads the image on the host with OpenCV, converts BGR to RGB, and produces contiguous UInt8 pixels with shape [H, W, 3]. C++ wraps an owning pixel vector in a PCIe tensor; Python supplies a NumPy array. Do not normalize the pixels or substitute a card-local image path.

pcie_host/tutorials/029_run_genai_over_pcie/run_vlm.cpp
const auto bgr = cv::imread(args.media, cv::IMREAD_COLOR);
if (bgr.empty())
  throw std::invalid_argument("cannot read image: " + args.media);
cv::Mat rgb;
cv::cvtColor(bgr, rgb, cv::COLOR_BGR2RGB);
if (!rgb.isContinuous())
  rgb = rgb.clone();
auto image = pcie::Tensor::from_vector(
    std::vector<std::uint8_t>(rgb.data, rgb.data + rgb.total() * rgb.elemSize()),
    {rgb.rows, rgb.cols, 3}, "", pcie::PixelFormat::RGB);

Ask about the image

Load a VLM and attach the RGB input to request.images alongside a prompt. The model handles its own image preprocessing. The returned text answers the question; there is no image-file field on the PCIe request.

pcie_host/tutorials/029_run_genai_over_pcie/run_vlm.cpp
genai::GenAIModel model(args.model, args.connection);
if (!model.accepts_image())
  throw std::invalid_argument("choose a vision-language model directory");
genai::GenerationRequest request;
request.prompt = args.prompt;
request.images = {image};
request.max_new_tokens = args.max_tokens;
std::cout << model.run(request).text << '\n';
model.close();

Transcribe speech with Whisper

Load the Whisper directory and set audio_file to a host audio path. Use language="auto" for language detection or specify a source language with --language. By default the request transcribes; --translate selects translation into English. Read the transcript, language, and optional no-speech probability. Do not add a chat prompt to an ASR request.

pcie_host/tutorials/029_run_genai_over_pcie/run_whisper.cpp
genai::GenAIModel model(args.model, args.connection);
if (model.task() != genai::GenAITask::ASR || !model.accepts_audio())
  throw std::invalid_argument("choose a Whisper model directory");
genai::GenerationRequest request;
request.audio_file = args.media;
request.language = args.language;
request.asr_task = args.translate ? genai::ASRTask::Translate : genai::ASRTask::Transcribe;
const auto result = model.run(request);
std::cout << result.text << '\n';
std::cout << "Language: " << result.language << '\n';
if (result.no_speech_prob)
  std::cout << "No-speech probability: " << *result.no_speech_prob << '\n';
model.close();

Run

Follow Install PCIe Host and Tutorial Setup. Use matching host and card packages that expose the PCIe GenAI API. Run all commands below on the host, from the extracted PCIe extras root.

Download prepared models

These SiMa.ai Model Zoo repositories already contain compiled LLiMa assets: Qwen3, LFM2-VL, and Whisper. Download only the models you plan to run. Review their model cards and use assets compatible with your installed platform.

source ~/pyneatpcie/bin/activate
python -m pip install huggingface_hub opencv-python
mkdir -p models
hf download simaai/Qwen3-0.6B-Autoround-a16w4 \
  --local-dir models/Qwen3-0.6B-Autoround-a16w4
hf download simaai/LFM2-VL-450M-Autoround-a16w4 \
  --local-dir models/LFM2-VL-450M-Autoround-a16w4
hf download simaai/whisper-small-a16w8 \
  --local-dir models/whisper-small-a16w8

Keep each complete directory, including its tokenizer, configuration, and compiled assets. No unpacking or manual copy to the card is required. A raw Hugging Face checkpoint is not a prepared LLiMa directory; for custom models, follow GenAI with LLiMa before running this tutorial.

Prepare the media

The VLM commands use the bundled street-scene image. For Whisper, record or provide a short speech clip on the host. With FFmpeg installed, convert an existing recording into a 16 kHz mono WAV:

ffmpeg -i recording.mp3 -ar 16000 -ac 1 -c:a pcm_s16le speech.wav

Do not use silence to judge transcription quality. The examples accept another image or audio path through --image or --audio.

C++ (prebuilt):

./lib/sima-pcie-host/tutorials/tutorial_029_run_llm \
  --model models/Qwen3-0.6B-Autoround-a16w4 --stream
./lib/sima-pcie-host/tutorials/tutorial_029_run_vlm \
  --model models/LFM2-VL-450M-Autoround-a16w4 \
  --image share/sima-pcie-host/tutorials/assets/street-scene.png
./lib/sima-pcie-host/tutorials/tutorial_029_run_whisper \
  --model models/whisper-small-a16w8 --audio speech.wav

C++ (build from source):

./build.sh --target tutorial_029_run_llm
./build.sh --target tutorial_029_run_vlm
./build.sh --target tutorial_029_run_whisper
./build/tutorials-standalone/tutorial_029_run_llm \
  --model models/Qwen3-0.6B-Autoround-a16w4 --stream
./build/tutorials-standalone/tutorial_029_run_vlm \
  --model models/LFM2-VL-450M-Autoround-a16w4 \
  --image share/sima-pcie-host/tutorials/assets/street-scene.png
./build/tutorials-standalone/tutorial_029_run_whisper \
  --model models/whisper-small-a16w8 --audio speech.wav

The C++ source build needs the PCIe development package and OpenCV development files. All examples default to card 0 and the connection's default SSH user, sima. Add --card N, --user USER, --ssh-key PATH, or --card-host ADDRESS when your installation differs. Use a provisioned SSH account; the examples do not configure access.

Expect an LLM answer, a description of the scene, or a transcript matching the speech recording. Wording depends on the model and input; successful execution alone is not a quality check.

In Practice

Use separate handles for independent models, subject to card memory and compute resources. For chat, supply history through messages instead of prompt; independent calls do not remember previous turns. For stream cancellation, call cancel() and consume the terminal sample before reusing the model.

See PCIe GenAI APIs for the API overview. The standalone GenAI tutorials use a different namespace and local model execution; do not substitute their model handles here.

Full source

Show the complete source programs

Llm

pcie_host/tutorials/029_run_genai_over_pcie/run_llm.cpp
// Run one text request, optionally streaming the answer, from the host over PCIe.
#include "tutorial_args.h"

namespace genai = simaai::neat::pcie::genai;

int main(int argc, char** argv) {
  try {
    const auto args = tutorial::parse_args(argc, argv, "llm");

    genai::GenAIModel model(args.model, args.connection);
    if (!model.accepts_text() || model.task() == genai::GenAITask::ASR)
      throw std::invalid_argument("choose a text-generation model directory");

    genai::GenerationRequest request;
    request.prompt = args.prompt;
    request.max_new_tokens = args.max_tokens;
    if (!args.stream) {
      const auto result = model.run(request);
      std::cout << result.text << '\n';
      std::cout << "Generated tokens: " << result.metrics.generated_tokens << '\n';
    }
    else {
      auto stream = model.stream(request);
      for (const auto& sample : stream) {
        std::cout << sample.text << std::flush;
        if (sample.is_final)
          std::cout << "\nGenerated tokens: " << sample.metrics.generated_tokens << '\n';
      }
    }
    model.close();
    return 0;
  } catch (const std::exception& error) {
    std::cerr << "LLM tutorial: " << error.what() << '\n';
    return 1;
  }
}

Vlm

pcie_host/tutorials/029_run_genai_over_pcie/run_vlm.cpp
// Load an image on the host and send RGB pixels with a question over PCIe.
#include "tutorial_args.h"

#include <opencv2/imgcodecs.hpp>
#include <opencv2/imgproc.hpp>

#include <cstdint>
#include <vector>

namespace genai = simaai::neat::pcie::genai;
namespace pcie = simaai::neat::pcie;

int main(int argc, char** argv) {
  try {
    const auto args = tutorial::parse_args(argc, argv, "vlm");

    const auto bgr = cv::imread(args.media, cv::IMREAD_COLOR);
    if (bgr.empty())
      throw std::invalid_argument("cannot read image: " + args.media);
    cv::Mat rgb;
    cv::cvtColor(bgr, rgb, cv::COLOR_BGR2RGB);
    if (!rgb.isContinuous())
      rgb = rgb.clone();
    auto image = pcie::Tensor::from_vector(
        std::vector<std::uint8_t>(rgb.data, rgb.data + rgb.total() * rgb.elemSize()),
        {rgb.rows, rgb.cols, 3}, "", pcie::PixelFormat::RGB);

    genai::GenAIModel model(args.model, args.connection);
    if (!model.accepts_image())
      throw std::invalid_argument("choose a vision-language model directory");
    genai::GenerationRequest request;
    request.prompt = args.prompt;
    request.images = {image};
    request.max_new_tokens = args.max_tokens;
    std::cout << model.run(request).text << '\n';
    model.close();
    return 0;
  } catch (const std::exception& error) {
    std::cerr << "VLM tutorial: " << error.what() << '\n';
    return 1;
  }
}

Whisper

pcie_host/tutorials/029_run_genai_over_pcie/run_whisper.cpp
// Transcribe a host audio file on the card, or translate its speech into English.
#include "tutorial_args.h"

#include <filesystem>

namespace genai = simaai::neat::pcie::genai;

int main(int argc, char** argv) {
  try {
    const auto args = tutorial::parse_args(argc, argv, "whisper");
    if (!std::filesystem::is_regular_file(args.media))
      throw std::invalid_argument("audio file does not exist: " + args.media);

    genai::GenAIModel model(args.model, args.connection);
    if (model.task() != genai::GenAITask::ASR || !model.accepts_audio())
      throw std::invalid_argument("choose a Whisper model directory");
    genai::GenerationRequest request;
    request.audio_file = args.media;
    request.language = args.language;
    request.asr_task = args.translate ? genai::ASRTask::Translate : genai::ASRTask::Transcribe;
    const auto result = model.run(request);
    std::cout << result.text << '\n';
    std::cout << "Language: " << result.language << '\n';
    if (result.no_speech_prob)
      std::cout << "No-speech probability: " << *result.no_speech_prob << '\n';
    model.close();
    return 0;
  } catch (const std::exception& error) {
    std::cerr << "Whisper tutorial: " << error.what() << '\n';
    return 1;
  }
}

Walkthrough​

Load an LLM​

Send a text request​

Stream the answer​

Close the model​

Prepare an RGB image​

Ask about the image​

Transcribe speech with Whisper​

Run​

Download prepared models​

Prepare the media​

In Practice​

Full source​

Llm​

Vlm​

Whisper​

Source​