Skip to main content

PCIe GenAI APIs

For runnable C++ and Python examples, follow Run GenAI Models over PCIe.

Use simaai::neat::pcie::genai in C++ or pyneatpcie.genai in Python to run LLM, VLM, and ASR models from a host machine on a connected Modalix PCIe Card.

The API follows the Direct GenAI APIs request and streaming pattern. Model files, inputs, and results travel over PCIe. Unlike vision co-processing, GenAI does not take a vision queue number or require build().

Before you start​

  • Follow Install PCIe Host, with matching host and card packages that support GenAI.
  • Ensure the connected card is configured as described during installation.
  • Prepare a LLiMa model directory on the host. Pass an absolute path or a path relative to your application's current working directory, not a card-local path. Keep the model files readable and unchanged while the model is open.

Run a text request​

This example assumes Qwen3-0.6B-Autoround-a16w4 is in ./models/ beneath your application's current working directory.

#include <simaai/neat/pcie/genai/GenAIModel.h>
#include <iostream>

int main() {
  namespace genai = simaai::neat::pcie::genai;
  genai::ConnectionOptions connection;
  connection.card_id = 0;

  genai::GenAIModel model("./models/Qwen3-0.6B-Autoround-a16w4", connection);
  genai::GenerationRequest request;
  request.prompt = "Explain PCIe in one sentence.";
  request.max_new_tokens = 64;

  std::cout << model.run(request).text << '\n';
  model.close();
}

The constructor loads the remote model. run() returns the complete result; close() releases the model resources. The Python context manager closes it on exit, including when an exception occurs. Link C++ applications with the installed SimaPCIeHost::sima_neat_pcie_host CMake target.

connection.card_id selects the card and its default address: 0 uses 10.0.0.2, 1 uses 10.0.1.2, and so on. Set connection.card_host only when your card uses a different address.

For speculative decoding, pass the parent directory of a prepared target/draft pair—for example, ./models/speculative-pair, containing target/ and draft/. Their compiled configurations identify the target and draft roles. Both models load in the same card session; no extra request option is needed. Allow card memory for both models.

For text generation, set request.reasoning_effort to "low" (the default), "medium", or "high". This is independent of request.enable_thinking, which controls whether reasoning is returned. Use matching host and card packages: GenAI protocol revision 3 rejects older peers rather than silently ignoring the requested effort. Rebuild C++ host applications after updating the packages.

Streaming, media, and multiple models​

  • Use model.stream(request) and iterate over TokenSample values to stream output. Call cancel() and consume the terminal sample before reusing the model. Only one request may be active per model.
  • For VLMs, use request.images (UInt8 HWC RGB PCIe tensors; Python also accepts NumPy arrays). Load image files in your application and convert to RGB before assigning them. For ASR, use request.audio_file or request.audio (Float32 mono samples) with request.sample_rate.
  • Supply conversation history through request.messages; separate requests do not inherit earlier history.
  • Create separate handles for independent models. GenAI models can coexist with vision models, subject to card resources; throughput is workload-dependent.

See the Direct GenAI APIs for shared request, result, and streaming concepts. Use PCIe request/tensor types instead of local Core types. LoRA switching, cached-image reuse, and GenAI graph composition are not exposed by this remote API.