Skip to main content

Direct GenAI APIs

Use Neat's direct GenAI APIs when an LLM, VLM, or ASR model runs in the same process as your application. Load a deployed LLiMa model directory, create a GenerationRequest, then wait for a complete result or stream tokens as they are generated.

If a browser, service, or remote client needs to call the model over HTTP, use the GenAI Server instead. See the GenAI Model overview for help choosing between the two application boundaries.

Choose the right handle​

For most applications, start with GenAIModel. It auto-detects the model task from the model directory and exposes capability checks:

  • accepts_text()
  • accepts_image()
  • accepts_audio()
  • task()
  • model_id()

Use the task-specific handles when the application knows what it is loading and wants the narrower API:

HandleUse for
genai::GenAIModelAuto-detected LLM, VLM, or ASR model directories.
genai::VisionLanguageModelText-only LLMs and image-capable VLMs.
genai::ASRModelSpeech-to-text models.

Run a text request​

#include "neat/genai.h"

#include <iostream>

int main() {
  simaai::neat::genai::GenAIModel model(
      "/media/nvme/llima/models/Qwen3-4B-Instruct-2507-GPTQ-a16w4");

  simaai::neat::genai::GenerationRequest request;
  request.prompt = "Explain what an API gateway is in one sentence.";
  request.max_new_tokens = 64;

  auto result = model.run(request);
  std::cout << result.text << "\n";
}

run() is synchronous: it returns after generation finishes. It is the simplest shape for tests, scripts, and request/response application code.

Stream generated tokens​

Use stream() when the caller should see output as it is generated. Each item is a TokenSample containing the latest text fragment, current metrics, and final status when generation ends.

simaai::neat::genai::GenerationRequest request;
request.prompt = "Give me three practical tips for designing a small REST API.";
request.max_new_tokens = 96;

simaai::neat::genai::GenerationStream stream_handle = model.stream(request);
for (const auto& token : stream_handle) {
  std::cout << token.text << std::flush;
}
std::cout << "\n";

Call cancel() on the stream if the user closes the request, changes prompts, or your application times out the generation.

Add images for VLMs​

VLMs accept text plus one or more images. Images are passed through GenerationRequest.images for a simple prompt, or through ChatMessage.images when you use chat history.

Images passed as Tensor values should be uint8 HWC RGB tensors. OpenCV cv::Mat inputs follow the Neat/OpenCV convention: three-channel matrices are treated as BGR and converted to RGB before they are stored in the request.

simaai::neat::genai::VisionLanguageModel model(
    "/media/nvme/llima/models/Qwen3-VL-4B-Instruct-GPTQ-a16w4");

cv::Mat image = cv::imread("scene.jpg");

simaai::neat::genai::GenerationRequest request;
request.prompt = "What is visible in this image?";
request.images = {image};
request.max_new_tokens = 128;

auto result = model.run(request);
std::cout << result.text << "\n";

For repeated questions about the same image, VisionLanguageModel.encode(...) can cache image embeddings in the model. Then set request.use_cached_images = true or use a chat message with use_cached_images = true.

Switch LoRA adapters​

Models compiled with LLiMa's LORA_BRANCH mode can switch compatible adapters stored under npy_files/<adapter-name>:

model.set_lora("customer-adapter");
auto adapted = model.run(request);
model.unset_lora();

The adapter name must be one directory name, not a path. Switching waits for any active generation to finish and clears the language model's cached token state before the next request. Dynamic switching is not available for ASR, speculative-decoding packages, or permanently merged LORA_MERGED weights.

Transcribe audio​

ASR models use the same request/result shape, but the request must provide audio. Use audio_file for a file path or audio for an audio tensor.

simaai::neat::genai::ASRModel model("/media/nvme/llima/models/whisper-model");

simaai::neat::genai::GenerationRequest request;
request.audio_file = "meeting.wav";

auto result = model.run(request);
std::cout << result.text << "\n";
std::cout << "language=" << result.language << "\n";

The default language is auto, so multilingual Whisper models detect the source language. Set request.language to a supported language code or name when the source language is known. Results also expose no_speech_prob and avg_logprob when the loaded Whisper artifact provides those probes. A higher no_speech_prob means the input is more likely to contain no speech. A higher (less negative) avg_logprob means Whisper assigned greater average probability to the generated tokens.

To translate speech into English, select the translation task:

request.asr_task = simaai::neat::genai::ASRTask::Translate;
auto result = model.run(request);

Compose GenAI into a Graph​

Direct run() / stream() calls are the shortest path for most GenAI applications. When GenAI is one stage in a larger Neat graph, use the public Graph fragments:

  • genai::graphs::VisionLanguage(...)
  • genai::graphs::SpeechTranscriber(...)

These fragments expose GenAI stages through named graph endpoints so you can compose them with the same Graph and Run model used by the rest of Neat.

For ASR graphs, SpeechTranscriberOptions uses automatic source-language detection and transcription by default. Select translation explicitly:

auto model = std::make_shared<simaai::neat::genai::ASRModel>(
    "/media/nvme/llima/models/whisper-small-a16w8");

simaai::neat::genai::SpeechTranscriberOptions options;
options.task = simaai::neat::genai::ASRTask::Translate;
options.streaming = true;

auto fragment = simaai::neat::genai::graphs::SpeechTranscriber(model, options);

The fragment accepts audio and audio_path. Its final done bundle includes text, finish_reason, language, no_speech_prob, and avg_logprob when the model provides the probe outputs.

auto model = std::make_shared<simaai::neat::genai::VisionLanguageModel>(
    "/media/nvme/llima/models/Qwen3-VL-4B-Instruct-GPTQ-a16w4");

simaai::neat::genai::VisionLanguageOptions options;
options.max_new_tokens = 128;
options.streaming = true;

simaai::neat::Graph fragment =
    simaai::neat::genai::graphs::VisionLanguage(model, options, "vlm");

Request rules​

GenerationRequest is intentionally explicit:

  • Use either prompt or messages, not both.
  • Use system_prompt only with prompt.
  • Attach images directly only with prompt; attach per-message images through ChatMessage.images.
  • Use either direct images or cached images, not both.
  • ASR requests use audio fields, not text or image fields.

Set GenerationRequest.enable_thinking for reasoning models like Qwen3 or Gemma 4 E2B/E4B to enable reasoning. Complete results keep reasoning in GenerationResult.reasoning and the final answer in GenerationResult.text; streamed TokenSample objects likewise use reasoning and text.

These rules let Neat fail early with a clear request error instead of sending an ambiguous prompt to the runtime.

Next steps​