Skip to main content

GenAI Model

Neat uses the GenAI APIs for LLM, VLM, and ASR models prepared with LLiMa. This is the generative-model counterpart to Model: load a deployed model directory, create a request, then run it inside your application or serve it over HTTP.

Use these APIs when your application asks a model to generate text, answer questions about images, call tools, or transcribe audio. Use the classic Model API for fixed-shape discriminative models such as classification, detection, segmentation, or embedding models.

Where GenAI fits​

The GenAI path has the same high-level shape as the rest of Neat:

  1. Prepare or download a model artifact for Modalix.
  2. Put the model on the target, commonly under /media/nvme/llima/models/.
  3. Load the model from your C++ or Python application.
  4. Send a request directly or expose the model through GenAIServer.
  5. Read the complete result or consume streamed output.

LLiMa owns GenAI model preparation, command-line testing, and benchmarking. Neat owns the application-facing API and runtime integration once the model is used inside your app. For model preparation details, see GenAI with LLiMa.

Choose an application boundary​

Serve applications and clients

Recommended for most applications. Use the GenAI server to expose common API endpoints.

Run in your process

Use the direct APIs when your C++ or Python application owns the model call.

Use direct calls for embedded application logic and tests that should run in the same process as the model. Use the server when clients should not link against the Neat runtime or when one process should route requests to multiple served models.

Next steps​