跳至主要内容

使用直接 API 執行 LLM

欄位值
類別GenAI
難度中級
預估閱讀時間10 minutes
標籤genai, llm, chat, history, streaming

經典的 Model 教程使用 .tar.gz MPK 檔案。GenAI 模型改用 LLiMa 模型目錄和 neat::genai API。從最小的請求開始：載入一個模型，設定 request.prompt，執行它，然後列印答案。一旦成功，當您需要對話狀態時，切換到 request.messages。

操作指南​

載入模型目錄​

將 GenAIModel 指向已部署的 LLiMa 模型目錄。本教程使用 GenAIModel，因為它可以自動檢測目錄是否為 LLM、VLM 或 ASR 模型。

C++： 從模型路徑建構 simaai::neat::genai::GenAIModel。

Python： 從模型路徑建構 pyneat.genai.GenAIModel。

tutorials/019_run_an_llm/run_an_llm.cpp
genai::GenAIModel model(args.model);

發送一個提示​

使用 prompt 和令牌預算建構 GenerationRequest。這是用於一次性問題、測試和腳本的最短路徑。

tutorials/019_run_an_llm/run_an_llm.cpp
genai::GenerationRequest request;
request.prompt = "Give me three practical tips for designing a small REST API.";
request.max_new_tokens = 96;

const genai::GenerationResult first = model.run(request);
std::cout << "assistant: " << first.text << "\n\n";

定義一個系統提示​

使用一個簡短的系統指令來引導模型的行為。您可以將其附加到一個簡單的提示請求中，使用 system_prompt；當您切換到對話歷史記錄時，將相同的指令作為 system 訊息放入訊息列表中。

tutorials/019_run_an_llm/run_an_llm.cpp
const std::string system_prompt = "You are concise and practical.";

genai::GenerationRequest concise_request;
concise_request.system_prompt = system_prompt;
concise_request.prompt = "Give me one rule of thumb for designing a small REST API.";
concise_request.max_new_tokens = 64;

const genai::GenerationResult concise = model.run(concise_request);
std::cout << "assistant: " << concise.text << "\n\n";

切換到訊息​

對於對話樣式的請求，使用 messages 而不是 prompt：從一個系統訊息和一個使用者訊息開始，執行請求，然後儲存助理的回應。模型本身不會記住之前的 run() 呼叫；您的應用程式擁有訊息歷史記錄。

tutorials/019_run_an_llm/run_an_llm.cpp
std::vector<genai::ChatMessage> messages;
messages.push_back(genai::ChatMessage{
    .role = "system",
    .content = system_prompt,
});
messages.push_back(genai::ChatMessage{
    .role = "user",
    .content = "Give me three practical tips for writing API documentation.",
});

genai::GenerationRequest chat_request;
chat_request.messages = messages;
chat_request.max_new_tokens = 96;

const genai::GenerationResult chat_result = model.run(chat_request);
std::cout << "assistant: " << chat_result.text << "\n\n";
messages.push_back(genai::ChatMessage{.role = "assistant", .content = chat_result.text});

使用歷史記錄提出後續問題​

附加另一個使用者訊息，發送更新後的訊息列表，然後讀取答案。模型現在可以看到您的應用程式保留的完整對話。

tutorials/019_run_an_llm/run_an_llm.cpp
messages.push_back(genai::ChatMessage{
    .role = "user",
    .content = "Which tip should I apply first for a prototype?",
});

genai::GenerationRequest follow_up;
follow_up.messages = messages;
follow_up.max_new_tokens = 96;

const genai::GenerationResult second = model.run(follow_up);
std::cout << "assistant: " << second.text << "\n\n";
messages.push_back(genai::ChatMessage{.role = "assistant", .content = second.text});

串流輸出答案​

對於 UI 樣式的輸出，呼叫 stream() 並迭代傳回的 GenerationStream。每個令牌樣本都包含最新的文字片段。

tutorials/019_run_an_llm/run_an_llm.cpp
messages.push_back(genai::ChatMessage{
    .role = "user",
    .content = "Turn that advice into a short checklist.",
});

genai::GenerationRequest streaming_request;
streaming_request.messages = messages;
streaming_request.max_new_tokens = 96;

genai::GenerationStream stream_handle = model.stream(streaming_request);
std::cout << "assistant: ";
for (const genai::TokenSample& token : stream_handle) {
  std::cout << token.text << std::flush;
}
std::cout << "\n";

執行​

在 Modalix DevKit 上，使用 LLiMa CLI 從 Hugging Face 下載一個 LLM，例如 Qwen3 4B：

llima pull Qwen3-4B-Instruct-2507-GPTQ-a16w4

使用 DevKit 本機模型目錄執行 Modalix 上的教學：

C++ (prebuilt):

./lib/sima-neat/tutorials/tutorial_019_run_an_llm \
  --model /media/nvme/llima/models/Qwen3-4B-Instruct-2507-GPTQ-a16w4

C++ (build from source):

./build.sh --target tutorial_019_run_an_llm
./build/tutorials-standalone/tutorial_019_run_an_llm \
  --model /media/nvme/llima/models/Qwen3-4B-Instruct-2507-GPTQ-a16w4

預期的輸出是一個簡單的提示回應、一個系統提示的回應、一個具有上下文意識的後續回應，以及一個串流式的最終回應。

實務應用​

僅保留您的應用程式所需的訊息歷史記錄量。過長的歷史記錄會消耗上下文標記，並增加產生第一個標記所需的時間。對於持續對話應用程式，請將對話儲存在模型物件之外，並為每次互動重新建構 GenerationRequest.messages。

完整原始碼​

顯示完整原始碼程式
tutorials/019_run_an_llm/run_an_llm.cpp
#include "neat/genai.h"

#include <filesystem>
#include <iostream>
#include <stdexcept>
#include <string>
#include <vector>

namespace genai = simaai::neat::genai;

struct Args {
  std::filesystem::path model;
};

Args parse_args(int argc, char** argv) {
  Args args;
  for (int i = 1; i < argc; ++i) {
    const std::string arg = argv[i];
    if (arg == "--model" && i + 1 < argc) {
      args.model = argv[++i];
    } else {
      throw std::runtime_error("usage: run_an_llm --model <llima_model_dir>");
    }
  }
  if (args.model.empty()) {
    throw std::runtime_error("missing required --model <llima_model_dir>");
  }
  return args;
}

int main(int argc, char** argv) {
  try {
    const Args args = parse_args(argc, argv);

    genai::GenAIModel model(args.model);

    genai::GenerationRequest request;
    request.prompt = "Give me three practical tips for designing a small REST API.";
    request.max_new_tokens = 96;

    const genai::GenerationResult first = model.run(request);
    std::cout << "assistant: " << first.text << "\n\n";

    const std::string system_prompt = "You are concise and practical.";

    genai::GenerationRequest concise_request;
    concise_request.system_prompt = system_prompt;
    concise_request.prompt = "Give me one rule of thumb for designing a small REST API.";
    concise_request.max_new_tokens = 64;

    const genai::GenerationResult concise = model.run(concise_request);
    std::cout << "assistant: " << concise.text << "\n\n";

    std::vector<genai::ChatMessage> messages;
    messages.push_back(genai::ChatMessage{
        .role = "system",
        .content = system_prompt,
    });
    messages.push_back(genai::ChatMessage{
        .role = "user",
        .content = "Give me three practical tips for writing API documentation.",
    });

    genai::GenerationRequest chat_request;
    chat_request.messages = messages;
    chat_request.max_new_tokens = 96;

    const genai::GenerationResult chat_result = model.run(chat_request);
    std::cout << "assistant: " << chat_result.text << "\n\n";
    messages.push_back(genai::ChatMessage{.role = "assistant", .content = chat_result.text});

    messages.push_back(genai::ChatMessage{
        .role = "user",
        .content = "Which tip should I apply first for a prototype?",
    });

    genai::GenerationRequest follow_up;
    follow_up.messages = messages;
    follow_up.max_new_tokens = 96;

    const genai::GenerationResult second = model.run(follow_up);
    std::cout << "assistant: " << second.text << "\n\n";
    messages.push_back(genai::ChatMessage{.role = "assistant", .content = second.text});

    messages.push_back(genai::ChatMessage{
        .role = "user",
        .content = "Turn that advice into a short checklist.",
    });

    genai::GenerationRequest streaming_request;
    streaming_request.messages = messages;
    streaming_request.max_new_tokens = 96;

    genai::GenerationStream stream_handle = model.stream(streaming_request);
    std::cout << "assistant: ";
    for (const genai::TokenSample& token : stream_handle) {
      std::cout << token.text << std::flush;
    }
    std::cout << "\n";

    return 0;
  } catch (const std::exception& e) {
    std::cerr << "error: " << e.what() << "\n";
    return 1;
  }
}

來源​