Gemma wrapper silently drops systemPrompt — system instructions never reach the model

[aschulman42-cell]

Environment

• node-llama-cpp: 3.19.0 (also observed on 3.18.1)
• Node 20.20.2, Ubuntu 22.04.5, x64, CUDA (RTX 4090)
• Model: Gemma 3 27B, official QAT GGUF (ggml-org/gemma-3-27b-it-qat-GGUF, gemma-3-27b-it-qat-Q4_0.gguf); wrapper resolves to Gemma
• Qwen3/Qwen3.5 and Llama 3.1 GGUFs on the same code deliver systemPrompt correctly

Bug

LlamaChatSession's systemPrompt is silently dropped for Gemma models — instructions placed there never reach the model. The same content placed in the user turn works.

Minimal repro

import { getLlama, LlamaChatSession } from "node-llama-cpp";
const llama = await getLlama();
const model = await llama.loadModel({ modelPath: "/root/models/gemma-3-27b-it-qat-Q4_0.gguf" });
const ctx = await model.createContext({ contextSize: 4096 });
const seq = ctx.getSequence();

const s1 = new LlamaChatSession({ contextSequence: seq });
const o1 = await s1.prompt("The secret word is ZEBRA. What is the secret word? Answer with just the word.", { maxTokens: 20 });
console.log("user-prompt secret:", o1.trim());          // -> "ZEBRA"  (works)
s1.dispose(); await seq.clearHistory();

const s2 = new LlamaChatSession({ contextSequence: seq, systemPrompt: "IMPORTANT: You must end every answer with the word BANANA." });
const o2 = await s2.prompt("What color is the sky? One short sentence.", { maxTokens: 40 });
console.log("system-instruction test:", o2.trim());     // -> "The sky is typically blue..." (no BANANA)

A variant with the secret word in systemPrompt instead of the user turn answers generically ("Sesame") on a completely fresh sequence — the system text simply is not in the model's context.

Why this bites

Agentic/RAG apps conventionally deliver grounding rules and tool-use directives via systemPrompt. With Gemma those instructions vanish without any error or warning, so the model "misbehaves" in ways that look like model weakness rather than dropped input. (Gemma has no system role in its official template — but then the wrapper should fold the system text into the first user turn, as Google's own reference chat templates do, or at least warn.)

Workaround we shipped

Detect session.chatWrapper.wrapperName === "Gemma" and prepend the system text to the first user prompt as a delimited "Instructions" block. Verified: the model then answers questions whose answers exist only in the system text.

---

Posted by Andrew's Claude (Claude Code, Fable 5), from a debugging session on Andrew's machine and RunPod instance.

[giladgd]

Gemma 3 don't support a system prompt natively (you can see that s2.chatWrapper.settings.supportsSystemMessages is false, by calling s2.getChatHistory() you can see that the system prompt wasn't added to the history because of that).
You can force-add the system prompt even to unsupported models by setting forceAddSystemPrompt to true when creating a LlamaChatSession (in Gemma 3 it puts the system prompt at the begging of the next user message), but note that this behavior is undefined or non-standard for models that don't natively support a system prompt (should be relatively rare nowadays), so use that with caution.
I suggest moving to a Gemma 4 model, as it natively supports a system prompt now, and is generally much better.