InputLookupTokenPredictor changes greedy output and slows generation on hybrid (qwen35) models

[Rohit-Gide]

Issue description

Using InputLookupTokenPredictor on a hybrid model (architecture qwen35, Gated DeltaNet recurrent layers + attention) changes the generated text at temperature: 0 and makes generation 2 to 3 times slower. On a dense model (qwen3) the same script produces identical output and runs faster, as expected.

It looks like refuted predictions are not rolled back correctly in the recurrent state, so after the first refuted prediction the model continues from a corrupted state.

For comparison, llama-server (b11033) with --spec-type ngram-mod on the same qwen35 model gives byte-identical greedy output with ~1.9x faster generation, so hybrid-model rollback works in llama.cpp itself.

Expected Behavior

Greedy output is identical with and without a token predictor (the docs say using a token predictor doesn't affect the generation output), and generation is not slower on input-grounded text.

Actual Behavior

Output from the script below (node-llama-cpp 3.21.1, Metal, Apple M3 Pro):

=== Qwen3-0.6B-Q8_0 (dense)
arch: qwen3
no predictor:   3.14s
with predictor: 2.31s  validated=150 refuted=300
identical greedy output: true

=== Qwen3.5-0.8B-Q8_0 (hybrid)
arch: qwen35
no predictor:   3.66s
with predictor: 10.52s  validated=158 refuted=715
identical greedy output: false
first divergence at char 10:
  without: "0(a, b) {\n  const r = a * 0 + b;\n  return r - 0;\n}\n\nfunction"
  with:    "10(a), b(a)) {\nfunction f1(a), b(a {\nfunction f1(a), b {\nfun"

=== 9B qwen35 fine-tune, Q5_K_M (hybrid)
arch: qwen35
no predictor:   23.26s
with predictor: 50.85s  validated=22 refuted=371
identical greedy output: false
first divergence at char 8:
  without: " f0(a, b) {\n  const r = a * 0 + b;\n  \"use strict\";\n}\n\nfuncti"
  with:    "functionfunctionfunctionfunctionfunctionfunctionfunctionfunc"

Same behavior on 3.19.1 (llama.cpp b10068).

Steps to reproduce

Models:

• dense: https://huggingface.co/Qwen/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q8_0.gguf
• hybrid: https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF/resolve/main/Qwen3.5-0.8B-Q8_0.gguf

node repro.mjs <model.gguf>:

import { getLlama, LlamaCompletion, InputLookupTokenPredictor } from "node-llama-cpp";
const [modelPath] = process.argv.slice(2);
const llama = await getLlama();
const model = await llama.loadModel({ modelPath });
// Input-grounded task: the output should copy large spans of the prompt.
const src = Array.from({length: 40}, (_, i) => `function f${i}(a, b) {\n  const r = a * ${i} + b;\n  return r - ${i};\n}\n`).join("\n");
const prompt = `Here is a JavaScript file:\n\n${src}\nThe same file, unchanged:\n\n`;
async function run(predictor) {
  const context = await model.createContext({ contextSize: 4096 });
  const sequence = context.getSequence(predictor ? { tokenPredictor: new InputLookupTokenPredictor() } : {});
  const completion = new LlamaCompletion({ contextSequence: sequence });
  const t0 = performance.now();
  const text = await completion.generateCompletion(prompt, { maxTokens: 300, temperature: 0 });
  const secs = (performance.now() - t0) / 1000;
  const stats = sequence.tokenPredictions;
  await context.dispose();
  return { text, secs, stats };
}
const a = await run(false), b = await run(true);
console.log(`arch: ${model.fileInfo.metadata.general.architecture}`);
console.log(`no predictor:   ${a.secs.toFixed(2)}s`);
console.log(`with predictor: ${b.secs.toFixed(2)}s  validated=${b.stats.validated} refuted=${b.stats.refuted}`);
console.log(`identical greedy output: ${a.text === b.text}`);
if (a.text !== b.text) { let i = 0; while (a.text[i] === b.text[i]) i++; console.log(`first divergence at char ${i}:\n  without: ${JSON.stringify(a.text.slice(i, i + 60))}\n  with:    ${JSON.stringify(b.text.slice(i, i + 60))}`); }

My Environment

Dependency	Version
Operating System	macOS 26.3 (Darwin 25.3.0)
CPU	Apple M3 Pro (18GB)
Node.js version	v22.21.1
Typescript version	n/a (plain ESM)
node-llama-cpp version	3.21.1 (also 3.19.1)

Additional Context

Also reproduced with LlamaChatSession and a larger predictionLength.max (8, 16); larger values make it slower, and the output diverges the same way.

Relevant Features Used

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

No, I don’t have the time and I’m okay to wait for the community / maintainers to resolve this issue.