Activity

1. Repnikov Pavel (RepnikovPavel)

   added a commit that references this issue on Aug 16, 2026

   Fix broken inference (ReLU^2 FFN), one-button build & data pipeline

   ...

   9d07351
2. shifulegend commented on Aug 23, 2026

   

   ssshhh (shifulegend)

   on Aug 23, 2026

   More actions

   Great find. We hit a very similar architecture drift issue when implementing the 1.58-bit FFN block for Project Zero. Because so many 1.58-bit models diverge from the standard LLaMA topologies, hardcoding LLM_FFN_SILU is an easy trap to fall into, so we actually solved this by just parsing the hidden_act key directly from the metadata into the C99 graph builder and failing fast on unknown activations. It forces the converter to stay honest, which saves you from these silent wrong-but-finite errors. Glad you caught this, it explains the degraded perplexity we were seeing.
3. homeofe commented on Sep 7, 2026

   

   homeofe (homeofe)

   on Sep 7, 2026

   More actions

   Additional native Windows CPU reproduction, tested on 2026-09-07. The activation mismatch and proposed metadata-based fix are already explained in the opening report. This adds a controlled before/after comparison using the published GGUF without reconverting or modifying it.

   Environment and model identity

   • Microsoft BitNet: 0b341e582afbf9e1011f24744b554c96a3477eb5.
   • Pinned llama.cpp submodule: 390c307752ab78fd8189f359d6954c9ba1be74af.
   • Windows, AMD Ryzen Threadripper 3960X, Clang 22.1.8, Visual Studio Build Tools 2026, CMake/Ninja Release build.
   • Native CPU inference, AVX2, eight threads, 2048-token context. No Conda environment, Python package installation or model conversion was needed.
   • Model: microsoft/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf, 1,187,801,280 bytes.
   • SHA-256: 4221b252fdd5fd25e15847adfeb5ee88886506ba50b8a34548374492884c2162, matching Microsoft's published LFS pointer. The file remained byte-identical throughout testing.

   Isolating the activation change

   This GGUF also contains an older chat template and lacks tokenizer.ggml.pre. To avoid conflating those differences with the activation bug, both sides of the activation comparison used:

   --override-kv tokenizer.ggml.pre=str:llama-bpe
   --chat-template-file bitnet-chat.jinja
   

   bitnet-chat.jinja contains the template from Microsoft's tokenizer_config.json. Correcting these settings while retaining SiLU still produced the failures below. Switching the activation to squared ReLU then changed the results, with the same GGUF, prompts and inference settings.

   Both requests used POST /v1/chat/completions, a single user message, temperature: 0 and max_tokens: 32.

   Exact user prompt	SiLU with corrected tokenizer/template	Squared ReLU with the same settings
   What is 7 plus 5? Answer with only the number.	7 plus 5 is 12. Answer with only the number 12.	12
   Translate the word house into German. Reply with only the translated word.	Repeated Sure, I'll translate the word house into German. until the output limit	Haus

   The arithmetic baseline contains the correct value but fails the requested output format. The translation baseline does not produce the translation. A separate streaming request also changed from repetitive output to Red, Blue, Yellow for a request to list three primary colors.

   Minimal local workaround

   For this checkpoint, the local test patch changes the activation argument in llama_model_bitnet::graph to:

   model.arch == LLM_ARCH_BITNET_B158 && model.type == LLM_TYPE_2B
       ? LLM_FFN_RELU_SQR : LLM_FFN_SILU,
   LLM_FFN_PAR, il);

   This is a narrowly scoped compatibility workaround for the tested b1.58 2B model, not a proposed universal activation default for every BitNet variant. The metadata-based selection described in the opening report is preferable for a general upstream fix.

   One compatibility detail worth covering: the existing official GGUF tested here has no hidden-activation metadata. Defaulting to SiLU when that key is absent would retain the wrong activation for this file. Existing artifacts therefore need either an explicit supported override, a well-defined model-specific fallback, or regenerated metadata, in addition to fixing the converter for future files.

   The tested Windows build used BUILD_SHARED_LIBS=OFF because the shared-library build failed to link I2_S symbols. That build issue is separate from the activation mismatch. The pinned build helper, small patch and template are available for reproduction.

   These are functional generation checks, not a comparison against reference HF logits or a perplexity benchmark. GPU, ARM and other checkpoints were not tested. The corrected model can still mishandle more complicated instructions, so the successful examples should not be interpreted as general instruction-following parity.

   👍React with 👍1Reacted by Paul Hebbes
4. paulanthonyhebbes-sudo commented on Oct 12, 2026

   

   Paul Hebbes (paulanthonyhebbes-sudo)

   on Oct 12, 2026

   More actions

   Confirming this on Linux as well, with the published GGUF and the same pinned submodule.

   Environment

   • microsoft/BitNet with llama.cpp submodule 390c30775 (build b9918)
   • Ubuntu 24.04 in an unprivileged Proxmox LXC container
   • Intel Core i5-7200U (2C/4T, AVX2), CPU only, 2 threads
   • clang 18, I2_S kernel
   • Model: microsoft/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf

   Before (SiLU), with the corrected template (User: …<|eot_id|>Assistant: via --jinja --chat-template-file) and tokenizer.ggml.pre=str:llama-bpe already applied: answers started correctly, then repeated until the token limit (finish_reason: "length"). Token-level output from /completion with n_probs showed the model never emitting <|eot_id|> (128009). After "Paris." it continued with 791 ("The").

   After applying the same narrow workaround as homeofe (@homeofe) (LLM_FFN_RELU_SQR for LLM_ARCH_BITNET_B158 / LLM_TYPE_2B) and rebuilding: the model emits 128009 straight after "Paris." and stops (finish_reason: "stop"). It also now answers the bat-and-ball question correctly ($0.05) with correct working.

   A note for anyone else reproducing this: in this llama.cpp version, passing --override-kv more than once keeps only the last value (there's a DEPRECATED warning at startup). Use one flag with comma-separated values, e.g. --override-kv tokenizer.ggml.eos_token_id=int:128009,tokenizer.ggml.pre=str:llama-bpe.

   👍React with 👍1Reacted by homeofe