This folder builds the Windows llama.cpp that WelsonJS uses, with WelsonJS’s patches applied. The base source is the llama.cpp fork that Microsoft distributes with bitnet.cpp as its 3rdparty/llama.cpp submodule. The fork was created to support bitnet.cpp’s quantization. (The repository itself is hosted on a contributor’s account, isHuangXin/llama.cpp; see Files.) The output contains the same files as the official llama-<version>-bin-win-cpu-<arch>.zip package: llama-server, llama-cli, llama-quantize, and the ggml backends. The x64 build has every x86-64 CPU variant, chosen at run time; the arm64 build has one CPU backend.
The fork adds BitNet’s I2_S type and kernels, so besides the usual GGUF files the build loads 1.58-bit BitNet models such as microsoft/BitNet-b1.58-2B-4T-gguf, which upstream llama.cpp cannot load.
On top of the fork, this build adds the STQ1_0 ternary type and kernels from Tencent’s AngelSlim quantization (ggml-org/llama.cpp#22836, not merged upstream; see 0001-stq1_0.patch). It loads 1.25-bit models produced by AngelSlim, such as tencent/Hy-MT2-1.8B-1.25Bit-GGUF, which neither the fork nor upstream llama.cpp can load.
The launcher runs llama-server in router mode, with the installed models in %APPDATA%\WelsonJS, and uses only its HTTP API; requests name their model (docs/llamacpp-launcher-api.md).
| File | Purpose |
|---|---|
commit.txt |
The llama.cpp commit that is built, as a full 40-character SHA. The source is fetched by this SHA rather than by a branch, because a branch can move and a commit cannot. |
version.txt |
The version label, <upstream release>.<commit>.1 (llama.cpp records no version in its tree, so the release stands for it): the last upstream llama.cpp release (bNNNN) before the commit where the fork diverges from upstream, the first seven characters of the pinned commit (commit.txt), and .1 for the WelsonJS build. It is a label only (BUILDINFO.txt and the build ID) and does not select the source. |
source-archive.txt |
The fallback source archive (url=) and its SHA-256 (sha256=), used when git cannot fetch the commit. |
patches/*.patch |
Patches applied in name order with git apply. They are stored with LF line endings, and .gitattributes keeps them byte-exact on checkout. |
build-llamacpp.ps1 |
Fetches the source, applies the patches, builds, and writes the output. |
Current pin: commit 390c307752ab78fd8189f359d6954c9ba1be74af of https://github.com/isHuangXin/llama.cpp.git (branch release-bitnet-embedding-0.6b-270m, 2026-07-15). It is the submodule commit of BitNet 0b341e582afbf9e1011f24744b554c96a3477eb5. It diverges from upstream llama.cpp (ggml-org/llama.cpp) after 665892536dfb (2026-07-04, 46 commits of its own); the last upstream release before that commit is b9871, so version.txt is b9871.390c307.1.
The fork is older than upstream llama.cpp, so upstream changes made after it are not included.
bitnet.cpp builds the fork as a subproject of the BitNet repository, which adds two things:
src/ggml-bitnet-lut.cpp, include/ggml-bitnet.h, include/bitnet-lut-kernels.h). These need code generated for one model’s shapes (utils/codegen_tl*.py, run by setup_env.py), so a build with them serves only that model.The I2_S type and kernels (ggml/src/ggml-cpu/ggml-cpu-i2s.c, AVX2, and AVX-VNNI or AVX512-VNNI when available) are part of the fork. They need one header from the BitNet repository, gemm-config.h (the kernels’ block sizes), which 0002-bitnet-i2s.patch (part 1) copies into the fork.
The other BitNet files are used only inside #if defined(GGML_BITNET_ARM_TL1) || defined(GGML_BITNET_X86_TL2) in the fork’s code. Only the fork’s CMake refers to them unconditionally, and 0002-bitnet-i2s.patch (part 1) changes that. So with TL1/TL2 off, the fork builds on its own, with MSVC like upstream llama.cpp.
powershell -ExecutionPolicy Bypass -NoProfile -File native\llamacpp\build-llamacpp.ps1 `
[-Arch x64|x86|arm64] [-OutDir <dir>] [-WorkDir <dir>] [-Source auto|git|archive] [-LlvmDir <dir>] [-InstallLlvm] [-Force]
Requirements:
choco install llvm), because ggml does not support MSVC on ARM.Microsoft.VisualStudio.Component.VC.Tools.ARM64). The script checks for VC\Tools\MSVC\<version>\lib\arm64\msvcrt.lib, because the lib\arm64 folder can exist with only the AddressSanitizer libraries.| x64 | arm64 | |
|---|---|---|
| Compiler | MSVC, Visual Studio generator | clang (LLVM) with the fork’s cmake\arm64-windows-llvm.cmake toolchain, Ninja |
| Environment | Visual Studio default | Visual Studio x64_arm64 developer environment (ARM64 libraries, x64-hosted ARM64 linker); the MSVC version with the ARM64 C runtime is passed to vcvarsall.bat |
| CPU target | every x86-64 variant (GGML_CPU_ALL_VARIANTS), chosen at run time |
-march=armv8.7-a, one backend ggml-cpu.dll (GGML_CPU_ALL_VARIANTS is not supported for ARM on Windows) |
| OpenMP | on (VCOMP140.DLL from the VC++ redistributable) |
off: with clang it needs libomp140.aarch64.dll, which Visual Studio ships only as debug_nonredist; ggml uses its own thread pool |
Host tool (llama-ui-embed) |
built as part of the build | built for x64 with host-cxx.cmd in the work folder (HOST_CXX_COMPILER), see below |
x86 (32-bit) is built like x64, with MSVC and the Visual Studio generator (-A Win32), but with one CPU backend (ggml-cpu.dll) with the AVX2 baseline of GGML_NATIVE=OFF and without BMI2 (GGML_CPU_ALL_VARIANTS=OFF, GGML_BMI2=OFF): ggml’s BMI2 code uses _pdep_u64, which 32-bit MSVC lacks, and several of the CPU variants enable BMI2. The executables are linked with /LARGEADDRESSAWARE, so a 32-bit process gets 4 GB of address space on 64-bit Windows instead of 2 GB; without it, the model and the KV cache of an 8192-token context (the launcher’s LlamaCppContextSize) could not be allocated. The x86 build gives the same greedy answers as the x64 build (Qwen3Guard 0.6B Q4_K_M, Hy-MT2 1.8B Q4_K_M and BitNet b1.58 2B I2_S), with an 8192-token context. A larger context or model may not fit in 4 GB.
-InstallLlvm runs choco install llvm -y when an arm64 build is needed and clang is missing.PATH, ahead of other copies (for example, Strawberry Perl’s CMake 3.29).tools/ui builds llama-ui-embed for the host (x64) with the first g++ or clang++ on PATH, passing only -O2 -std=c++17 -o.
x64_arm64 environment, LIB points to the ARM64 libraries, so clang++ failed to link it on AppVeyor (LNK2019, LNK4272). Locally, Strawberry Perl’s g++ was found first and hid the problem.host-cxx.cmd, which sets LIB to the x64 libraries (the LIB entries with arm64 replaced by x64) and calls clang++ --target=x86_64-pc-windows-msvc. It passes the wrapper as HOST_CXX_COMPILER, so every machine uses the same compiler.GGML_CCACHE=OFF for both, so that a ccache on PATH is not used.-march=armv8.7-a is the fork’s default for Windows ARM64; it targets Snapdragon X class CPUs. Older ARM64 PCs (for example, Snapdragon 8cx Gen 1-3) may not run the build.-Source auto (the default) fetches commit.txt with git and checks that HEAD is that commit. If git fails, it falls back to the archive.-Source git and -Source archive force one method.source-archive.txt,llama.cpp-<commit>/.source-archive.txt was compared file by file with a git checkout of the same commit.
spm-headers and spm-sources) are written as small text files holding the link target, as git does with core.symlinks=false. The build does not use them.core.longpaths=true.\\?\ prefix, so the MAX_PATH limit does not apply.core.autocrlf=false so that the LF patches apply. This matters on AppVeyor, which sets core.autocrlf=true globally.git apply runs with -c core.autocrlf=false. Outside a repository (archive source) it otherwise follows the global setting and writes the patched files with CRLF..git folder, so the script passes LLAMA_BUILD_COMMIT and --version still reports the commit.GGML_BACKEND_DL=ON, and for x64 GGML_CPU_ALL_VARIANTS=ON, as in the official package. The fork compiles the I2_S kernels into every CPU variant.GGML_BITNET_ARM_TL1=OFF and GGML_BITNET_X86_TL2=OFF (the defaults), stated explicitly: the TL kernels need the BitNet repository.llama app are not built.LLAMA_OPENSSL=OFF. The tools only need HTTP on localhost.LLAMA_WIN_ICON is set to app\favicon.ico, so the executables show the WelsonJS icon, and every executable and DLL gets version information: the WelsonJS version (0.2.7.62, the launcher’s AssemblyFileVersion; -ProductVersion overrides it), a description that says llama.cpp built for WelsonJS, CompanyName Catswords Research, and the copyright of all authors. The options come from a patch; see 0003-windows-build.patch, part 1.LLAMA_USE_PREBUILT_UI=OFF and LLAMA_BUILD_UI=OFF. By default the build downloads the server Web UI from a floating Hugging Face location, and the launcher does not use the Web UI.The files are written to <OutDir>\<arch>; in CI this is artifacts\llamacpp\x64, artifacts\llamacpp\arm64 and artifacts\llamacpp\x86. No archive is created. The folder also contains:
LICENSE-llama.cpp.txt (MIT).BUILDINFO.txt: the version label, the product version, commit, source used (git, or the archive URL and hash), compiler, CPU backends, BitNet kernels, patches, and the build ID.SHA256SUMS.txt: one <sha256> <file name> line for every file.The build ID is a SHA-256 over these inputs:
version.txt, commit.txt, and the architecture,app\favicon.ico, the icon embedded in the executables,The script skips the build when BUILDINFO.txt has the same build ID and every file matches SHA256SUMS.txt. Otherwise, for example when an input changed or a file was modified, it builds again. -Force always builds.
The following are not part of the build ID:
source-archive.txt, because both sources contain the same commit.-Force for that.The binaries are not built in CI: building llama.cpp and CrispEmbed for x64, arm64 and x86 takes longer than the 60-minute AppVeyor job limit. They are built on a developer PC with this script, published to the WelsonJS blob storage, and restored by the Release job, which packages and signs them with the other artifacts.
-Arch x64, -Arch arm64 and -Arch x86 (on a PC whose CMake on PATH defaults to another generator, pass -Generator "Visual Studio 18 2026" for x64 and x86).native\publish-prebuilt.ps1 -Component llamacpp -Arch <arch> -BuildDir <OutDir>\<arch>.
SHA256SUMS.txt, and that BUILDINFO.txt names the current version.txt, commit.txt and architecture.<file>.gz) with AzCopy (azcopy login first) to packages/native/llamacpp/<version>-<arch>/, never overwriting an existing blob, so a published build does not change.SHA256SUMS.txt to native\llamacpp\prebuilt\<arch>.sha256.prebuilt\<arch>.sha256 files with the change.The Debug and Release jobs run native\restore-prebuilt.ps1 -Component llamacpp -Arch <arch> for each architecture, into artifacts\llamacpp\<arch>. It downloads the files listed in prebuilt\<arch>.sha256 from <version>-<arch> as <file>.gz and decompresses them (when a .gz cannot be downloaded or used, it downloads the file itself), and uses a file only if its SHA-256 matches, so the repository, not the storage, decides what is shipped. Both packages are signed (Debug with the test-signing policy, Release with release-signing). Each job caches artifacts\llamacpp, keyed on version.txt and the three prebuilt\<arch>.sha256 files: when they are unchanged, the restored folders match their records and nothing is downloaded.
.artifact-configuration.xml lists every executable and DLL of the three architectures by name (no wildcards), and the build script fails when a built binary is not listed, so a new binary cannot be left unsigned. Authenticode signing changes the files, so in the signed package SHA256SUMS.txt lists the hashes of the unsigned files.
The current builds (b9871.390c307.1) were made with Visual Studio 2026: MSVC 19.50 (toolset 14.50) for x64 and x86, and clang 23 with the MSVC 14.51 libraries for arm64. A PC that runs them needs a Visual C++ 2015-2026 redistributable at least as new as those tools.
The installer uses this build; it no longer downloads the official llama.cpp package (data\DownloadUrls.psd1 has no llamacpp entry).
llamacpp component (and every GGUF model component, which selects it) also selects artifacts, because the build is part of the artifacts package.artifacts.zip, folder llamacpp\<arch>) so that it is signed with the other binaries.postInstall.ps1 (Install-ArtifactsPackage) extracts the package to a staging folder.
llamacpp\<arch> to %APPDATA%\WelsonJS\llamacpp, after removing the files of a previous installation. That is the only place the launcher runs llama-server.exe from. The folder is added to the user’s PATH as before.%APPDATA%\WelsonJS\bin, as before. The llama.cpp build is not left in bin.data\DownloadUrls.psd1) and get the arm64 build. The x86 WelsonJS binaries in the package run under x86 emulation. If a package has no arm64 build, the script falls back to x64, which runs under x64 emulation on Windows 11.The fork numbers some ggml tensor types differently from upstream llama.cpp:
| ggml tensor type | This build (fork) | Upstream llama.cpp |
|---|---|---|
| 36 | I2_S (BitNet) |
unused |
| 37 | I8_S (BitNet, run time only) |
unused |
| 38 | TL1 (BitNet) |
unused |
| 42 | TL2 (BitNet), or STQ1_0 in Tencent’s files (detected by size, 0001-stq1_0.patch part 3) |
Q2_0 |
| 43 | STQ1_0 (0001-stq1_0.patch, the PR’s ID) |
F8_E4M3 |
I2_S GGUF files and STQ1_0 GGUF files (Tencent’s, with type 42, and the PR’s, with type 43) load with this build and not with upstream llama.cpp.Q2_0 (42) or F8_E4M3 (43) tensors are not compatible: this build reads those IDs as TL2/STQ1_0 and STQ1_0. The types have different sizes, so such files are expected to fail the GGUF loader’s size checks rather than load with wrong data; this was not tested.TL2 tensors keep their ID, but this build has no TL2 kernels to run them.hunyuan-dense, qwen35, qwen3, bitnet-b1.58):
tencent/Hy-MT2-1.8B-1.25Bit-GGUF) stores STQ1_0 tensors as type 42 and loads only with this build (0001-stq1_0.patch part 3); its other tensors are F32 and Q6_K.microsoft/bitnet-b1.58-2B-4T-gguf, component bitnet_b1_58_2b_4t) stores I2_S tensors (type 36) and loads only with this build; see BitNet (I2_S) models.microsoft/BitNet-b1.58-2B-4T-gguf works through llama-server’s chat API with no extra options. That depends on parts 4-7 of 0002-bitnet-i2s.patch, which fix four problems found while testing it:
| Problem without the patch | Cause | Part |
|---|---|---|
| Answers the first sentence, then repeats itself until the token limit | The fork’s FFN uses SiLU for the 2B-4T model, which uses squared ReLU | 5 |
Only ? tokens on CPUs without AVX2 (the x64, sse42, and sandybridge variants) |
The fork’s non-AVX2 I2_S fallbacks compute something else than its AVX2 code | 4 |
Invents a conversation through /v1/chat/completions |
The GGUF file embeds a chat template the model does not follow | 6 |
| Warning “GENERATION QUALITY WILL BE DEGRADED”; some text tokenized differently | The GGUF file has no tokenizer.ggml.pre |
7 |
The I2_S kernels themselves were correct. A GGUF with the I2_S weights converted to F16 generated the same tokens as the I2_S file.
general.file_type of the GGUF file is 40, which is Q1_0 in the fork. 0001-stq1_0.patch (part 3) reports it as I2_S, because the file has I2_S tensors and no Q1_0 tensors. The file type is metadata only and does not affect loading.
llama-quantize.exe is installed with the other tools in %APPDATA%\WelsonJS\llamacpp, which is on the user’s PATH. llama-quantize --help lists the types it accepts.
Select the type by name, not by number. llama-quantize accepts both: it first looks for the name (case does not matter), then for a number in its list of file types. The numbers of the BitNet and STQ1_0 types differ between builds, so a number from another build or file can select a different type:
| Number | This build | Elsewhere |
|---|---|---|
| 41 | I2_S |
general.file_type 41 in Tencent’s 1.25-bit files means STQ1_0 |
| 42 | not accepted (TL1, not in the list) |
STQ1_0 in PR #22836 |
| 44 | STQ1_0 |
STQ1_0 is 42 in PR #22836 and 41 in Tencent’s files |
For example, 41 taken from a Tencent file quantizes to I2_S, not STQ1_0, and damages a model that is not ternary. The names I2_S and STQ1_0 select the same type in every build that has it.
Examples:
# BitNet b1.58: an F32 (or F16) GGUF with ternary weights, as the BitNet converters write it, to I2_S.
# The output weights and the token embeddings stay F16.
llama-quantize ggml-model-f32.gguf ggml-model-i2_s.gguf I2_S
# A Sherry-trained checkpoint (for example, AngelSlim/Hy-MT1.5-1.8B-1.25bit) converted to BF16, to STQ1_0
llama-quantize Hy-MT1.5-1.8B-bf16.gguf Hy-MT1.5-1.8B-STQ1_0.gguf STQ1_0
# An STQ1_0 file to another type (requantizing needs --allow-requantize)
llama-quantize --allow-requantize Hy-MT2-1.8B-1.25Bit.gguf Hy-MT2-1.8B-Q4_K_M.gguf Q4_K_M
to_float (0002-bitnet-i2s.patch, part 8).The llamacpp\x64 files of artifacts.zip from the Release job of build 0.2.7.build2427 (MSVC 19.29; commit c68128af, before signing) were tested on a 12-core x64 PC:
SHA256SUMS.txt matches every file of both architectures (x64 22 files, arm64 14).llama-cli with each of three CPU variants. Only the best variant for the PC was kept, or only ggml-cpu-sse42.dll, or only ggml-cpu-x64.dll:
| Variant | BitNet b1.58 2B-4T (I2_S) | Hy-MT2 1.25-bit (STQ1_0), into Korean |
|---|---|---|
| best (AVX2) | The capital of France is Paris. (2.6 s) |
오늘 날씨가 정말 좋으니 공원에서 산책을 하러 가자. (2.8 s) |
sse42 |
same (4.9 s) | same (4.8 s) |
x64 (generic code) |
same (4.3 s) | same (43.8 s) |
I2_S tensors and stopped with <|eot_id|> (token 128009) on every variant.llama-server chat API (best variant): BitNet 25 tokens/s, Hy-MT2 1.25-bit 58 tokens/s.Built with MSVC 19.50 (Visual Studio 2026) on a 12-core PC, with all patches.
llama-server loaded each model and answered “What is the capital of France? Answer in one short sentence.” through /v1/chat/completions (temperature 0, no template or tokenizer options). The CPU variant chosen for that PC has AVX2 and AVX-VNNI.
| Model | Answer | Generation speed |
|---|---|---|
| Qwen3Guard-Gen 0.6B Q4_K_M | Safety: Safe Categories: None (its classifier output) |
71 tokens/s |
| Hy-MT2 1.8B Q4_K_M | Paris is the capital of France. |
32 tokens/s |
| FrogNano 4B 2609 Q4_K_M | The capital of France is Paris., after 182 tokens of reasoning (reasoning_content) |
10 tokens/s |
| BitNet b1.58 2B-4T (I2_S) | The capital of France is Paris., then <|eot_id|> |
24 tokens/s |
Hy-MT2 (translation) with the README’s default prompt (“Translate the following text into {target_lang}. Note that you should only output the translated result without any additional explanation:”), temperature 0. Every file translated these three texts and stopped by itself:
| Text | Into |
|---|---|
| The weather is really nice today, so let’s go for a walk in the park. | Korean |
| 오늘 회의는 오후 세 시로 변경되었습니다. 늦지 않게 참석해 주세요. | English |
| 今天天气真好,我们去公园散步吧。 | English |
| File | Example (into Korean) | Generation speed |
|---|---|---|
tencent/Hy-MT2-1.8B-1.25Bit-GGUF (STQ1_0) |
오늘 날씨가 정말 좋으니 공원에서 산책을 하러 가자. | 28-30 tokens/s |
tencent/Hy-MT2-1.8B-GGUF Q4_K_M |
오늘 날씨가 정말 좋으니, 공원에서 산책하러 가요. | 27 tokens/s |
tencent/Hy-MT2-1.8B-GGUF Q6_K |
오늘 날씨가 정말 좋으니, 공원에서 산책을 하러 가보죠. | 23 tokens/s |
tencent/Hy-MT2-1.8B-GGUF Q8_0 |
오늘 날씨가 정말 좋으니, 공원에서 산책을 하러 가보죠. | 18 tokens/s |
Also checked:
x64 variant. With only ggml-cpu-x64.dll (no AVX2), the BitNet model generated the same tokens as on the AVX2 variant, ending with <|eot_id|>.--override-kv tokenizer.ggml.pre=str:llama-bpe.sandybridge alone: 23-24 tokens/s,sse42 alone: 10 tokens/s,x64 alone (generic code): 2 tokens/s.tl2.llama-cli. The downloaded files were also run directly with llama-cli -st --temp 0 (single turn): every Hy-MT2 file gave the translations above in both directions, and the BitNet model answered the capital question and a two-sentence explanation, stopping by itself.0xAA64), and they import only system DLLs, the VC++/UCRT runtime, and each other. They contain the icon. They could not be run here (x64 PC).llama-quantize.
0002-bitnet-i2s.patch part 9). microsoft/BitNet-b1.58-2B-4T-gguf was converted to F16 (its I2_S weights times the scale) and quantized back with llama-quantize <f16> <out> I2_S. The result has the original file’s tensor types (210 I2_S, F16 token embeddings, 121 F32), the same 2-bit codes in all 210 I2_S tensors, scales within F16 rounding (4.6e-4), and identical other tensors. With 41 1 (one thread) the file is byte-identical. It answered The capital of France is Paris. on the AVX2, sse42, and x64 variants and through llama-server (26 tokens/s).
--temp 0, llama-cli). Four answers were identical. The code prompt (a factorial function) gave a different but also correct answer: a loop instead of recursion.type i2_s unsupported for integer quantization: no dequantization available.llama-quantize --help lists STQ1_0 (44) and I2_S (41). llama-server.exe, llama-cli.exe, and llama-quantize.exe contain the icon.The patches are applied in name order. Each one groups related changes as numbered parts; the header of each patch file describes every part.
| File | Parts |
|---|---|
0001-stq1_0.patch |
1. STQ1_0 (PR #22836), 2. x86 SIMD vec_dot for STQ1_0, 3. STQ1_0 files with type 42 |
0002-bitnet-i2s.patch |
1. build without the BitNet repository, 2. I2_S reference quantization in ggml-base, 3. I2_S fast path without llamafile, 4. non-AVX2 I2_S fallbacks, 5. squared ReLU, 6. chat template, 7. pre-tokenizer (5-7 for BitNet b1.58 2B-4T), 8. I2_S type traits, 9. llama-quantize for I2_S |
0003-windows-build.patch |
1. Windows icon and version information for every executable and DLL, 2. unsupported MSVC options, 3. MSVC conversion warnings, 4. jinja with the Visual Studio 2019 STL |
0004-embeddinggemma2.patch |
1. the gemma-embedding2 architecture, 2. the EmbeddingGemma 2 text encoder (backport of upstream #30054) |
This part adds the STQ1_0 ternary quantization type (1.3125 bits per weight) with an ARM NEON vec_dot kernel. It comes from ggml-org/llama.cpp#22836, PR head 1e411d8f5a1e23525fa3265dfb4bd76265465397. The PR is open and not merged upstream. STQ1_0 is the llama.cpp format of Tencent’s AngelSlim 1.25-bit quantization; the Hy-MT2 1.25-bit model card names this PR as the kernel it needs.
The PR was written against upstream llama.cpp, so it was rebased onto the fork. The differences from the PR are:
llama.cpp numbers types in two separate enums:
enum ggml_type, ggml.h). It is stored for each tensor in a GGUF file, and the loader uses it to read the tensor data.enum llama_ftype, llama.h). It is stored once in the general.file_type metadata, describes the file’s overall quantization, and is the number shown by llama-quantize --help.The fork’s IDs are kept. STQ1_0 takes the next free IDs:
I2_S, I8_S, TL1, and TL2, and llama file types 41-43 for I2_S, TL1, and TL2. It has neither Q2_0 nor F8_E4M3.| Type | ggml tensor type | llama file type |
|---|---|---|
STQ1_0 here |
43 (same as the PR) | 44 (the PR uses 42, which is TL1 in the fork) |
GGML_TYPE_COUNT |
44 |
The same change is made in gguf-py/gguf/constants.py. Every type table in the code is indexed by enum name (designated initializers), so only the enum definitions use the numbers.
Tencent’s files use other IDs:
tencent/Hy-MT2-1.8B-1.25Bit-GGUF stores its STQ1_0 tensors as ggml type 42 (42 bytes per 256 weights, measured from the tensor offsets), and general.file_type 41.TL2 and 41 is I2_S. Part 3 tells them apart by the tensor sizes. Without it, the file fails to load (“failed to load model”).Compatibility:
llama-quantize, 44 means STQ1_0 here. Use the type names instead; see Quantizing with llama-quantize.Q2_0 entries, so the context lines were adjusted and the STQ1_0 lines follow the TQ2_0 lines.ggml/src/ggml-cpu/arch-fallback.h: the fork’s x86 section already has an empty quants.c block, which receives the STQ1_0 generic mapping.gguf-py/gguf/constants.py: the fork defines LlamaFileType twice (the second definition wins). MOSTLY_STQ1_0 is added to both.The PR’s fast kernel is ARM NEON only; part 2 adds one for x86.
The PR includes llama-quantize support (STQ1_0, file type 44 here). Its instructions convert a Sherry-trained checkpoint (for example, AngelSlim/Hy-MT1.5-1.8B-1.25bit) to a BF16 GGUF with convert_hf_to_gguf.py and then run llama-quantize <bf16.gguf> <out.gguf> STQ1_0.
to_float.This part adds ggml_vec_dot_stq1_0_q8_K to ggml/src/ggml-cpu/arch/x86/quants.c.
sse42, sandybridge, and newer variants. GGML_SSE42 is also tested, because MSVC does not define __SSE4_1__. Other CPUs (the x64 variant) still use the generic code.arch-fallback.h that part 1 adds, which mapped the generic function to this name.How it works:
g (0-15) holds weights g + p*16 (p = 0-3). So lane g of a 16-byte vector is group g._mm_shuffle_epi8, one per codebook half, and picks between them by the sign bit._mm_maddubs_epi16.sum(q * y) - sum(y), with sum(y) from the q8_K bsums.The arithmetic is integer, so the result equals the generic code’s. The translations were the same on the AVX2, sandybridge, sse42, and x64 variants.
This part lets the build read STQ1_0 files that use type 42, without changing any type ID.
ggml type 42 has two meanings:
GGML_TYPE_TL2 in the fork (bitnet.cpp’s lookup-table type),tencent/Hy-MT2-1.8B-1.25Bit-GGUF).This build keeps the fork’s TL2 = 42 and numbers STQ1_0 43, as the PR does. So files that follow either numbering keep their meaning.
Tensor type (ggml/src/gguf.cpp). The two formats have different sizes, and a GGUF file’s tensor offsets must match the tensor sizes; gguf_init_from_file_impl checks that right after reading the tensor infos. Before that check, the patch looks at files that have type-42 tensors:
TL2, nothing changes.N tensors of type 42 have the size of stq1_0, not tl2; reading them as stq1_0.File type (src/llama-model-loader.cpp). general.file_type has the same problem with three numbers, so the loader goes by the tensor types:
general.file_type |
Meaning here | Also used by | Reported as |
|---|---|---|---|
| 41 | MOSTLY_I2_S |
Tencent’s STQ1_0 files | MOSTLY_STQ1_0, when the file has STQ1_0 tensors and no I2_S tensors |
| 40 | MOSTLY_Q1_0 |
microsoft/BitNet-b1.58-2B-4T-gguf |
MOSTLY_I2_S, when the file has I2_S tensors and no Q1_0 tensors |
| 42 | MOSTLY_TL1 |
STQ1_0 files made with PR #22836 (MOSTLY_STQ1_0 = 42 there, 44 here) |
MOSTLY_STQ1_0, when the file has STQ1_0 tensors and no TL1 tensors |
The file type is metadata only.
Tested:
STQ1_0 - 1.31 bpw ternary.I2_S (before the patch: Q1_0).general.file_type 42, as PR #22836 writes it, shows the file type STQ1_0 - 1.31 bpw ternary (before: TL1).tl2.This part removes the fork’s dependency on the BitNet repository when TL1/TL2 are off:
ggml/src/CMakeLists.txt adds ../../../../include (BitNet’s headers) to ggml-base only when GGML_BITNET_ARM_TL1 or GGML_BITNET_X86_TL2 is on. ggml/src/ggml-cpu/CMakeLists.txt does the same for ../../../../src/ggml-bitnet-lut.cpp and ../../../../include in every CPU variant.
gemm-config.h. The I2_S kernels include it, so the patch adds ggml/src/ggml-cpu/gemm-config.h, next to ggml-cpu-i2s.c.
include/gemm-config.h at 0b341e5 (MIT, Copyright (c) Microsoft Corporation), with a fallback added at the end.ROW_BLOCK_SIZE and COL_BLOCK_SIZE only for CPUs with AVX/SSSE3 (or NEON). The non-AVX2 path of ggml-cpu-i2s.c uses them, so the baseline x64 variant of GGML_CPU_ALL_VARIANTS, and every MSVC variant below AVX (MSVC does not define __SSSE3__), would not compile. The fallback uses the x86 values.utils/tune_gemm_config.py rewrites this file to tune the block sizes; this build uses the values as published.When the fork is built inside the BitNet repository with TL1 or TL2 on, the CMake change does nothing, and the copied header takes the place of BitNet’s for ggml-cpu-i2s.c (same values).
This part moves dequantize_row_i2_s and quantize_i2_s from ggml/src/ggml-cpu/quants.c (the CPU backend) to ggml/src/ggml-quants.c (ggml-base), and declares them GGML_API.
ggml.c, which is part of ggml-base, uses quantize_i2_s in ggml_quantize_chunk. In the fork it also registered dequantize_row_i2_s as the I2_S to_float, which part 8 removes.GGML_BACKEND_DL on Windows, each CPU variant is a separate DLL, and ggml-base.dll cannot link against it. Without the patch, the MSVC build fails with LNK2001/LNK2019 for these two symbols.Both functions are plain scalar C and are moved unchanged; part 9 then rewrites quantize_i2_s. The CPU backend (ops.cpp uses dequantize_row_i2_s) imports them from ggml-base, like the other reference quantization functions.
This part makes the fork’s I2_S GEMV/GEMM fast path in ggml_compute_forward_mul_mat call ggml_is_contiguous(src1) instead of using src1_cont.
src1_cont is declared only inside the #if GGML_USE_LLAMAFILE block above it.ggml-cpu.c undefines GGML_USE_LLAMAFILE when the target has __ARM_FEATURE_SVE or __ARM_FEATURE_MATMUL_INT8. The ARM64 build (-march=armv8.7-a) failed with “use of undeclared identifier ‘src1_cont’”.GGML_USE_LLAMAFILE stays defined there.This part fixes the code that the x64, sse42, and sandybridge CPU variants (and any CPU without AVX2) use for I2_S. Without it, the 2B-4T model generated only ? tokens on those variants.
ggml_vec_dot_i2_i8_s (ggml-cpu/quants.c). The scalar fallback differed from the AVX2 code in three ways:
gp holds weights gp, gp+32, gp+64, and gp+96 in bits 7-6, 5-4, 3-2, and 1-0, as dequantize_row_i2_s reads them.r at r * (n/4) instead of r * bx / 4.The patch rewrites it to compute what the AVX2 code computes.
ggml_gemm_i2_i8_s (ggml-cpu/ggml-cpu-i2s.c, used for prompts of four or more tokens). With ACT_PARALLEL, which gemm-config.h sets, the fallback called ggml_vec_dot_i2_i8_s as if that function iterated over activation columns with an output stride. It iterates over weight rows and writes them consecutively. The patch calls it once per activation column, which gives the AVX2 code’s layout s[col * bs + row].With the patch, the x64 variant generates the same tokens as the AVX2 variants.
This part makes the bitnet-b1.58 architecture use LLM_FFN_RELU_SQR in its feed-forward network.
bitnet-b1.58 (the architecture of microsoft/BitNet-b1.58-2B-4T-gguf) with the bitnet graph, whose FFN uses SiLU.hidden_act is relu2 in config.json of microsoft/bitnet-b1.58-2B-4T.bitnet_b1_58 models (bitnet) keep SiLU.How the cause was found:
<|eot_id|>, repeating itself until the token limit.<|eot_id|>.This part replaces the chat template of microsoft/BitNet-b1.58-2B-4T-gguf with that of the original model.
Human: ...\n\nBITNETAssistant: , which the model does not follow. Through /v1/chat/completions it invents a conversation instead of answering.microsoft/bitnet-b1.58-2B-4T, tokenizer_config.json) uses User: ...<|eot_id|>Assistant: .llama_model_chat_template returns the original template when the architecture is bitnet-b1.58 and the embedded template contains BITNETAssistant. The function already has a one-off fix of this kind for Mistral-Small-2503.--chat-template and --chat-template-file still take precedence.This part gives bitnet-b1.58 the llama-bpe pre-tokenizer when the file has no tokenizer.ggml.pre.
microsoft/BitNet-b1.58-2B-4T-gguf has no tokenizer.ggml.pre. llama.cpp then falls back to the default pre-tokenizer and warns that generation quality will be degraded.default split I'm, they'll, won't, and blank lines into different tokens than llama-bpe.tokenizer.ggml.pre, or --override-kv tokenizer.ggml.pre=..., is not affected.This part fixes two problems in the fork’s type_traits table in ggml/src/ggml.c, which clang reports:
-Winitializer-overrides. The entries [36], [37], and [38] (the removed IQ4_NL_4_4, IQ4_NL_4_8, and IQ4_NL_8_8) were initialized, then overridden by the fork’s I2_S, I8_S, and TL1, which use the same numbers. The part removes the three placeholders.-Wcast-function-type-mismatch. I2_S registered (ggml_to_float_t) dequantize_row_i2_s, but dequantize_row_i2_s takes a fourth argument, the tensor scale.
ggml_to_float_t, which takes only a row pointer and a count. Called through that type, the scale was undefined.to_float to NULL. get_rows and mul_mat handle I2_S themselves.to_float refuses the type instead of producing wrong values. For example, llama-quantize requantization checks for NULL.This part lets llama-quantize produce I2_S files. bitnet.cpp’s setup_env.py runs llama-quantize <f32> <out> I2_S 1, but with the fork that failed:
llama-quantize rejected I2_S and 41 (“invalid ftype”), and llama-quant.cpp did not map the file type to a tensor type. The fork has the file type LLAMA_FTYPE_MOSTLY_I2_S and an I2_S case in ggml_quantize_chunk.quantize_i2_s packed the 2-bit codes sequentially, but the kernels and dequantize_row_i2_s read 128 weights per 32 bytes (byte gp holds weights gp, gp+32, gp+64, and gp+96). Its output would have given wrong results.llama-quantize splits a tensor into chunks for its threads, and each chunk would have written its own scale.ggml_validate_row_data rejected type 36, so llama-quantize could not read an I2_S file at all. Requantizing one failed with invalid type 36 instead of the message for types without to_float (part 8).The changes:
quantize_i2_s packs the codes in the kernels’ layout and zeroes the rest of the last 32 bytes (the original files leave 28 of them uninitialized).
{-s, 0, +s} with one s per tensor, as the BitNet b1.58 converters write them.s is the largest absolute value. A weight whose magnitude is above s/2 becomes +1 or -1; the others become 0.ggml_quantize_chunk asserts that I2_S starts at row 0.llama-quant.cpp maps the file type to GGML_TYPE_I2_S.
--output-tensor-type and --token-embedding-type still apply.llama-quantize lists I2_S.ggml_validate_row_data accepts I2_S (it checks the scale) and the other BitNet types.llama.cpp has no Windows resource files, so its executables and DLLs have no icon and no version information (the “Description” in the file properties is empty). This part adds cmake/win-resources.cmake, included at the end of the top-level CMakeLists.txt, with these cache options:
| Option | Set by the build script to | Effect |
|---|---|---|
LLAMA_WIN_ICON |
app\favicon.ico |
The icon is embedded as the application icon of the executables. |
LLAMA_WIN_DESCRIPTION |
llama.cpp built for WelsonJS |
A VERSIONINFO resource is added to every executable and DLL. |
LLAMA_WIN_VERSION |
the WelsonJS version, 0.2.7.62 (the launcher’s AssemblyFileVersion, or -ProductVersion) |
FileVersion and ProductVersion (default 0.0.<LLAMA_BUILD_NUMBER>.0). |
LLAMA_WIN_COMPANY |
Catswords Research |
CompanyName. |
LLAMA_WIN_COPYRIGHT |
the authors of the code (below) | LegalCopyright. |
The LegalCopyright value names everyone whose code is in the build:
Copyright (c) 2023-2026 The ggml authors (llama.cpp), Copyright (c) Microsoft Corporation (bitnet.cpp), Copyright (c) 2025 Tencent (AngelSlim, STQ1_0), Copyright (c) 2026 Catswords Research (WelsonJS)
LICENSE-llama.cpp.txt in the output).gemm-config.h (MIT).The VERSIONINFO resource of each binary has these values:
| Value | Example (llama-server.exe) |
Example (ggml-cpu-haswell.dll) |
|---|---|---|
FileDescription (shown as “Description”) |
llama-server - OpenAI-compatible LLM server (llama.cpp built for WelsonJS) |
ggml-cpu-haswell - ggml CPU backend, haswell variant (llama.cpp built for WelsonJS) |
FileVersion, ProductVersion |
0.2.7.62 |
0.2.7.62 |
CompanyName |
Catswords Research |
Catswords Research |
LegalCopyright |
the value above | the value above |
ProductName |
llama.cpp |
llama.cpp |
InternalName, OriginalFilename |
llama-server, llama-server.exe |
ggml-cpu-haswell, ggml-cpu-haswell.dll |
Comments |
llama.cpp built for WelsonJS. llama.cpp 390c307, build 1. |
same |
The roles in FileDescription: ggml tensor library, ggml base library, ggml CPU backend, <variant> variant, llama.cpp library, llama.cpp common library, llama.cpp multimodal library, <tool> implementation library, OpenAI-compatible LLM server, LLM command-line interface, and GGUF model quantizer. A target without a known role gets <target> (<text>).
How the patch does it:
-impl libraries), and gives each its own resource script (<build>/win-resources/<target>.rc). The executables also get the icon.#pragma code_page(65001), so a build folder with non-ASCII characters in its path works regardless of the ANSI code page.FILEOS and FILETYPE use numeric values (VFT_APP for executables, VFT_DLL for DLLs) instead of the winver.h names, so no include path is needed, whichever resource compiler CMake picks.configure_file, which touches a file only when its content changes, so a new CMake run does not relink.Comments is the commit count of the checkout: 1 for the shallow git clone, 0 for the archive source.The icon file stays in the WelsonJS repository; the patch only adds the options. With LLAMA_WIN_ICON and LLAMA_WIN_DESCRIPTION empty (the default), the build is the same as without the patch. A missing icon file or a malformed version stops the configuration with an error.
The build script checks the output: every executable and DLL must have the FileDescription, the version, CompanyName, and LegalCopyright above, or the build fails.
This part removes two sources of MSVC warning D9002 (“ignoring unknown option”) in ggml/src/ggml-cpu/CMakeLists.txt:
| Option | Problem | Change |
|---|---|---|
/arch:SSE4.2 (the sse42 CPU variant) |
Exists only in newer MSVC. MSVC 19.29 (Visual Studio 2019, the AppVeyor image) ignores it for every source file of the variant. MSVC 19.50 accepts it. | Added only when check_c_compiler_flag accepts it; CMake counts D9002 as a failed check. |
-fno-lto (the CPU feature detection objects, ggml-cpu-*-feats) |
A GCC/clang option, ignored by every MSVC. | Passed only when not MSVC. MSVC does not use LTO unless /GL is given. |
Without /arch:SSE4.2, the sse42 variant is compiled for SSE2 code generation. That is safe: MSVC accepts SSE4 intrinsics with any /arch, and the variant is chosen at run time only on CPUs with SSE4.2. It only loses auto-vectorization with SSE4 instructions on the AppVeyor image.
This part handles the many C4244 warnings (“possible loss of data”) that MSVC reports in the llama, mtmd, and other targets. With MSVC 19.50 there are 31 such lines; MSVC 19.29 (AppVeyor) reports more.
std::fill of a float vector with 0.0 or 0 in tools/mtmd/mtmd-audio.cpp,std::pair<int64_t, uint32_t> built from a small int64_t in src/llama-model.cpp.ggml/CMakeLists.txt already disables C4244, C4267, and C4305 (among others) for the ggml targets. The patch does the same in the top-level CMakeLists.txt, before src, common, and tools are added. It applies to MSVC only, so the clang (ARM64) build is not affected.
src/llama-quant.cpp (llama_tensor_dequantize_impl), the worker lambda took the element count as int, but it is passed a size_t. The functions it calls take int64_t: to_float, ggml_fp16_to_fp32_row, and ggml_bf16_to_fp32_row.
llama-quantize requantizes a very large tensor with few threads.int64_t.This part changes value_func_t in common/jinja/value.h, which called std::function::target<func_hptr>() with the function type func_hptr.
reinterpret_cast from 'const void *' to '... (*)(const jinja::func_args &)' casts away qualifiers.std::function stores a plain function as a function pointer, so the target type is func_hptr *.The patch adds function_pointer(), which asks for target<func_hptr *>() and returns the stored function pointer (nullptr for another callable). unique_hash() and equivalent() use it; their comments describe comparing function pointers. Both are unused for now.
The fork predates EmbeddingGemma 2 (google/embeddinggemma-2, Apache-2.0), whose GGUF files (for example ggml-org/embeddinggemma-2-GGUF) have the architecture gemma-embedding2. This patch backports the C++ part of upstream llama.cpp 4fbc76dec5 (ggml-org/llama.cpp#30054, MIT), so llama-server --embedding serves its text embeddings. The launcher runs it that way for the setup component llamacpp\embeddinggemma_2 (see docs/llamacpp-launcher-api.md).
LLM_ARCH_GEMMA_EMBEDDING2 is registered next to LLM_ARCH_GEMMA_EMBEDDING: its name in llama-arch, its model class in llama_model_mapping, the models without a KV cache (create_memory) and the NEOX rope models (llama_model_rope_type).
src/models/gemma-embedding2.cpp and llama_model_gemma_embedding2 in models.h: a bidirectional Gemma 4-style encoder (24 layers, hidden size 512) with symmetric sliding window attention on five of every six layers, per-layer head sizes and KV heads, Q/K/V RMS norms, per-layer inputs from a projection only, a learned output scale per layer, and a projection of the final hidden state to the 768-dimensional embedding. The pooling (mean) comes from the GGUF. The fork already has everything else the model uses: the gemma4 tokenizer, per-layer inputs and per-layer head sizes (Gemma 3n and Gemma 4).
The file is upstream’s, with two changes for the fork’s API:
llama_model_base::load_swa_pattern. The sliding window pattern is read as one flag per layer, as the GGUF stores it (get_arr), or as a period, 6 by default (set_swa_pattern), which is what load_swa_pattern does upstream.build_inp_embd takes no scale. The token embeddings are scaled by sqrt(n_embd) afterwards, for token input only, as the fork’s gemma-embedding.cpp does.models/*.cpp is collected by file(GLOB) in src/CMakeLists.txt, so the new file needs no CMake change. The vision and audio encoders (the mmproj files) and the Python conversion are not part of the patch; the GGUF files are converted upstream.
git ls-tree <BitNet sha> 3rdparty/llama.cpp and BitNet’s .gitmodules. Set commit.txt to it, and RepositoryUrl in the script if the fork URL changed. Set version.txt to <release>.<first seven characters of the commit>.1, where <release> is the last upstream release before the fork diverges: the merge base is merge_base_commit of gh api repos/ggml-org/llama.cpp/compare/master...<fork owner>:llama.cpp:<sha>, and the release is the highest bNNNN tag for which gh api repos/ggml-org/llama.cpp/compare/bNNNN...<merge base> reports behind_by 0.git archive --format=zip --prefix=llama.cpp-<sha>/ <sha>), then update the URL and SHA-256 in source-archive.txt.git -c core.autocrlf=false apply --check. Rebase the patches if needed, and check the type IDs in ggml.h, llama.h, and gguf-py/gguf/constants.py against the fork’s types.0002-bitnet-i2s.patch, part 1), and that the I2_S sources are compiled into every CPU variant.llama-quantize --help lists STQ1_0,llama-server answers a chat request with one of the installed models and with an I2_S model,llama-server --embedding -m embeddinggemma-2-Q8_0.gguf answers POST /v1/embeddings with 768 values per input.native\publish-prebuilt.ps1, see CI) and commit with the new prebuilt\<arch>.sha256 files. The changed files invalidate the CI cache, so CI downloads the new build.