welsonjs

llama.cpp build for WelsonJS

한국어 문서

This folder builds the Windows llama.cpp that WelsonJS uses, with WelsonJS’s patches applied. The base source is the llama.cpp fork that Microsoft distributes with bitnet.cpp as its 3rdparty/llama.cpp submodule. The fork was created to support bitnet.cpp’s quantization. (The repository itself is hosted on a contributor’s account, isHuangXin/llama.cpp; see Files.) The output contains the same files as the official llama-<version>-bin-win-cpu-<arch>.zip package: llama-server, llama-cli, llama-quantize, and the ggml backends. The x64 build has every x86-64 CPU variant, chosen at run time; the arm64 build has one CPU backend.

The fork adds BitNet’s I2_S type and kernels, so besides the usual GGUF files the build loads 1.58-bit BitNet models such as microsoft/BitNet-b1.58-2B-4T-gguf, which upstream llama.cpp cannot load.

On top of the fork, this build adds the STQ1_0 ternary type and kernels from Tencent’s AngelSlim quantization (ggml-org/llama.cpp#22836, not merged upstream; see 0001-stq1_0.patch). It loads 1.25-bit models produced by AngelSlim, such as tencent/Hy-MT2-1.8B-1.25Bit-GGUF, which neither the fork nor upstream llama.cpp can load.

The launcher runs llama-server in router mode, with the installed models in %APPDATA%\WelsonJS, and uses only its HTTP API; requests name their model (docs/llamacpp-launcher-api.md).

Files

File Purpose
commit.txt The llama.cpp commit that is built, as a full 40-character SHA. The source is fetched by this SHA rather than by a branch, because a branch can move and a commit cannot.
version.txt The version label, <upstream release>.<commit>.1 (llama.cpp records no version in its tree, so the release stands for it): the last upstream llama.cpp release (bNNNN) before the commit where the fork diverges from upstream, the first seven characters of the pinned commit (commit.txt), and .1 for the WelsonJS build. It is a label only (BUILDINFO.txt and the build ID) and does not select the source.
source-archive.txt The fallback source archive (url=) and its SHA-256 (sha256=), used when git cannot fetch the commit.
patches/*.patch Patches applied in name order with git apply. They are stored with LF line endings, and .gitattributes keeps them byte-exact on checkout.
build-llamacpp.ps1 Fetches the source, applies the patches, builds, and writes the output.

Current pin: commit 390c307752ab78fd8189f359d6954c9ba1be74af of https://github.com/isHuangXin/llama.cpp.git (branch release-bitnet-embedding-0.6b-270m, 2026-07-15). It is the submodule commit of BitNet 0b341e582afbf9e1011f24744b554c96a3477eb5. It diverges from upstream llama.cpp (ggml-org/llama.cpp) after 665892536dfb (2026-07-04, 46 commits of its own); the last upstream release before that commit is b9871, so version.txt is b9871.390c307.1.

The fork is older than upstream llama.cpp, so upstream changes made after it are not included.

Why the BitNet repository is not used

bitnet.cpp builds the fork as a subproject of the BitNet repository, which adds two things:

The I2_S type and kernels (ggml/src/ggml-cpu/ggml-cpu-i2s.c, AVX2, and AVX-VNNI or AVX512-VNNI when available) are part of the fork. They need one header from the BitNet repository, gemm-config.h (the kernels’ block sizes), which 0002-bitnet-i2s.patch (part 1) copies into the fork.

The other BitNet files are used only inside #if defined(GGML_BITNET_ARM_TL1) || defined(GGML_BITNET_X86_TL2) in the fork’s code. Only the fork’s CMake refers to them unconditionally, and 0002-bitnet-i2s.patch (part 1) changes that. So with TL1/TL2 off, the fork builds on its own, with MSVC like upstream llama.cpp.

Build

powershell -ExecutionPolicy Bypass -NoProfile -File native\llamacpp\build-llamacpp.ps1 `
    [-Arch x64|x86|arm64] [-OutDir <dir>] [-WorkDir <dir>] [-Source auto|git|archive] [-LlvmDir <dir>] [-InstallLlvm] [-Force]

Requirements:

Architectures

  x64 arm64
Compiler MSVC, Visual Studio generator clang (LLVM) with the fork’s cmake\arm64-windows-llvm.cmake toolchain, Ninja
Environment Visual Studio default Visual Studio x64_arm64 developer environment (ARM64 libraries, x64-hosted ARM64 linker); the MSVC version with the ARM64 C runtime is passed to vcvarsall.bat
CPU target every x86-64 variant (GGML_CPU_ALL_VARIANTS), chosen at run time -march=armv8.7-a, one backend ggml-cpu.dll (GGML_CPU_ALL_VARIANTS is not supported for ARM on Windows)
OpenMP on (VCOMP140.DLL from the VC++ redistributable) off: with clang it needs libomp140.aarch64.dll, which Visual Studio ships only as debug_nonredist; ggml uses its own thread pool
Host tool (llama-ui-embed) built as part of the build built for x64 with host-cxx.cmd in the work folder (HOST_CXX_COMPILER), see below

x86 (32-bit) is built like x64, with MSVC and the Visual Studio generator (-A Win32), but with one CPU backend (ggml-cpu.dll) with the AVX2 baseline of GGML_NATIVE=OFF and without BMI2 (GGML_CPU_ALL_VARIANTS=OFF, GGML_BMI2=OFF): ggml’s BMI2 code uses _pdep_u64, which 32-bit MSVC lacks, and several of the CPU variants enable BMI2. The executables are linked with /LARGEADDRESSAWARE, so a 32-bit process gets 4 GB of address space on 64-bit Windows instead of 2 GB; without it, the model and the KV cache of an 8192-token context (the launcher’s LlamaCppContextSize) could not be allocated. The x86 build gives the same greedy answers as the x64 build (Qwen3Guard 0.6B Q4_K_M, Hy-MT2 1.8B Q4_K_M and BitNet b1.58 2B I2_S), with an 8192-token context. A larger context or model may not fit in 4 GB.

Source

Configuration

Output and skipping the build

The files are written to <OutDir>\<arch>; in CI this is artifacts\llamacpp\x64, artifacts\llamacpp\arm64 and artifacts\llamacpp\x86. No archive is created. The folder also contains:

The build ID is a SHA-256 over these inputs:

The script skips the build when BUILDINFO.txt has the same build ID and every file matches SHA256SUMS.txt. Otherwise, for example when an input changed or a file was modified, it builds again. -Force always builds.

The following are not part of the build ID:

CI (AppVeyor)

The binaries are not built in CI: building llama.cpp and CrispEmbed for x64, arm64 and x86 takes longer than the 60-minute AppVeyor job limit. They are built on a developer PC with this script, published to the WelsonJS blob storage, and restored by the Release job, which packages and signs them with the other artifacts.

  1. Build each architecture: -Arch x64, -Arch arm64 and -Arch x86 (on a PC whose CMake on PATH defaults to another generator, pass -Generator "Visual Studio 18 2026" for x64 and x86).
  2. Publish each build: native\publish-prebuilt.ps1 -Component llamacpp -Arch <arch> -BuildDir <OutDir>\<arch>.
    • It checks that the folder matches its SHA256SUMS.txt, and that BUILDINFO.txt names the current version.txt, commit.txt and architecture.
    • It uploads each file as it is and gzip-compressed (<file>.gz) with AzCopy (azcopy login first) to packages/native/llamacpp/<version>-<arch>/, never overwriting an existing blob, so a published build does not change.
    • It copies SHA256SUMS.txt to native\llamacpp\prebuilt\<arch>.sha256.
  3. Commit the three prebuilt\<arch>.sha256 files with the change.

The Debug and Release jobs run native\restore-prebuilt.ps1 -Component llamacpp -Arch <arch> for each architecture, into artifacts\llamacpp\<arch>. It downloads the files listed in prebuilt\<arch>.sha256 from <version>-<arch> as <file>.gz and decompresses them (when a .gz cannot be downloaded or used, it downloads the file itself), and uses a file only if its SHA-256 matches, so the repository, not the storage, decides what is shipped. Both packages are signed (Debug with the test-signing policy, Release with release-signing). Each job caches artifacts\llamacpp, keyed on version.txt and the three prebuilt\<arch>.sha256 files: when they are unchanged, the restored folders match their records and nothing is downloaded.

.artifact-configuration.xml lists every executable and DLL of the three architectures by name (no wildcards), and the build script fails when a built binary is not listed, so a new binary cannot be left unsigned. Authenticode signing changes the files, so in the signed package SHA256SUMS.txt lists the hashes of the unsigned files.

The current builds (b9871.390c307.1) were made with Visual Studio 2026: MSVC 19.50 (toolset 14.50) for x64 and x86, and clang 23 with the MSVC 14.51 libraries for arm64. A PC that runs them needs a Visual C++ 2015-2026 redistributable at least as new as those tools.

Installation

The installer uses this build; it no longer downloads the official llama.cpp package (data\DownloadUrls.psd1 has no llamacpp entry).

Compatibility with upstream llama.cpp

The fork numbers some ggml tensor types differently from upstream llama.cpp:

ggml tensor type This build (fork) Upstream llama.cpp
36 I2_S (BitNet) unused
37 I8_S (BitNet, run time only) unused
38 TL1 (BitNet) unused
42 TL2 (BitNet), or STQ1_0 in Tencent’s files (detected by size, 0001-stq1_0.patch part 3) Q2_0
43 STQ1_0 (0001-stq1_0.patch, the PR’s ID) F8_E4M3

BitNet (I2_S) models

microsoft/BitNet-b1.58-2B-4T-gguf works through llama-server’s chat API with no extra options. That depends on parts 4-7 of 0002-bitnet-i2s.patch, which fix four problems found while testing it:

Problem without the patch Cause Part
Answers the first sentence, then repeats itself until the token limit The fork’s FFN uses SiLU for the 2B-4T model, which uses squared ReLU 5
Only ? tokens on CPUs without AVX2 (the x64, sse42, and sandybridge variants) The fork’s non-AVX2 I2_S fallbacks compute something else than its AVX2 code 4
Invents a conversation through /v1/chat/completions The GGUF file embeds a chat template the model does not follow 6
Warning “GENERATION QUALITY WILL BE DEGRADED”; some text tokenized differently The GGUF file has no tokenizer.ggml.pre 7

The I2_S kernels themselves were correct. A GGUF with the I2_S weights converted to F16 generated the same tokens as the I2_S file.

general.file_type of the GGUF file is 40, which is Q1_0 in the fork. 0001-stq1_0.patch (part 3) reports it as I2_S, because the file has I2_S tensors and no Q1_0 tensors. The file type is metadata only and does not affect loading.

Quantizing with llama-quantize

llama-quantize.exe is installed with the other tools in %APPDATA%\WelsonJS\llamacpp, which is on the user’s PATH. llama-quantize --help lists the types it accepts.

Select the type by name, not by number. llama-quantize accepts both: it first looks for the name (case does not matter), then for a number in its list of file types. The numbers of the BitNet and STQ1_0 types differ between builds, so a number from another build or file can select a different type:

Number This build Elsewhere
41 I2_S general.file_type 41 in Tencent’s 1.25-bit files means STQ1_0
42 not accepted (TL1, not in the list) STQ1_0 in PR #22836
44 STQ1_0 STQ1_0 is 42 in PR #22836 and 41 in Tencent’s files

For example, 41 taken from a Tencent file quantizes to I2_S, not STQ1_0, and damages a model that is not ternary. The names I2_S and STQ1_0 select the same type in every build that has it.

Examples:

# BitNet b1.58: an F32 (or F16) GGUF with ternary weights, as the BitNet converters write it, to I2_S.
# The output weights and the token embeddings stay F16.
llama-quantize ggml-model-f32.gguf ggml-model-i2_s.gguf I2_S

# A Sherry-trained checkpoint (for example, AngelSlim/Hy-MT1.5-1.8B-1.25bit) converted to BF16, to STQ1_0
llama-quantize Hy-MT1.5-1.8B-bf16.gguf Hy-MT1.5-1.8B-STQ1_0.gguf STQ1_0

# An STQ1_0 file to another type (requantizing needs --allow-requantize)
llama-quantize --allow-requantize Hy-MT2-1.8B-1.25Bit.gguf Hy-MT2-1.8B-Q4_K_M.gguf Q4_K_M

Test results

AppVeyor build

The llamacpp\x64 files of artifacts.zip from the Release job of build 0.2.7.build2427 (MSVC 19.29; commit c68128af, before signing) were tested on a 12-core x64 PC:

Local build

Built with MSVC 19.50 (Visual Studio 2026) on a 12-core PC, with all patches.

llama-server loaded each model and answered “What is the capital of France? Answer in one short sentence.” through /v1/chat/completions (temperature 0, no template or tokenizer options). The CPU variant chosen for that PC has AVX2 and AVX-VNNI.

Model Answer Generation speed
Qwen3Guard-Gen 0.6B Q4_K_M Safety: Safe Categories: None (its classifier output) 71 tokens/s
Hy-MT2 1.8B Q4_K_M Paris is the capital of France. 32 tokens/s
FrogNano 4B 2609 Q4_K_M The capital of France is Paris., after 182 tokens of reasoning (reasoning_content) 10 tokens/s
BitNet b1.58 2B-4T (I2_S) The capital of France is Paris., then <|eot_id|> 24 tokens/s

Hy-MT2 (translation) with the README’s default prompt (“Translate the following text into {target_lang}. Note that you should only output the translated result without any additional explanation:”), temperature 0. Every file translated these three texts and stopped by itself:

Text Into
The weather is really nice today, so let’s go for a walk in the park. Korean
오늘 회의는 오후 세 시로 변경되었습니다. 늦지 않게 참석해 주세요. English
今天天气真好,我们去公园散步吧。 English
File Example (into Korean) Generation speed
tencent/Hy-MT2-1.8B-1.25Bit-GGUF (STQ1_0) 오늘 날씨가 정말 좋으니 공원에서 산책을 하러 가자. 28-30 tokens/s
tencent/Hy-MT2-1.8B-GGUF Q4_K_M 오늘 날씨가 정말 좋으니, 공원에서 산책하러 가요. 27 tokens/s
tencent/Hy-MT2-1.8B-GGUF Q6_K 오늘 날씨가 정말 좋으니, 공원에서 산책을 하러 가보죠. 23 tokens/s
tencent/Hy-MT2-1.8B-GGUF Q8_0 오늘 날씨가 정말 좋으니, 공원에서 산책을 하러 가보죠. 18 tokens/s

Also checked:

Patches

The patches are applied in name order. Each one groups related changes as numbered parts; the header of each patch file describes every part.

File Parts
0001-stq1_0.patch 1. STQ1_0 (PR #22836), 2. x86 SIMD vec_dot for STQ1_0, 3. STQ1_0 files with type 42
0002-bitnet-i2s.patch 1. build without the BitNet repository, 2. I2_S reference quantization in ggml-base, 3. I2_S fast path without llamafile, 4. non-AVX2 I2_S fallbacks, 5. squared ReLU, 6. chat template, 7. pre-tokenizer (5-7 for BitNet b1.58 2B-4T), 8. I2_S type traits, 9. llama-quantize for I2_S
0003-windows-build.patch 1. Windows icon and version information for every executable and DLL, 2. unsupported MSVC options, 3. MSVC conversion warnings, 4. jinja with the Visual Studio 2019 STL
0004-embeddinggemma2.patch 1. the gemma-embedding2 architecture, 2. the EmbeddingGemma 2 text encoder (backport of upstream #30054)

0001-stq1_0.patch

Part 1: STQ1_0 (PR #22836)

This part adds the STQ1_0 ternary quantization type (1.3125 bits per weight) with an ARM NEON vec_dot kernel. It comes from ggml-org/llama.cpp#22836, PR head 1e411d8f5a1e23525fa3265dfb4bd76265465397. The PR is open and not merged upstream. STQ1_0 is the llama.cpp format of Tencent’s AngelSlim 1.25-bit quantization; the Hy-MT2 1.25-bit model card names this PR as the kernel it needs.

The PR was written against upstream llama.cpp, so it was rebased onto the fork. The differences from the PR are:

Type IDs

llama.cpp numbers types in two separate enums:

The fork’s IDs are kept. STQ1_0 takes the next free IDs:

Type ggml tensor type llama file type
STQ1_0 here 43 (same as the PR) 44 (the PR uses 42, which is TL1 in the fork)
GGML_TYPE_COUNT 44  

The same change is made in gguf-py/gguf/constants.py. Every type table in the code is indexed by enum name (designated initializers), so only the enum definitions use the numbers.

Tencent’s files use other IDs:

Compatibility:

Other adjustments

The PR’s fast kernel is ARM NEON only; part 2 adds one for x86.

Quantizing with llama-quantize

The PR includes llama-quantize support (STQ1_0, file type 44 here). Its instructions convert a Sherry-trained checkpoint (for example, AngelSlim/Hy-MT1.5-1.8B-1.25bit) to a BF16 GGUF with convert_hf_to_gguf.py and then run llama-quantize <bf16.gguf> <out.gguf> STQ1_0.

Part 2: x86 SIMD vec_dot for STQ1_0

This part adds ggml_vec_dot_stq1_0_q8_K to ggml/src/ggml-cpu/arch/x86/quants.c.

How it works:

  1. In each 64-weight chunk, group g (0-15) holds weights g + p*16 (p = 0-3). So lane g of a 16-byte vector is group g.
  2. The kernel builds the 16 five-bit codebook indices of a chunk (4-bit code and sign bit). It looks them up with two _mm_shuffle_epi8, one per codebook half, and picks between them by the sign bit.
  3. It multiplies each 2-bit field with 16 activations using _mm_maddubs_epi16.
  4. The codes are 0/1/2 for -1/0/+1, so the result is sum(q * y) - sum(y), with sum(y) from the q8_K bsums.

The arithmetic is integer, so the result equals the generic code’s. The translations were the same on the AVX2, sandybridge, sse42, and x64 variants.

Part 3: STQ1_0 files with type 42

This part lets the build read STQ1_0 files that use type 42, without changing any type ID.

ggml type 42 has two meanings:

This build keeps the fork’s TL2 = 42 and numbers STQ1_0 43, as the PR does. So files that follow either numbering keep their meaning.

Tensor type (ggml/src/gguf.cpp). The two formats have different sizes, and a GGUF file’s tensor offsets must match the tensor sizes; gguf_init_from_file_impl checks that right after reading the tensor infos. Before that check, the patch looks at files that have type-42 tensors:

File type (src/llama-model-loader.cpp). general.file_type has the same problem with three numbers, so the loader goes by the tensor types:

general.file_type Meaning here Also used by Reported as
41 MOSTLY_I2_S Tencent’s STQ1_0 files MOSTLY_STQ1_0, when the file has STQ1_0 tensors and no I2_S tensors
40 MOSTLY_Q1_0 microsoft/BitNet-b1.58-2B-4T-gguf MOSTLY_I2_S, when the file has I2_S tensors and no Q1_0 tensors
42 MOSTLY_TL1 STQ1_0 files made with PR #22836 (MOSTLY_STQ1_0 = 42 there, 44 here) MOSTLY_STQ1_0, when the file has STQ1_0 tensors and no TL1 tensors

The file type is metadata only.

Tested:

0002-bitnet-i2s.patch

Part 1: build without the BitNet repository

This part removes the fork’s dependency on the BitNet repository when TL1/TL2 are off:

When the fork is built inside the BitNet repository with TL1 or TL2 on, the CMake change does nothing, and the copied header takes the place of BitNet’s for ggml-cpu-i2s.c (same values).

Part 2: I2_S reference quantization in ggml-base

This part moves dequantize_row_i2_s and quantize_i2_s from ggml/src/ggml-cpu/quants.c (the CPU backend) to ggml/src/ggml-quants.c (ggml-base), and declares them GGML_API.

Both functions are plain scalar C and are moved unchanged; part 9 then rewrites quantize_i2_s. The CPU backend (ops.cpp uses dequantize_row_i2_s) imports them from ggml-base, like the other reference quantization functions.

Part 3: I2_S fast path without llamafile

This part makes the fork’s I2_S GEMV/GEMM fast path in ggml_compute_forward_mul_mat call ggml_is_contiguous(src1) instead of using src1_cont.

Part 4: non-AVX2 I2_S fallbacks

This part fixes the code that the x64, sse42, and sandybridge CPU variants (and any CPU without AVX2) use for I2_S. Without it, the 2B-4T model generated only ? tokens on those variants.

With the patch, the x64 variant generates the same tokens as the AVX2 variants.

Part 5: squared ReLU for BitNet b1.58 2B-4T

This part makes the bitnet-b1.58 architecture use LLM_FFN_RELU_SQR in its feed-forward network.

How the cause was found:

Part 6: chat template for BitNet b1.58 2B-4T

This part replaces the chat template of microsoft/BitNet-b1.58-2B-4T-gguf with that of the original model.

Part 7: pre-tokenizer for BitNet b1.58 2B-4T

This part gives bitnet-b1.58 the llama-bpe pre-tokenizer when the file has no tokenizer.ggml.pre.

Part 8: I2_S type traits

This part fixes two problems in the fork’s type_traits table in ggml/src/ggml.c, which clang reports:

Part 9: llama-quantize for I2_S

This part lets llama-quantize produce I2_S files. bitnet.cpp’s setup_env.py runs llama-quantize <f32> <out> I2_S 1, but with the fork that failed:

The changes:

0003-windows-build.patch

Part 1: Windows icon and version information

llama.cpp has no Windows resource files, so its executables and DLLs have no icon and no version information (the “Description” in the file properties is empty). This part adds cmake/win-resources.cmake, included at the end of the top-level CMakeLists.txt, with these cache options:

Option Set by the build script to Effect
LLAMA_WIN_ICON app\favicon.ico The icon is embedded as the application icon of the executables.
LLAMA_WIN_DESCRIPTION llama.cpp built for WelsonJS A VERSIONINFO resource is added to every executable and DLL.
LLAMA_WIN_VERSION the WelsonJS version, 0.2.7.62 (the launcher’s AssemblyFileVersion, or -ProductVersion) FileVersion and ProductVersion (default 0.0.<LLAMA_BUILD_NUMBER>.0).
LLAMA_WIN_COMPANY Catswords Research CompanyName.
LLAMA_WIN_COPYRIGHT the authors of the code (below) LegalCopyright.

The LegalCopyright value names everyone whose code is in the build:

Copyright (c) 2023-2026 The ggml authors (llama.cpp), Copyright (c) Microsoft Corporation (bitnet.cpp), Copyright (c) 2025 Tencent (AngelSlim, STQ1_0), Copyright (c) 2026 Catswords Research (WelsonJS)

The VERSIONINFO resource of each binary has these values:

Value Example (llama-server.exe) Example (ggml-cpu-haswell.dll)
FileDescription (shown as “Description”) llama-server - OpenAI-compatible LLM server (llama.cpp built for WelsonJS) ggml-cpu-haswell - ggml CPU backend, haswell variant (llama.cpp built for WelsonJS)
FileVersion, ProductVersion 0.2.7.62 0.2.7.62
CompanyName Catswords Research Catswords Research
LegalCopyright the value above the value above
ProductName llama.cpp llama.cpp
InternalName, OriginalFilename llama-server, llama-server.exe ggml-cpu-haswell, ggml-cpu-haswell.dll
Comments llama.cpp built for WelsonJS. llama.cpp 390c307, build 1. same

The roles in FileDescription: ggml tensor library, ggml base library, ggml CPU backend, <variant> variant, llama.cpp library, llama.cpp common library, llama.cpp multimodal library, <tool> implementation library, OpenAI-compatible LLM server, LLM command-line interface, and GGUF model quantizer. A target without a known role gets <target> (<text>).

How the patch does it:

The icon file stays in the WelsonJS repository; the patch only adds the options. With LLAMA_WIN_ICON and LLAMA_WIN_DESCRIPTION empty (the default), the build is the same as without the patch. A missing icon file or a malformed version stops the configuration with an error.

The build script checks the output: every executable and DLL must have the FileDescription, the version, CompanyName, and LegalCopyright above, or the build fails.

Part 2: unsupported MSVC options

This part removes two sources of MSVC warning D9002 (“ignoring unknown option”) in ggml/src/ggml-cpu/CMakeLists.txt:

Option Problem Change
/arch:SSE4.2 (the sse42 CPU variant) Exists only in newer MSVC. MSVC 19.29 (Visual Studio 2019, the AppVeyor image) ignores it for every source file of the variant. MSVC 19.50 accepts it. Added only when check_c_compiler_flag accepts it; CMake counts D9002 as a failed check.
-fno-lto (the CPU feature detection objects, ggml-cpu-*-feats) A GCC/clang option, ignored by every MSVC. Passed only when not MSVC. MSVC does not use LTO unless /GL is given.

Without /arch:SSE4.2, the sse42 variant is compiled for SSE2 code generation. That is safe: MSVC accepts SSE4 intrinsics with any /arch, and the variant is chosen at run time only on CPUs with SSE4.2. It only loses auto-vectorization with SSE4 instructions on the AppVeyor image.

Part 3: MSVC conversion warnings (C4244)

This part handles the many C4244 warnings (“possible loss of data”) that MSVC reports in the llama, mtmd, and other targets. With MSVC 19.50 there are 31 such lines; MSVC 19.29 (AppVeyor) reports more.

Part 4: jinja with the Visual Studio 2019 STL

This part changes value_func_t in common/jinja/value.h, which called std::function::target<func_hptr>() with the function type func_hptr.

The patch adds function_pointer(), which asks for target<func_hptr *>() and returns the stored function pointer (nullptr for another callable). unique_hash() and equivalent() use it; their comments describe comparing function pointers. Both are unused for now.

0004-embeddinggemma2.patch

The fork predates EmbeddingGemma 2 (google/embeddinggemma-2, Apache-2.0), whose GGUF files (for example ggml-org/embeddinggemma-2-GGUF) have the architecture gemma-embedding2. This patch backports the C++ part of upstream llama.cpp 4fbc76dec5 (ggml-org/llama.cpp#30054, MIT), so llama-server --embedding serves its text embeddings. The launcher runs it that way for the setup component llamacpp\embeddinggemma_2 (see docs/llamacpp-launcher-api.md).

Part 1: the gemma-embedding2 architecture

LLM_ARCH_GEMMA_EMBEDDING2 is registered next to LLM_ARCH_GEMMA_EMBEDDING: its name in llama-arch, its model class in llama_model_mapping, the models without a KV cache (create_memory) and the NEOX rope models (llama_model_rope_type).

Part 2: the EmbeddingGemma 2 text encoder

src/models/gemma-embedding2.cpp and llama_model_gemma_embedding2 in models.h: a bidirectional Gemma 4-style encoder (24 layers, hidden size 512) with symmetric sliding window attention on five of every six layers, per-layer head sizes and KV heads, Q/K/V RMS norms, per-layer inputs from a projection only, a learned output scale per layer, and a projection of the final hidden state to the 768-dimensional embedding. The pooling (mean) comes from the GGUF. The fork already has everything else the model uses: the gemma4 tokenizer, per-layer inputs and per-layer head sizes (Gemma 3n and Gemma 4).

The file is upstream’s, with two changes for the fork’s API:

models/*.cpp is collected by file(GLOB) in src/CMakeLists.txt, so the new file needs no CMake change. The vision and audio encoders (the mmproj files) and the Python conversion are not part of the patch; the GGUF files are converted upstream.

Updating

  1. Find the llama.cpp commit that the new BitNet commit uses: git ls-tree <BitNet sha> 3rdparty/llama.cpp and BitNet’s .gitmodules. Set commit.txt to it, and RepositoryUrl in the script if the fork URL changed. Set version.txt to <release>.<first seven characters of the commit>.1, where <release> is the last upstream release before the fork diverges: the merge base is merge_base_commit of gh api repos/ggml-org/llama.cpp/compare/master...<fork owner>:llama.cpp:<sha>, and the release is the highest bNNNN tag for which gh api repos/ggml-org/llama.cpp/compare/bNNNN...<merge base> reports behind_by 0.
  2. Upload the source archive for that commit (git archive --format=zip --prefix=llama.cpp-<sha>/ <sha>), then update the URL and SHA-256 in source-archive.txt.
  3. Check that each patch applies to a clean checkout with git -c core.autocrlf=false apply --check. Rebase the patches if needed, and check the type IDs in ggml.h, llama.h, and gguf-py/gguf/constants.py against the fork’s types.
  4. Check that the fork still keeps every use of the BitNet repository behind the TL1/TL2 options (0002-bitnet-i2s.patch, part 1), and that the I2_S sources are compiled into every CPU variant.
  5. Build locally with the script, for x64, arm64 and x86. Check that:
    • llama-quantize --help lists STQ1_0,
    • llama-server answers a chat request with one of the installed models and with an I2_S model,
    • llama-server --embedding -m embeddinggemma-2-Q8_0.gguf answers POST /v1/embeddings with 768 values per input.
  6. Publish the three builds (native\publish-prebuilt.ps1, see CI) and commit with the new prebuilt\<arch>.sha256 files. The changed files invalidate the CI cache, so CI downloads the new build.