The files you will produce

A checkpoint is a set of files containing the learned numbers, or weights, that a model uses to answer a request. Quantization stores many of those numbers with fewer bits, reducing disk space and memory use. It changes the stored weights; it is not additional training.

GGUF is the model-file format used by llama.cpp and several programs built on that runtime. MLX is Apple's machine-learning framework; mlx-lm loads and converts compatible language models for Apple Silicon. Their quantizers have different layouts and rules. An 8-bit GGUF and an 8-bit MLX checkpoint are separate products, even when they share one source.

The four GGUFs total 148.68 GB. The accepted importance matrix adds 27.17 MB. Keep the original BF16 source alongside the derived files. Each lower-precision model starts from its full-precision reference. Converting a 4-bit file back to 8-bit cannot restore the information lost during the first conversion.

A complete checkpoint and a tested context limit describe different things. Every original parameter is present here. The tested prompt capacity is lower than the publisher's maximum metadata value.

Key terms

What the operation cost

The example used the East US Linux retail rate recorded for Standard_NC24ads_A100_v4: $3.673 per allocated hour. This size provides 24 virtual CPUs, 220 GiB host RAM and one A100 with an 80-GB product label. The two brief storage-inventory sessions used Standard_D4as_v5 at $0.172/hour. A D4 has no GPU; it cannot perform the CUDA acceptance tests.

The rows use decimal GB and a 730-hour planning month. Storage uses the full 5.99-hour recorded window because a persistent disk continues billing during gaps between compute allocations; compute uses only its 5.05 allocated hours. The disk row is an hourly-normalized estimate for the tutorial's operation, not the full monthly storage bill or a reconstructed invoice. A retained disk continues billing outside those hours. Azure's disk meters and billing period determine the account's actual proration. The recorded machine used an existing disk expanded from 512 GiB to 1024 GiB; the table charges the full 1-TiB capacity during the operation so a reader starting with a fresh VM does not inherit the example's cheaper baseline.

The 148.709-GB delivery quantity includes the four accepted files and the accepted matrix, with resumed data counted only once. Failed transfers, SSH control traffic and protocol overhead add bytes. Metered disk operations, other account-specific meters, public-IP charges, tax, discounts and optional services are not included in the operation range. The $23–$32 range is a planning estimate, not an invoice total or a maximum bill. Staff time is excluded.

Allocated compute time includes compatibility diagnosis, builds, test retries, verification and delivery, not just quantization.

At the same normalized disk rate, continued 1-TiB retention is about $0.105/hour or $2.52/day. Public IPs and other retained resources can also bill separately after compute is released.

Check the VM rate, disk meter, bandwidth route and remaining monthly allowance in your Azure account before starting. Keep the estimated disk and transfer cost in the budget while the VM is running. An Azure budget alert is a notification; it does not stop the VM for you.

Choose the two machines

Use an Azure VM for CUDA validation and the Mac for MLX. A VM is a rented computer in Microsoft's data center. Its CPU, system RAM, GPU and disk are separate resources. The 220 GiB of host RAM does not turn an A100's GPU memory into 220 GiB.

For this exact example, select Standard_NC24ads_A100_v4 with one full A100. The Mac was an M4 Max with 128 GiB unified memory. Unified memory is shared by Apple's CPU and GPU. A smaller Mac may run a quantized model but cannot be assumed to complete the original BF16 baseline and the same 64K tests.

Reserve at least 400 GB of available local disk before starting the downloads and at least 300 GB of free remote space before the heavy writes. These are headroom checks, not estimates of the final footprint. Keep room for partial downloads, runtime builds, the BF16 reference, the matrix and all derived artifacts. The recorded VM finished with approximately 352.8 GB free on a 1-TiB OS disk. It needed no 2-TiB expansion.

Choose a dedicated VM when possible. On a shared VM, identify other users, running jobs, anything that must stay running, and persistent storage before any resize or shutdown. A successful quota check means your account may request the VM size; it does not guarantee that the region has a machine available. See Create the Azure VM for allocation failures.

Prepare accounts and existing tools

You need an Azure account with permission to create or operate the selected VM and disks, and access to the public publisher files. Use the Azure portal's Virtual machines service for this workflow. A managed model endpoint in Microsoft Foundry is a different product; it does not supply the local GPU filesystem and build environment used here.

On the Mac, install Python 3.13, Git, CMake, Apple's command-line developer tools and the Azure CLI through their official distribution instructions. Homebrew is a package installer for macOS. If it is not installed, follow the Installation section at https://brew.sh and its Apple Silicon shell-path instructions, then reopen Terminal. Confirm brew --version works. With Homebrew ready, install these existing tools:

Mac terminal:

xcode-select --install
brew install python@3.13 cmake rsync azure-cli

Complete the developer-tools installer before compiling. Do not run the installer again if the tools are already installed. A current rsync installation is useful for resumable delivery. Keep Azure sign-in on the management side. Do not copy its credential cache, SSH agent socket, keychain or cloud environment variables into the inference process.

Check the tools in the Mac terminal:

Mac terminal:

python3.13 --version
git --version
cmake --version
az version
rsync --version

Sign in to Azure through the CLI's interactive browser or device-code flow. Finish that sign-in before running resource commands. List the subscriptions available to your account and choose your own target; no account identifier from the worked example is needed.

Mac terminal:

az login
az account list --output table
az account set --subscription YOUR_SUBSCRIPTION_ID

YOUR_SUBSCRIPTION_ID is an instruction to enter your own existing identifier, not a value to copy. Keep the Azure terminal separate from the restricted model environment.

Create an isolated working directory on the Mac

Pick a directory that is not synced to a cloud backup service during testing. The examples use a new directory under your home directory. They read the standard HOME variable without changing it.

Mac terminal:

export WORKSPACE="$HOME/nanbeige_work"
mkdir -p "$WORKSPACE"
cd "$WORKSPACE"
export NANBEIGE_ARTIFACT_ROOT="$WORKSPACE/nanbeige_4_3_33b"

Download nanbeige_azure_gguf_mlx.zip from the reference package link, then extract it into WORKSPACE. The ZIP has no enclosing top-level directory:

Mac terminal:

cd "$WORKSPACE"
unzip -q "$HOME/Downloads/nanbeige_azure_gguf_mlx.zip" -d "$WORKSPACE"
test -f tutorial.py && test -d nanbeige_runtime && test -f source_manifest.json
shasum -a 256 -c package_files.sha256

The checksum list detects missing or changed package files; it is not an independent signature and does not authenticate the publisher. Stop on a missing file or checksum mismatch. Every new Mac terminal needs the WORKSPACE and NANBEIGE_ARTIFACT_ROOT exports again.

The package sits in WORKSPACE. Its top level contains tutorial.py, nanbeige_runtime/, source_manifest.json, dependency snapshots, calibration/ and evaluation/. The package provides importable functions rather than a new executable command. Existing Python runs those functions throughout the tutorial.

Create the Mac Python environment, then install the recorded packages. The installer is allowed to reach package publishers; subsequent model jobs will have no network access.

Mac terminal:

python3.13 -m venv .venv
.venv/bin/python -m pip install --only-binary=:all: \
  mlx==0.32.4 mlx-lm==0.32.2 transformers==5.20.2 \
  torch==2.11.0 numpy==2.5.4 safetensors==0.8.0 \
  sentencepiece==0.2.2 protobuf==7.36.2
.venv/bin/python -m pip freeze > mac_packages_actual.txt
.venv/bin/python -c 'import tutorial; tutorial.prepare()'

The package's complete dependency snapshots document the recorded environment. Save your resolved packages separately; a version pin is not a complete supply-chain audit. If a recorded release is unavailable, stop and review a replacement rather than silently switching versions.

The working tree has separate folders for source/, gguf/, mlx/, runtime_sources/, runtime_build/, calibration/, evaluation/, model_status/, status/ and scratch/. Original source files remain read-only. Conversion writes go to new output folders. A file ending in .partial is unfinished work, even if its size looks plausible.

Acquire the exact publisher revision

The model repository is Nanbeige/Nanbeige4.3-33B. The reference pins revision c1093c501cc6460427c0ac634db7223d0450c0ee. A branch name such as main can move; the exact revision binds the checkpoint, tokenizer and template to one source state.

The anonymous acquisition function requests that revision's file metadata, permits only the model's flat allowlist, downloads the 14 Safetensors shards and required small files, and verifies every completed file. Run this download with Internet access. Later conversion and inference commands run inside the network-blocking sandbox:

Mac terminal:

.venv/bin/python -c 'import tutorial; tutorial.acquire()'

It downloads 22 files totaling 66,256,520,718 bytes. Large files are checked against publisher LFS SHA-256 values. Small files are checked against Git blob identities and also receive local SHA-256 values. A Git blob hash includes a header before the file bytes; it is not the same calculation as a plain file hash.

If a download stops, keep its .partial file. Resume only when the server honors the requested byte range. The function rejects oversized partials, mismatched hashes and an unexpected source revision. It does not treat a partial file as a verified checkpoint. Completed files become mode 0444 before moving into place.

The download creates evidence/source_files.json and status/acquire_local.json under NANBEIGE_ARTIFACT_ROOT. Inspect those records and confirm that the local source_manifest.json revision matches the download. Use the sandboxed commands below for conversion and inference so model processes cannot access the network. Repeat the same pinned acquisition on the VM later; the VM source must match the Mac source file by file.

Inspect the checkpoint before loading it

Safetensors stores tensors without Python pickle execution. That removes one loading risk, but it does not establish that the publisher's code, dependencies or model behavior is safe. Keep trust_remote_code=False and do not download repository executables or adapters as a shortcut.

tutorial.acquire() runs source_validation.verify() automatically after downloading. The source check reads shard headers and the index without first loading all weights. It confirms that every indexed tensor appears once, that its byte range is valid and that the model's configuration agrees with the expected architecture. The reference contains 471 tensors and 33,118,977,024 BF16 parameters.

RoPE supplies position information to attention. Preserve its frequency base and the context metadata when converting. The tokenizer has 805 fewer real IDs than the model's padded output rows. Keep the padded weight rows. For the CUDA GGUF comparison, compare log probabilities over the real tokenizer IDs, renormalized to the same domain. The native MLX comparison keeps both outputs on their common full model-vocabulary width, as explained in Verify and test each MLX checkpoint.

The pinned card declares Apache-2.0; the inspected tree has no separate LICENSE file. The final source scan recheck showed three files marked safe and 19 queued, including all 14 weight shards. Queued scanning and correct hashes do not establish safe model behavior. Review publisher terms and notices before redistributing model artifacts. This tutorial's package does not redistribute the weights.

Apply the tokenizer patch before paid GPU work

Native architecture support is only half the compatibility check. Nanbeige's tokenizer uses Metaspace BPE: it represents spaces with a special marker and follows specific prefix, merge, fallback and added-token rules. A runtime can load a Llama tensor layout while tokenizing its text incorrectly.

The worked example added native support to a pinned llama.cpp checkout. It did not rename the tokenizer to a vaguely similar preset or decode and retokenize messages through another runtime. The patch modifies conversion/llama.py and src/llama-vocab.cpp and identifies the pre-tokenizer as nanbeige43.

Run these steps in the Mac terminal:

Mac terminal:

git clone https://github.com/ggml-org/llama.cpp \
  "$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout"
cd "$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout"
git checkout --detach 6050b467c07a95c7b87debd4ddac2f8ba8d9b807
git apply --check "$WORKSPACE/nanbeige_runtime/patches/nanbeige_tokenizer.patch"
git apply "$WORKSPACE/nanbeige_runtime/patches/nanbeige_tokenizer.patch"
git diff --stat
cd "$WORKSPACE"

A failed --check is a compatibility stop. Confirm that the expected commit is checked out and that no earlier patch is already present. Do not repeatedly apply the patch or force it onto a newer tree.

Build a CPU-only tokenizer library on the Mac. This check does not need a GPU VM or load the full model weights:

Mac terminal:

cmake -S "$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout" \
  -B "$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout/build_tokenizer" \
  -DGGML_METAL=OFF -DLLAMA_BUILD_TESTS=OFF \
  -DLLAMA_BUILD_SERVER=OFF -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build "$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout/build_tokenizer" \
  --target llama -j 8
.venv/bin/python -c 'from nanbeige_runtime.tokenizer_parity import create_probe, run; create_probe(); print(run())'

The vocabulary-only probe compares 330 cases and 1,980 comparisons, including leading spaces, Chinese, code, byte fallback, Unicode, special tokens and chat templates. The recorded run had zero mismatches. Require zero mismatches before converting the full checkpoint. Keep the upstream commit, patch hash and actual built-library hash in your evidence. A patched checkout is not an unmodified upstream release.

Verify the Mac sandbox

An offline flag asks a library not to download. A sandbox enforces a boundary even when code attempts a connection. The reference Mac profile blocks networking, keychain services, unrelated user-file reads, and execution of other programs while allowing the chosen Python process and Metal operations.

Run this boundary probe before any full model conversion or inference. The vocabulary-only setup check in Apply the tokenizer patch before paid GPU work uses public pinned tokenizer files; it is not a full model job:

Mac terminal:

.venv/bin/python -c 'from nanbeige_runtime.isolation import verify; print(verify())'

The probe creates a nonsensitive task canary outside the allowed model roots. It requires a denied read of that canary, denied Internet and instance-metadata connections, denied execution of another binary, absent credentials and a successful small Metal computation. A timeout alone is not evidence of a deny rule; the Mac test requires the operating system's permission-denied result.

The profile derives readable directories from the working directory and Python executable. If your interpreter or project is elsewhere, review the resolved paths. Do not fix a denied import by allowing the entire home directory or keychain. Save the probe results, then keep model conversion and inference inside the same boundary.

Create the Azure VM

Create a dedicated Azure VM using the following portal settings.

1. Open Virtual machines, then Create and Azure virtual machine. Select your own subscription and a resource group reserved for this tutorial.
2. On Basics, choose the region after checking your account's NC A100 vCPU quota and regional availability. One NC24 request needs 24 family vCPUs. Quota permission and physical capacity are separate.
3. Choose Ubuntu Server 24.04 LTS if available and supported for your selected GPU size. Its Python 3.12 matches the Linux version used here. Record the actual image/version rather than treating a moving latest image as a reproducible identifier.
4. Select Standard_NC24ads_A100_v4. Confirm its details show one A100, 24 virtual CPUs and 220 GiB host RAM. Use SSH public-key authentication with your existing key; retain your private key only on the management computer.
5. On Disks, select a persistent Standard SSD LRS OS disk and set its capacity to 1024 GiB. Recheck the actual price and throughput; a temporary data disk is not a substitute for persistent model output storage.
6. On Networking, permit SSH only from your observed management IP or approved private network. If a portal default permits SSH from Any, replace that rule before creating the VM. Leave model-service ports closed.
7. On Review + create, inspect the selected image, GPU, disk, network rules and estimated cost. Create the VM only after those choices are correct. Record the allocation start and resource identity privately.

If allocation fails, inspect Azure's error before retrying. Resolve a quota or subscription restriction first; for regional capacity errors, check availability before selecting another supported region or retrying. Once a VM is allocated, diagnose that VM's SSH readiness rather than creating another one.

For an existing VM, first confirm its current state and ownership. Resize only with the required authority. The example expanded an existing Standard SSD OS disk from 512 GiB to 1024 GiB and grew the guest filesystem. It preserved existing data. A reader using a fresh dedicated VM can provision adequate persistent storage at creation.

An OS-disk size change in the Azure portal does not by itself prove that Linux can use the added capacity. After any approved expansion, inspect lsblk, findmnt and df. If the partition or filesystem still has the old size, follow Azure's documented procedure for that disk layout. A growpart/resize2fs command for one ext4 layout is not a universal recipe for LVM or XFS. Do not format a disk that contains data.

Keep the final source and models off the VM's temporary local disk. Temporary storage can disappear after deallocation or relocation. The example's model root was verified to be on the persistent OS filesystem; the folder's name alone did not establish persistence.

Check the running machine and create the remote workspace

Connect through your own SSH configuration. Create a task-specific SSH alias in your existing SSH configuration with your VM's observed address, chosen user and key path. Use only those real values. Use an editor to append a separate Host block to your existing ~/.ssh/config. Keep all existing entries. Replace the three uppercase fields below with the observed VM address, the login user chosen at VM creation and the real existing private-key path. This is a configuration example, not values to guess:

Host nanbeige-vm
    HostName YOUR_OBSERVED_VM_ADDRESS
    User YOUR_VM_LOGIN_USER
    IdentityFile YOUR_EXISTING_PRIVATE_KEY_PATH
    ForwardAgent no

The VM address is available on its Azure Overview page. The key path points to your existing key on the Mac, not a copy stored on the VM. Test the alias before copying. The examples call your configured host nanbeige-vm. That is a local alias you create for your own address and SSH key; it is not the worked example's address.

Mac terminal:

ssh nanbeige-vm

After SSH connects, run these checks.

VM terminal, before starting tmux:

nvidia-smi
python3 --version
lsblk
findmnt /
df -h /
cat /proc/sys/kernel/random/boot_id

Confirm one A100, usable CUDA, sufficient GPU memory and no other GPU processes. nvidia-smi shows driver capability as well as device state; its CUDA version label is not proof of the toolkit installed in your Python environment. The recorded driver was 580.178.04 and PyTorch reported 85,093,777,408 bytes of total device memory. Host RAM, disk space and GPU memory are separate checks.

A GPU VM can have a preinstalled driver, or a stock Ubuntu image can need one. Follow Azure's N-series Linux driver guidance for the actual image. First try nvidia-smi. If it fails because no usable NVIDIA driver is installed, a supported Ubuntu-packaged route starts with:

VM terminal:

sudo apt-get update
sudo apt-get install -y ubuntu-drivers-common
ubuntu-drivers list

Inspect the available recommended NVIDIA driver and its CUDA compatibility before installing. CUDA 13 requires a compatible recent driver; do not assume an older driver is sufficient because the GPU is visible. If Azure’s guidance for your image specifies a driver package or server branch, follow it instead of a generic recommendation. When this Ubuntu driver route is supported, install its recommended driver and reboot:

VM terminal:

sudo ubuntu-drivers install
sudo reboot

Reboot ends SSH and tmux sessions. Reconnect after the VM is ready, run nvidia-smi again, and record the loaded driver and total device memory. With Secure Boot enabled, follow the documented signed-module and enrollment requirements; do not disable security controls to hide a driver installation failure.

On the VM, use a persistent path appropriate to its filesystem. These examples use /opt/nanbeige_work, with the model root below it. Install operating-system build tools with Internet access from the administrator's terminal:

VM terminal, before starting tmux:

sudo apt-get update
sudo apt-get install -y python3-venv python3-dev git cmake \
  build-essential bubblewrap util-linux tmux
sudo mkdir -p /opt/nanbeige_work

Use tmux, an existing terminal multiplexer, to keep the long job alive if SSH disconnects. First start its session on the VM:

VM terminal, inside tmux:

tmux new -s nanbeige

Start tmux after any driver reboot because reboot ends its sessions. The following VM commands run inside that session. After a dropped connection, reconnect and use tmux attach -t nanbeige. Do not start another copy of a job already running there. A VM reboot ends the session; it is not the same as an SSH disconnect.

Copy only the extracted reference package, without the Mac’s environment or downloaded weights. Include its integrity list so the VM can verify the transferred subset. From the Mac, a staged copy can use:

Mac terminal:

cd "$WORKSPACE"
ssh nanbeige-vm 'mkdir -p /tmp/nanbeige_reference'
scp -r tutorial.py nanbeige_runtime source_manifest.json fixtures_manifest.json package_files.sha256 \
  calibration evaluation nanbeige-vm:/tmp/nanbeige_reference/
ssh nanbeige-vm 'sudo cp -R /tmp/nanbeige_reference/. /opt/nanbeige_work/'

The temporary folder contains only the named public reference inputs. Move the staged package through your existing authenticated SSH channel. Do not copy the Mac virtual environment; create Linux environments on Linux. In the VM's management terminal, select the remote model root before importing the functions:

VM terminal, inside tmux:

cd /opt/nanbeige_work
export NANBEIGE_ARTIFACT_ROOT=/opt/nanbeige_work/nanbeige_4_3_33b
sha256sum -c --ignore-missing package_files.sha256
sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT python3 -c 'import tutorial; tutorial.prepare()'

Every newly attached VM shell needs the working directory and root below. The complete management-Python commands later select conversion_venv after it is installed:

VM terminal, inside tmux:

cd /opt/nanbeige_work
export NANBEIGE_ARTIFACT_ROOT=/opt/nanbeige_work/nanbeige_4_3_33b

Do not paste multi-line loops into an interactive Python prompt. Each subsequent block includes a complete heredoc invocation, which passes the whole block to Python at once. Run it from /opt/nanbeige_work inside tmux. The management interpreter calls a separate isolated CPU or CUDA child.

Namespace commands require conversion_venv from Prepare a separate Linux conversion environment to exist; the mount is not optional. The Linux controller needs administrator authority to create namespaces and bind approved paths. Its model child drops to UID and GID 65534, clears capabilities and receives no-new-privileges. Root authority belongs to the controller; it is not given to the model process.

/work paths exist inside the child namespace. A host shell uses the model-root path; it does not use /work.

Download and cross-check the remote source

Run the same acquisition on the VM with Internet access, using the remote root. Subsequent model jobs run in the network-blocking namespace. The download checks each publisher hash independently and creates evidence/source_files.json. Compare every filename, size and SHA-256 against the Mac record. Keep the small records locally. VM terminal, inside tmux:

cd /opt/nanbeige_work
sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT python3 -c 'import tutorial; tutorial.acquire()'

An independent file-by-file cross-check can run from the Mac:

Mac terminal:

ssh nanbeige-vm 'cd /opt/nanbeige_work/nanbeige_4_3_33b/source && LC_ALL=C sha256sum * | LC_ALL=C sort -k2' > vm_source.sha256
(cd "$NANBEIGE_ARTIFACT_ROOT/source" && LC_ALL=C shasum -a 256 * | LC_ALL=C sort -k2) > mac_source.sha256
diff mac_source.sha256 vm_source.sha256

No diff output and exit status zero means the complete hash lists match. This re-reads both 66-GB sources; disk throughput and hashing time belong in the operation budget.

The download function uses standard Python and does not need a model-serving API or a stored Hugging Face token for this public checkpoint. The 400-GB acquisition reserve in this function is conservative for both copies. Later remote stage checks retain a 300-GB reserve. If your free space fails either check, expand the authorized persistent storage or choose a different empty workspace; do not remove unrelated model directories.

Make the read-only source directory traversable, create narrowly writable model outputs and caches, and preserve the controller directory as administrator-controlled. The example namespace maps only the approved inputs and output directories. It does not mount host home directories, other models or cloud credential stores.

VM terminal, inside tmux:

sudo chmod 755 "$NANBEIGE_ARTIFACT_ROOT" "$NANBEIGE_ARTIFACT_ROOT/source"
sudo chmod 444 "$NANBEIGE_ARTIFACT_ROOT"/source/*
sudo chown -R 65534:65534 "$NANBEIGE_ARTIFACT_ROOT/gguf" \
  "$NANBEIGE_ARTIFACT_ROOT/scratch" "$NANBEIGE_ARTIFACT_ROOT/model_status" \
  "$NANBEIGE_ARTIFACT_ROOT/runtime_build" "$NANBEIGE_ARTIFACT_ROOT/model_cache"

These permission changes target only the new task directories. Do not apply them recursively to /opt, an existing shared model root or a cloud administrator's home directory.

Prepare a separate Linux conversion environment

Create one environment for CPU conversion and one for CUDA/vLLM. A CPU PyTorch install cannot serve as evidence that CUDA works; a CUDA environment need not replace another workload's environment.

On the VM:

VM terminal, inside tmux:

sudo python3 -m venv "$NANBEIGE_ARTIFACT_ROOT/conversion_venv"
sudo "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" -m pip install \
  --index-url https://download.pytorch.org/whl/cpu torch==2.11.0
sudo "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" -m pip install \
  --only-binary=:all: numpy==2.5.4 transformers==5.20.2 \
  safetensors==0.8.0 sentencepiece==0.2.2 protobuf==7.36.2
sudo "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" -m pip freeze

The installed CPU wheel is recorded as torch 2.11.0+cpu. Save the resolved dependency list and package-install reports. The reference's complete Linux conversion lock is in the package. Package installation is an administrator-controlled network operation, followed by inference with egress denied.

Download the same pinned runtime source on the VM with Internet access, then apply the supplied patch. The build commands below run inside the restricted namespace:

VM terminal, inside tmux:

cd /opt/nanbeige_work
export NANBEIGE_ARTIFACT_ROOT=/opt/nanbeige_work/nanbeige_4_3_33b
LLAMA_SOURCE="$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout"
sudo git clone https://github.com/ggml-org/llama.cpp "$LLAMA_SOURCE"
sudo git -C "$LLAMA_SOURCE" checkout --detach 6050b467c07a95c7b87debd4ddac2f8ba8d9b807
sudo git -C "$LLAMA_SOURCE" apply --check /opt/nanbeige_work/nanbeige_runtime/patches/nanbeige_tokenizer.patch
sudo git -C "$LLAMA_SOURCE" apply /opt/nanbeige_work/nanbeige_runtime/patches/nanbeige_tokenizer.patch
sudo git -C "$LLAMA_SOURCE" diff --stat
sha256sum /opt/nanbeige_work/nanbeige_runtime/patches/nanbeige_tokenizer.patch

Compare the patch fingerprint and changed-file list with the Mac. An existing checkout requires inspection before these commands; do not clone over it or apply the patch twice. Build a CPU configuration with GGML_CUDA=OFF, GGML_METAL=OFF, server and curl disabled, and Release optimization. Use at most 16 build threads on this 24-vCPU VM so management remains responsive.

Run the CPU configure/build with the same importable controller:

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.execute("configure_cpu", ["/usr/bin/cmake",
    "-S","/work/runtime_sources/llama_cpp_checkout",
    "-B","/work/runtime_build/cpu","-DGGML_CUDA=OFF","-DGGML_METAL=OFF",
    "-DLLAMA_BUILD_TESTS=OFF","-DLLAMA_BUILD_SERVER=OFF",
    "-DLLAMA_CURL=OFF","-DCMAKE_BUILD_TYPE=Release"],gpu=False)
tutorial.execute("build_cpu", ["/usr/bin/cmake","--build",
    "/work/runtime_build/cpu","--target","llama","llama-quantize",
    "-j","16"],gpu=False)
PYTHON

The source tree is read-only inside the namespace; build outputs are in runtime_build/. Do not try to write an in-source CMake build directory through a read-only mount.

Build the private CUDA runtime

Create a new CUDA Python environment. The recorded dependency resolution used vLLM 0.32.0, PyTorch 2.13.0 with CUDA 13 support, and a private CUDA compiler from the cuda-toolkit package.

VM terminal, inside tmux:

sudo python3 -m venv "$NANBEIGE_ARTIFACT_ROOT/cuda_venv"
sudo "$NANBEIGE_ARTIFACT_ROOT/cuda_venv/bin/python" -m pip install \
  --only-binary=:all: vllm==0.32.0 'cuda-toolkit[nvcc]==13.0.3'
sudo "$NANBEIGE_ARTIFACT_ROOT/cuda_venv/bin/python" -m pip freeze

Locate the single nvcc compiler under cuda_venv/lib/python*/site-packages/nvidia/cu13/bin/. Resolve its toolkit root rather than reusing a compiler from another model environment. The Linux sandbox exposes this compiler through CUDA_HOME, CUDA_PATH and CUDAToolkit_ROOT.

Configure llama.cpp’s CUDA build through the namespace. Its /work paths are not host-shell paths.

80-real targets actual A100 machine code. The Python-version directory must match your environment. Build the llama, llama-imatrix and llama-quantize targets with 16 threads. Hash the resulting binaries and keep the configure flags, compiler version, upstream commit, patch hash and dependency lock together.

Run the configure and build through the namespace from the VM's management Python. The following block resolves the installed compiler directory instead of assuming a Python-version suffix:

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
from pathlib import Path
from nanbeige_runtime.records import ROOT
import tutorial
compilers=list((ROOT/"cuda_venv").glob("lib/python*/site-packages/nvidia/cu13/bin/nvcc"))
assert len(compilers)==1
toolkit="/work/"+str(compilers[0].parents[1].relative_to(ROOT))
tutorial.execute("configure_cuda", ["/usr/bin/cmake",
    "-S","/work/runtime_sources/llama_cpp_checkout",
    "-B","/work/runtime_build/cuda","-DGGML_CUDA=ON",
    "-DLLAMA_BUILD_TESTS=OFF","-DLLAMA_BUILD_SERVER=OFF",
    "-DLLAMA_CURL=OFF","-DCMAKE_BUILD_TYPE=Release",
    "-DCMAKE_CUDA_ARCHITECTURES=80-real",
    "-DCUDAToolkit_ROOT="+toolkit,
    "-DCMAKE_CUDA_COMPILER="+toolkit+"/bin/nvcc"],gpu=False)
tutorial.execute("build_cuda", ["/usr/bin/cmake","--build",
    "/work/runtime_build/cuda","--target","llama","llama-imatrix",
    "llama-quantize","-j","16"],gpu=False)
tutorial.execute("build_native_bridge", ["/usr/bin/g++","-std=c++17",
    "-shared","-fPIC","-O2","/work/python/nanbeige_runtime/native_bridge.cpp",
    "-I/work/runtime_sources/llama_cpp_checkout/include",
    "-I/work/runtime_sources/llama_cpp_checkout/ggml/include",
    "-L/work/runtime_build/cuda/bin","-lllama",
    "-Wl,-rpath,/work/runtime_build/cuda/bin",
    "-o","/work/runtime_build/native_bridge.so"],gpu=False)
PYTHON

Build native_bridge.cpp as a shared library linking the built llama library. On Linux, the bridge's runtime search path must point to /work/runtime_build/cuda/bin inside the namespace. The bridge uses the pinned runtime's native model APIs so Python can read logits, tokenize and generate without starting an API server.

Prove that the private GPU boundary works

The Linux namespace hides host home directories, unrelated models and administrator credentials, provides a private /proc and /tmp, mounts source read-only, and drops the model child to UID 65534. Its GPU version deliberately exposes the NVIDIA device nodes, driver metadata, compiler and a private /dev/shm needed by the GPU stack.

After any driver reboot, confirm that the GPU devices exist before building the namespace command:

VM terminal, inside tmux:

ls -l /dev/nvidia0 /dev/nvidiactl /dev/nvidia-uvm /dev/nvidia-uvm-tools

If UVM nodes are absent, the namespace cannot bind them. Use the installed driver's supported device-initialization procedure, then list the nodes again. Where nvidia-modprobe is installed and supported, its existing helper can initialize UVM:

VM terminal, inside tmux:

sudo nvidia-modprobe -u -c=0

Do not remove the namespace to hide a missing-device problem. Run the CPU namespace probe after creating both environments and the required directories. Then run the GPU kernel probe through tutorial.gpu_python. The GPU probe tests CUDA initialization, a small compiled kernel, vLLM normalization, BF16 grouped attention with 56 query heads and 8 KV heads, and a one-rank NCCL collective over private loopback.

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
from nanbeige_runtime.linux_isolation import probe
print(probe())
tutorial.gpu_python(
    "cuda_kernel_probe",
    "from nanbeige_runtime.cuda_validation import kernel_probe; kernel_probe()"
)
PYTHON

Use the VM's administrator-controlled Python interpreter to run this block from the workspace with NANBEIGE_ARTIFACT_ROOT selected. The child executes inside the namespace's CUDA environment.

Require denied Internet and metadata access, absent host homes and other models, absent credentials, zero effective capabilities, read-only source, working private shared-memory locks and successful GPU calculations. A GPU check that succeeds only after removing the network namespace is not a passed private-runtime check.

If the driver and user-space libraries disagree, record both versions and diagnose the installed driver through the supported Azure procedure. If MIG partitioning exposes no usable CUDA device, stop before loading the model. MIG divides a GPU into smaller instances. Changing it can affect other workloads. The successful recorded allocation exposed the full A100 with MIG disabled; no MIG configuration mutation was needed.

Run one heavy job at a time

The reference's tutorial.execute function holds a task-local heavy-job lock, runs a child in the restricted namespace, checks the boot identifier and records progress and exit status. tutorial.gpu_python and tutorial.cpu_python use that function for the two environments. They do not allocate or shut down Azure resources.

Keep the VM management channel separate from the model's network-denied process. Review stage metadata every few minutes and track allocated cost while waiting. Record the start request and verified deallocation times, not only the time spent generating tokens. A second management terminal can read the stage records and cost worksheet while the long job runs. It must not give model code access to Azure credentials.

Use five-second memory samples for long GPU jobs, but label them as samples. A maximum sampled nvidia-smi value can miss a brief peak. Keep operating-system RSS high-watermarks and context memory snapshots separate. If a progress file goes stale, check the recorded process and boot identity before treating a PID as the same job.

A failed stage keeps its output as a candidate. Resume only when input hashes, output hashes and evidence establish what already completed. Do not delete verified work or repeat a full test merely because a wrapper returned an error after the model finished.

Convert the BF16 GGUF and Q8_0

Run the pinned converter through the CPU namespace. Preserve original BF16 weights rather than forcing a lower-precision source. Use a new temporary output file, inspect it, calculate its complete hash, then publish it inside the task directory.

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.execute("convert_bf16_gguf", [
    "/work/conversion_venv/bin/python",
    "/work/runtime_sources/llama_cpp_checkout/convert_hf_to_gguf.py",
    "/work/source", "--outfile", "/work/gguf/nanbeige_4_3_33b_bf16.gguf.partial",
    "--outtype", "bf16"
], gpu=False)
PYTHON

After the converter exits successfully, inspect and promote its temporary file with the host-side conversion environment:

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.finalize_output("convert_bf16_gguf",
    "nanbeige_4_3_33b_bf16.gguf.partial","nanbeige_4_3_33b_bf16.gguf")
tutorial.execute("quantize_q8_0", [
    "/work/runtime_build/cpu/bin/llama-quantize",
    "/work/gguf/nanbeige_4_3_33b_bf16.gguf",
    "/work/gguf/nanbeige_4_3_33b_q8_0.gguf.partial","Q8_0","16"],gpu=False)
tutorial.finalize_output("quantize_q8_0",
    "nanbeige_4_3_33b_q8_0.gguf.partial","nanbeige_4_3_33b_q8_0.gguf")
PYTHON

finalize_output requires the stage's successful exit record, checks key GGUF metadata and 471 tensors, computes the whole-file hash, refuses to overwrite a final file and promotes the candidate. The full value comparison follows in Verify all tensors and metadata.

The converter performs the architecture-required query/key layout transforms. It preserves BF16 matrices and stores normalization vectors losslessly as F32. That is a representation change for the vectors, not a claim that all GGUF tensor storage has one dtype.

Create Q8_0 directly from that BF16 GGUF with the pinned llama-quantize binary and 16 CPU threads. Q8_0 in this recipe uses no importance matrix. Standard llama.cpp quantization runs on CPU threads, even when the computer also has an A100.

Verify all tensors and metadata

A successful converter exit is not a completeness check. Use the package's GGUF verification function after the BF16 and Q8_0 outputs have their final filenames. It maps all 471 source tensors to GGUF names, verifies shapes and compares every original BF16 matrix against its required transformed representation.

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.cpu_python(
    "gguf_structure",
    "from nanbeige_runtime.gguf_artifacts import verify; verify()"
)
PYTHON

The recorded BF16 checks covered 366 BF16 matrices, including 104 query/key matrices with the required permutation, plus 105 normalization vectors stored unchanged as F32 (471 total). Q8_0 had 366 Q8_0 matrices and the same 105 F32 vectors. Verify the architecture, layer count, dimensions, attention heads, tokenizer vocabulary, special-token IDs, chat template, RoPE value and context metadata.

Check the lower-bit artifacts against the same BF16 GGUF later. Q6_K contains 366 Q6_K matrices. Q4_K_M is mixed precision: the recorded file has 313 Q4_K matrices, 53 Q6_K matrices and 105 F32 vectors. Its M suffix does not mean every stored number has four bits.

Establish the original CUDA reference

Load the original Safetensors model in vLLM at 16,384 total tokens, one request and BF16. Keep trust_remote_code=False, CPU offload at zero, generation deterministic, output capped, request statistics disabled and the model process offline. The reference sets chunked prefill to 256-token batches and uses eager execution and the verified Flash Attention 2 backend.

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.gpu_python(
    "vllm_bf16_16384",
    "from nanbeige_runtime.cuda_validation import vllm_baseline; vllm_baseline(16384)"
)
PYTHON

Run the eight fixed correctness fixtures, parsing checks, reasoning termination tests, numerical sample and 16K retrieval check. Keep the numerical reference arrays in the task's scratch area; export their hashes and copy them privately with the evidence before shutdown. Generated responses remain in memory. No executable tool is attached.

Inspect actual free GPU memory before increasing capacity. The BF16 weights require about 61.7 GiB. Conventional BF16 KV memory adds about 3.25 GiB at 16K and 6.5 GiB at 32K, before other allocations. The first child exits and releases its model memory. If measured headroom permits, run the 32,768-token reference:

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.gpu_python(
    "vllm_bf16_32768",
    "from nanbeige_runtime.cuda_validation import vllm_baseline; vllm_baseline(32768)"
)
PYTHON

The recorded original BF16 passed both capacities. A preload estimate is a planning check, not an allocator guarantee. If 32K is blocked on another system, record the missing coverage rather than claiming it passed.

Validate BF16 GGUF with full GPU weight placement

Run the native BF16 GGUF through the same fixed tests and compare its numerical sample against the original vLLM BF16 sample.

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.gpu_python(
    "llama_bf16_gguf",
    "from nanbeige_runtime.cuda_validation import llama_validate; "
    "llama_validate('bf16_gguf','nanbeige_4_3_33b_bf16.gguf')"
)
PYTHON

Require actual CUDA execution and full model-weight placement, including the token embeddings. The bridge sets the embedding tensor to CUDA0. Inspect both the offload summary and every model-weight buffer. The recorded accepted run reported 53/53 layers offloaded and no CPU model-weight buffer.

The first candidate had an apparently good layer-offload count while retaining the BF16 embedding weights on CPU, about 2.23 GiB of tensor data (166,912 × 7,168 × 2 bytes). It did not meet the full-GPU criterion. Correcting the embedding placement and running a new candidate supplied the accepted result. CPU output and compute staging buffers can still exist; a CPU staging buffer is different from a CPU model-weight buffer.

Read the complete native result in the VM’s host-side management interpreter and apply its acceptance gate:

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
import json
from nanbeige_runtime.records import ROOT, atomic_json
result = json.loads((ROOT / "model_status/llama_bf16_gguf.json").read_text())
verdict = tutorial.accepted_result(result)
atomic_json(ROOT / "evidence/accepted_bf16_gguf.json", verdict)
PYTHON

Both the verdict and every target context must pass before using this BF16 file as an accepted calibration reference. Cross-runtime floating-point results need not be bit-identical. The recorded BF16 comparison had finite values, top-token agreement of 1.0 and mean KL of about 0.000449 nats on the three-text sample. Retain maximum and RMS log-probability differences, the fixture results and the exact evaluated token domain.

Calibrate an importance matrix on CUDA

Calibration runs sample text through the full-precision model and gathers statistics for quantization. Keep calibration separate from evaluation so your correctness tests do not become the quantizer's training examples.

The calibration tool writes its final filename directly. Treat that file as a candidate until finalize_importance_matrix succeeds and makes it read-only. The package supplies original nonsensitive English, Chinese, code and inert-tool text under CC0. Its selected text contains 79,995 tokens under the pinned tokenizer. The accepted imatrix run processed 156 chunks of 512 tokens: 79,872 tokens. Selection count and processed count are separate quantities.

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.execute("importance_matrix", [
    "/work/runtime_build/cuda/bin/llama-imatrix",
    "-m", "/work/gguf/nanbeige_4_3_33b_bf16.gguf",
    "-f", "/work/calibration/selected_79995_tokens.txt",
    "-o", "/work/gguf/nanbeige_4_3_33b_imatrix.gguf",
    "-ngl", "99", "-ot", "token_embd.weight=CUDA0",
    "-c", "512", "-b", "256", "-ub", "256", "-t", "16",
    "-fa", "on", "-ctk", "bf16", "-ctv", "bf16",
    "--no-ppl", "--chunks", "156", "--output-frequency", "10",
    "--verbosity", "4"
], gpu=True)
PYTHON

Run the host-side verifier before either lower-bit quantization:

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
print(tutorial.finalize_importance_matrix())
PYTHON

It checks the successful launch record, actual offload log, processed-chunk metadata and statistics count, then binds the complete matrix hash to its source, calibration and binary hashes. It makes the accepted matrix read-only. Confirm actual full GPU weight placement in the log, not only the requested -ngl value. The accepted matrix had 1,456 statistics tensors and a complete SHA-256 of ea512ef3fe60d445134e09ef3510e2aa3ed75d7a681ea7e1c8eb2084b8ebea1a. Record the BF16 input hash, selected-corpus hash, binary hash, chunk count, chunk size and matrix hash together.

Set verbosity to 4 before calibration so the runtime records the actual GPU offload and complete statistics. Keep the matrix as a candidate until finalize_importance_matrix verifies the required offload log and statistics. A completed calibration without that log is not sufficient for lower-bit quantization.

Quantize Q6_K and Q4_K_M

Both recipes start from the BF16 GGUF and the accepted importance matrix. Call tutorial.require_importance_matrix() immediately before the loop to recheck the matrix, BF16 and calibration fingerprints. Use CPU threads and write to a new temporary file for each output.

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
tutorial.require_importance_matrix()
for recipe, name in [
    ("Q6_K", "nanbeige_4_3_33b_q6_k.gguf.partial"),
    ("Q4_K_M", "nanbeige_4_3_33b_q4_k_m.gguf.partial")
]:
    tutorial.execute("quantize_"+recipe.lower(), [
        "/work/runtime_build/cuda/bin/llama-quantize",
        "--imatrix", "/work/gguf/nanbeige_4_3_33b_imatrix.gguf",
        "/work/gguf/nanbeige_4_3_33b_bf16.gguf",
        "/work/gguf/"+name, recipe, "16"
    ], gpu=False)
PYTHON

The binary was compiled in the CUDA build, but this quantization operation uses CPU execution. It does not generate a new matrix and does not read a Q8_0 artifact as its weight source.

After each successful quantization, promote and verify that artifact with the host-side conversion environment:

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
for recipe, filename in [("Q6_K","nanbeige_4_3_33b_q6_k.gguf"),
                         ("Q4_K_M","nanbeige_4_3_33b_q4_k_m.gguf")]:
    tutorial.finalize_output("quantize_"+recipe.lower(),filename+".partial",filename)
    print(tutorial.verify_quantized(filename,recipe))
PYTHON

If the CUDA-built quantizer fails to start in the CPU namespace with a missing CUDA driver library, use /work/runtime_build/cpu/bin/llama-quantize, built from the same patched commit in Prepare a separate Linux conversion environment. Record which binary hash produced each file.

After the quantizer exits successfully, finalize_output checks preliminary metadata and tensor count, hashes the candidate, makes it read-only and renames it. verify_quantized then checks the final file’s metadata, tensor shapes and types, and normalization values. If verification fails, retain the final-named file as an unaccepted candidate and block acceptance and delivery. Keep the accepted matrix read-only too.

Apply the same CUDA acceptance to every quantized GGUF

Run Q8_0, Q6_K and Q4_K_M separately. One model is loaded at a time. Each comparison uses the native BF16 GGUF numerical reference, not the MLX BF16 reference.

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
for key, filename, bits in [
    ("gguf_q8_0", "nanbeige_4_3_33b_q8_0.gguf", 8),
    ("gguf_q6_k", "nanbeige_4_3_33b_q6_k.gguf", 6),
    ("gguf_q4_k_m", "nanbeige_4_3_33b_q4_k_m.gguf", 4)
]:
    code=("from nanbeige_runtime.cuda_validation import llama_validate; "
          f"llama_validate({key!r},{filename!r},{bits})")
    tutorial.gpu_python("llama_"+key, code)
PYTHON

The tests cover eight correctness fixtures: English arithmetic and reading, Chinese arithmetic and reading, code reasoning, Python generation, inert tool JSON and reasoning termination. Two additional adversarial prompts record behavior; they have no scored expected answer and do not certify refusal or safety.

Parse generated Python with ast.parse and inspect the expected function shape; do not execute it. Parse the tool output with a JSON decoder and compare the declared name and arguments; do not attach a real tool. Require reasoning closure and an end-of-generation stop for the reasoning fixture. A response hitting its maximum output length does not establish correct termination.

These coarse engineering thresholds were declared for this experiment. The numeric relative-change values in JSON are fractions: 0.10 means 10%, not 0.10%. Measured changes were well inside these limits. They are not publisher benchmarks or universal quality guarantees. Apply tutorial.accepted_result to the complete native GGUF JSON result. It requires the expected fixtures, parse results, finite numerical values, correct reasoning stop, privacy flags, full CUDA weight placement and all requested contexts.

After each quantized test, read its full result and save a separate verdict. Run this block in the VM’s host-side management interpreter:

VM management Python, inside tmux:

sudo --preserve-env=NANBEIGE_ARTIFACT_ROOT \
  "$NANBEIGE_ARTIFACT_ROOT/conversion_venv/bin/python" - <<'PYTHON'
import tutorial
import json
from nanbeige_runtime.records import ROOT, atomic_json
for key, bits in [("gguf_q8_0", 8), ("gguf_q6_k", 6), ("gguf_q4_k_m", 4)]:
    result = json.loads((ROOT / "model_status" / ("llama_" + key + ".json")).read_text())
    verdict = tutorial.accepted_result(result, bits)
    atomic_json(ROOT / "evidence" / ("accepted_" + key + ".json"), verdict)
PYTHON

An assertion failure blocks delivery of that file. Keep its result and failure reason, fix the affected condition and test a new candidate.

Test real 16K, 32K and 64K retrieval

The retrieval cases run inside the validation calls in CUDA Reference, BF16 Validation, Quantized Acceptance, MLX Conversion and MLX Validation; no separate model job is required here. CUDA context records are under model_status, while Metal context records are under status. A configured maximum is not evidence that a model processed a prompt that long. The reference builds a canonical chat-template prompt containing a unique marker near the beginning, ordinary filler in the middle and a final retrieval request. It trims only middle filler token IDs, preserving the marker, query and template boundaries.

Keep the full BF16 KV cache, disable sliding or rotating caches and record actual prompt and generated-token counts. Retrieve the same marker after each complete prefill. Raising a server's max-model-len or using a short prompt with a 64K setting does not pass this test.

The three quantized GGUFs and all three MLX versions passed the three capacities. BF16 coverage ends at 16K and 32K; no 64K BF16 result is claimed. A 64K BF16 load is not ruled out by weight-plus-cache arithmetic alone: about 61.7 GiB of weights plus 13 GiB of KV cache is below the A100’s measured 79.25 GiB total. Workspace, allocator behavior and runtime limits still matter, so that capacity remains untested. The 64K retrieval check does not compare against a 64K BF16 reference. At 524,288 tokens the ordinary BF16 KV cache alone needs about 104 GiB. Combined with BF16 weights, that exceeds the combined memory of two A100 80 GB devices (about 158.5 GiB, twice the single-device total measured here) before workspace overhead. Testing that context needs a separately designed memory strategy. No 512K result is claimed here.

Read the quality and memory results

Mean negative log likelihood, or NLL, measures how costly the correct next tokens are under the model's predicted probabilities. KL measures a difference between probability distributions. Keep each model family paired with its own full-precision baseline.

The numerical sample comprises three short English, Chinese and code texts. NLL is averaged text by text; it is not a broad corpus benchmark. The distribution comparison uses the final evaluated position of each text over the same real token IDs. A negative NLL change on this sample is not evidence of a general quality improvement.

The memory column contains context snapshots. The quantized jobs also recorded five-second sampled device peaks and observed process RSS high-watermarks. Do not label a context snapshot as a full-job allocator peak or compare it directly with a differently sampled Mac measurement.

Run the independent MLX track on the Mac

The Mac work can overlap Azure work when the Mac is not also doing a memory-intensive task. It never consumes the VM's A100. Use the same read-only official BF16 source and the already verified Mac sandbox.

Run native BF16 Metal validation first so it supplies the MLX family's numerical reference. Use the sandboxed function rather than exposing private files to an unrestricted model loader:

Mac terminal:

cd "$WORKSPACE"
.venv/bin/python -c 'import tutorial; tutorial.metal_python("from nanbeige_runtime.metal import validate; validate(None)")'

Run the separate syntax and inert-argument parser against the same deterministic BF16 fixtures. It must both parse successfully and reproduce the prior fixture hashes:

Mac terminal:

.venv/bin/python - <<'PYTHON'
import tutorial
tutorial.metal_python(
    "from nanbeige_runtime.format_validation import validate; "
    "r=validate(None); assert r['all_parsed'] and "
    "all(x['same_output_as_original_fixture'] for x in r['fixtures'])"
)
PYTHON

After BF16 correctness, numerical, context and parsing checks, record the actual verdict:

Mac terminal:

.venv/bin/python -c 'import tutorial; print(tutorial.record_metal_acceptance())'

The BF16 Metal context test passed 16K and 32K using the canonical chat-template fixture. Its result is supplied as reference_results/validate_bf16_context_v2_result.json. A completed result with a failed retrieval is still a failed candidate.

Then convert one independent MLX version at a time:

Mac terminal:

.venv/bin/python -c 'import tutorial; tutorial.metal_python("from nanbeige_runtime.metal import convert; convert(8)")'
.venv/bin/python -c 'import tutorial; print(tutorial.verify_mlx(8))'
.venv/bin/python -c 'import tutorial; tutorial.metal_python("from nanbeige_runtime.metal import convert; convert(6)")'
.venv/bin/python -c 'import tutorial; print(tutorial.verify_mlx(6))'
.venv/bin/python -c 'import tutorial; tutorial.metal_python("from nanbeige_runtime.metal import convert; convert(4)")'
.venv/bin/python -c 'import tutorial; print(tutorial.verify_mlx(4))'

The converter uses quantize=True, affine mode, group size 64 and the requested bit count, with dtype=bfloat16 for the source. Group size 64 means one set of quantization parameters represents a group of 64 weight values. Affine quantization stores packed numbers plus scales and biases. The recipe differs from GGUF's Q6_K and Q4_K_M.

The recorded converter phases took about 17.0, 14.5 and 12.8 seconds for 8-, 6- and 4-bit respectively. Those phase timings exclude source acquisition and the much longer model validation. They are measurements on the stated Mac and software stack, not complete-operation estimates.

Each Metal call overwrites scratch/metal_stderr.txt. After a failure, copy that file aside before running the next command.

Verify and test each MLX checkpoint

The structure check verifies every logical source tensor, packed matrix shapes, BF16 scales/biases, unchanged one-dimensional tensors, architecture metadata and the exact chat template. Check tokenizer parity with the source too. A serialized tokenizer can change a field while still behaving correctly; establish behavior through token IDs and decoding, not only file text.

Run the same correctness, parsing, numerical and retrieval coverage through native Metal:

Mac terminal:

.venv/bin/python -c 'import tutorial; tutorial.metal_python("from nanbeige_runtime.metal import validate; validate(8)")'
.venv/bin/python -c 'import tutorial; tutorial.metal_python("from nanbeige_runtime.metal import validate; validate(6)")'
.venv/bin/python -c 'import tutorial; tutorial.metal_python("from nanbeige_runtime.metal import validate; validate(4)")'

The separate parser reuses the deterministic code and inert-tool cases, validates their structure without execution, and checks that their output hashes match the preceding correctness run:

Mac terminal:

.venv/bin/python - <<'PYTHON'
import tutorial
for bits in (8, 6, 4):
    tutorial.metal_python(
        "from nanbeige_runtime.format_validation import validate; "
        f"r=validate({bits}); "
        "assert r['all_parsed'] and "
        "all(x['same_output_as_original_fixture'] for x in r['fixtures'])"
    )
PYTHON

Read each complete result and apply an explicit acceptance decision after parsing:

Mac terminal:

.venv/bin/python - <<'PYTHON'
import tutorial
for bits in (8, 6, 4):
    print(tutorial.record_metal_acceptance(bits))
PYTHON

This gate requires the expected correctness fixtures, reasoning stop, successful syntax and inert-argument parsing, matching parser output hashes, finite numerical values inside the stated engineering thresholds, actual retrieval at every target length and ordinary BF16 cache records. Writing status=complete is not sufficient.

The MLX numerical comparison uses the full 166,912-row model output vocabulary in both native BF16 and its quantized derivative. CUDA GGUF comparisons use the 166,107 real tokenizer IDs renormalized to that common domain. Each quantizer is compared with its own runtime family’s BF16 baseline; the KL values are not a cross-family ranking.

Keep generation deterministic, bounded outputs, 256-token prefill steps, full ordinary KV caches and no quantized or rotating cache. The reference limits MLX's memory use to 104 GiB with a 2-GiB cache allowance on the 128-GiB Mac, leaving system headroom. These settings are not suitable assumptions for a smaller Mac.

The recorded MLX mean KL values were 0.001971 for 8-bit, 0.008405 for 6-bit and 0.029917 for 4-bit against native MLX BF16. All fixed correctness and parsing tests passed. All three retrieved the marker at 16K, 32K and 64K. The 64K retrieval tests took about 12.8–13.7 minutes per quantized model on this Mac; the conversion's few seconds did not remove that validation time.

Deliver only accepted GGUFs

Make each full remote download conditional on that artifact's CUDA acceptance. A structurally correct file is still a candidate until its required runtime tests pass. Copy the actual tested file; do not rebuild it on the Mac and call it the delivered CUDA artifact.

Use the existing authenticated private SSH channel and the Homebrew rsync installed in Prepare accounts and existing tools. Confirm /opt/homebrew/bin/rsync --version supports --append-verify. On the Mac, copy every accepted GGUF and the accepted matrix:

Mac terminal:

cd "$WORKSPACE"
for file in nanbeige_4_3_33b_bf16.gguf nanbeige_4_3_33b_q8_0.gguf \
            nanbeige_4_3_33b_q6_k.gguf nanbeige_4_3_33b_q4_k_m.gguf \
            nanbeige_4_3_33b_imatrix.gguf; do
  /opt/homebrew/bin/rsync --partial --append-verify -e ssh \
    "nanbeige-vm:/opt/nanbeige_work/nanbeige_4_3_33b/gguf/$file" \
    "$NANBEIGE_ARTIFACT_ROOT/gguf/"
done

rsync may retain an interrupted copy under the final-looking filename. Its name is not acceptance: the complete size/hash comparison and delivery record below remain mandatory. Resume only when its prefix belongs to the same accepted remote file. Parallelize transfer only after measuring network, CPU and disk headroom; keep one heavy memory job per machine.

The worked run used measured parallel raw SSH ranges for delivery after comparing one-stream, multiple-stream and gzip probes. Compression and additional streams can waste time when network or disk is already the bottleneck. The tutorial's rsync path is a portable delivery alternative; its transfer speed was not the measured raw-range speed and is not included as a claimed benchmark.

The SSH export commands below assume the VM login can run the listed administrative operations with non-interactive sudo. Check that access in the management session; do not put a sudo password in the commands or copied records. A hardened image needs its approved administrative export process. Export task-owned metadata and both numerical references from the Mac before shutdown. The namespace-generated records belong under model_status; controller stage records and decisions belong under status and evidence. Keep this archive private because your own operational paths and machine metadata can appear in it.

Mac terminal:

cd "$WORKSPACE"
umask 077
mkdir -p vm_evidence
ssh nanbeige-vm 'sudo tar -C /opt/nanbeige_work/nanbeige_4_3_33b -czf - evidence status model_status manifest.json scratch/vllm_bf16_numerical.npz scratch/llama_bf16_numerical.npz' > vm_evidence/nanbeige_evidence.tgz
tar -tzf vm_evidence/nanbeige_evidence.tgz
ssh nanbeige-vm 'sudo /opt/nanbeige_work/nanbeige_4_3_33b/conversion_venv/bin/python -m pip freeze' > vm_evidence/linux_converter_actual.txt
ssh nanbeige-vm 'sudo /opt/nanbeige_work/nanbeige_4_3_33b/cuda_venv/bin/python -m pip freeze' > vm_evidence/linux_cuda_actual.txt
ssh nanbeige-vm 'cd /opt/nanbeige_work/nanbeige_4_3_33b && sudo sha256sum scratch/vllm_bf16_numerical.npz scratch/llama_bf16_numerical.npz' > vm_evidence/numerical_references.sha256
tar -xzf vm_evidence/nanbeige_evidence.tgz -C vm_evidence
(cd vm_evidence && shasum -a 256 -c numerical_references.sha256)

The tar command lists only the task roots and the two exact NPZ files. A nonzero SSH/tar/hash result blocks shutdown: preserve the partial archive, fix the export and check it again. Export metadata and locks before releasing the VM; they explain what produced the accepted models. Do not archive unrelated home directories, credentials or all scratch files. Keep raw prompts, responses, credentials and unrelated files out of that export. A file-copy command exiting successfully does not establish that the full file matches the accepted artifact.

Verify the complete local bytes and reload offline

Compute SHA-256 on the complete remote file and the complete Mac copy. Match sizes and fingerprints. Reading only a prefix or checking the expected filename does not verify 66 GB.

In a new Mac terminal, repeat the two workspace/root exports from Create an isolated working directory on the Mac before continuing. Fetch complete remote fingerprints using the absolute task path, so a new SSH session does not depend on a previously exported variable. Then hash and record the Mac copies:

Mac terminal:

cd "$WORKSPACE"
ssh nanbeige-vm 'cd /opt/nanbeige_work/nanbeige_4_3_33b/gguf && sha256sum nanbeige_4_3_33b_bf16.gguf nanbeige_4_3_33b_q8_0.gguf nanbeige_4_3_33b_q6_k.gguf nanbeige_4_3_33b_q4_k_m.gguf nanbeige_4_3_33b_imatrix.gguf' > remote_hashes.txt
.venv/bin/python - <<'PYTHON'
import tutorial
keys = {
    "nanbeige_4_3_33b_bf16.gguf": "bf16_gguf",
    "nanbeige_4_3_33b_q8_0.gguf": "gguf_q8_0",
    "nanbeige_4_3_33b_q6_k.gguf": "gguf_q6_k",
    "nanbeige_4_3_33b_q4_k_m.gguf": "gguf_q4_k_m",
    "nanbeige_4_3_33b_imatrix.gguf": "importance_matrix",
}
seen = set()
for line in open("remote_hashes.txt"):
    digest, filename = line.split()
    if filename not in keys or filename in seen:
        raise RuntimeError("Unexpected or duplicate remote hash entry")
    seen.add(filename)
    print(tutorial.record_delivery(keys[filename], filename, digest))
if seen != set(keys):
    raise RuntimeError("A required complete remote fingerprint is missing")
PYTHON

record_delivery hashes the entire local file, checks the observed remote fingerprint, makes it read-only and saves the delivery record. The accepted matrix is transferred and fingerprinted too; it is not a model for offline inference. A mismatch blocks acceptance of the local copy. The package’s reference_artifact_hashes.json identifies the worked example; a reader must compare their own local file with their own accepted remote bytes. Rebuilding on another platform is not a substitute for delivery.

Compile the native bridge on the Mac against the CPU library built in Apply the tokenizer patch before paid GPU work:

Mac terminal:

c++ -std=c++17 -dynamiclib -fPIC -O2 \
  "$WORKSPACE/nanbeige_runtime/native_bridge.cpp" \
  -I"$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout/include" \
  -I"$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout/ggml/include" \
  -L"$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout/build_tokenizer/bin" \
  -lllama \
  -Wl,-rpath,"$NANBEIGE_ARTIFACT_ROOT/runtime_sources/llama_cpp_checkout/build_tokenizer/bin" \
  -o "$NANBEIGE_ARTIFACT_ROOT/runtime_build/native_bridge.so"

Hash the actual Mac library and bridge. Their hashes identify the local build; Linux and Mac binaries are different files. Keep the shared library in runtime_build/native_bridge.so; its runtime library search path points to the Mac library's bin directory. This is a Mac build, not the Linux CUDA bridge copied across operating systems.

Run the offline reloads inside the Mac sandbox:

Mac terminal:

.venv/bin/python -c 'import tutorial; print(tutorial.offline_reload("bf16_gguf"))'
.venv/bin/python -c 'import tutorial; print(tutorial.offline_reload("gguf_q8_0"))'
.venv/bin/python -c 'import tutorial; print(tutorial.offline_reload("gguf_q6_k"))'
.venv/bin/python -c 'import tutorial; print(tutorial.offline_reload("gguf_q4_k_m"))'

Each reload uses CPU, a 4,096-token capacity, zero GPU layers, denied networking and credential access, tokenizer parity and one arithmetic smoke fixture. The recorded four reloads passed. They demonstrate local reloadability of the exact accepted bytes; they do not repeat the full CUDA or Metal context suite.

A launcher failure before model loading does not invalidate an unchanged, verified artifact. Keep launcher errors separate from model-test results; check generated Python syntax and environment key names before retrying a corrected launcher.

Deallocate without losing the outputs

Finish Azure-dependent model work, complete the local artifact and evidence copies, match all hashes and pass the offline reloads before releasing compute. Mac-only MLX work and local review can continue afterward without renting the GPU again.

For a dedicated VM, verify its subscription, resource group, name and current state in the same management session used for the operation. Check that no model job or transfer is still active. For a shared VM, repeat the organization’s checks for the current boot, task ownership, other users and workloads that must remain running. Use the established shutdown process. A check performed hours earlier is insufficient if the workload or responsibility changed.

Use your own VM identifiers with Azure's existing CLI:

Mac terminal:

az vm deallocate --resource-group YOUR_RESOURCE_GROUP --name YOUR_VM_NAME
az vm get-instance-view --resource-group YOUR_RESOURCE_GROUP --name YOUR_VM_NAME \
  --query "instanceView.statuses[?starts_with(code, 'PowerState/')].code" --output tsv

Require PowerState/deallocated. A stopped guest, a disconnected SSH session or an accepted deallocation request alone is insufficient. If Azure reports deallocating, continue bounded readiness polling. Record failures and the actual verified end of allocated time rather than assuming billing stopped.

Retained persistent disks keep billing. Delete a temporary tutorial disk only after you have separately confirmed its exact identity, complete local preservation, that no other workload depends on it and the deletion decision. Deallocation and disk deletion are different actions. This example retained the disk and did not delete models or resources.

Keep a reproducible completion record

Save a manifest with the publisher revision, original file hashes, runtime commit, patch and binary hashes, resolved package locks, calibration hash/license, quantization recipe, output file sizes/hashes and token/template metadata. Save validation with backend, hardware, settings, each fixture result, actual context counts, numerical metrics, memory measurement scope and failures.

Keep allocated intervals, measured rates, transfer quantities and cost assumptions separate from acceptance. A cheaper run does not establish a better model, and a small fixture suite does not establish production suitability. Mark your completion record finished only after model work, required delivery and actual Azure deallocation are verified.

Nanbeige on Apple M1 Max and Apple M4 Max

Compare Nanbeige's 8-bit, 6-bit and 4-bit MLX models on an Apple M1 Max with 64 GB unified memory and an Apple M4 Max with 128 GB. The tables show generation speed, response time and MLX memory use at different prompt lengths.

Hardware

The model family is Nanbeige4.3-33B throughout. MLX download payloads were 35.21 GB for 8-bit, 26.93 GB for 6-bit and 18.65 GB for 4-bit, using decimal GB. These are file sizes, not total Mac memory requirements.

Apple M1 Max: native MLX performance

The 4K, 8K, 16K and 32K labels mean total test budgets of 4,096, 8,192, 16,384 and 32,768 tokens, not exact input lengths. Each budget reserves 256 output tokens and 64 headroom tokens.

8-bit had the highest prefill rate through 16K. At 32K, 4-bit had the highest prefill rate. A high generation rate does not remove the time needed to process a fresh long prompt.

Apple M4 Max: local-server performance

Each case used 8,192 or 16,384 input tokens, including chat formatting, and 256 output tokens.

Temperature was zero and thinking was off, with a short warmup per model. Only 24 input tokens were cached at 8K and 27 at 16K. Outside-generation time includes prompt processing and server overhead; it is not a first-token timing.

Nearby 8K and 16K inputs: observed rates, not hardware speedups

Both datasets produced 256 output tokens, but the input counts and execution methods differ. The Apple M4 Max server rates are higher in these six observations. The quotients below describe those recorded rates only; they do not isolate the effect of the chip or memory capacity.

Apple M4 Max: long-context retrieval

These native Metal retrieval tests used temperature zero, thinking off, full BF16 KV cache and no rotating cache. Both input sizes produced 10 tokens. The 32K case used 32,736 input tokens.

The 32K/16K total-time ratios were 2.73x for 8-bit, 2.54x for 6-bit and 2.57x for 4-bit. These totals include processing the prompt; they are not generation-speed measurements.

Apple M4 Max: short prompts and coding workflows

Short prompts, model-validation tests and coding workflows measure different work. Their rates are separate from the 8K/16K server results above.

What these results mean for your setup

On Apple M1 Max, 4-bit used less MLX memory and generated faster than 6-bit and 8-bit at every tested budget. On the Apple M4 Max server, 6-bit had the shortest total responses while 4-bit had the highest generation rates. Compare total response time and memory alongside generation speed when choosing a format.

Troubleshooting by symptom

Sources and reference material

Prices are recorded retail planning inputs for the conversion workflow, not a current quote.

The reference package supplies redacted CUDA and Metal results under reference_results/, including validate_mlx_8_result.json, validate_mlx_6_result.json and validate_mlx_4_result.json. Dependency snapshots are supplied as mac_dependencies_lock.txt, linux_converter_dependencies_lock.txt and cuda_dependencies_lock.txt at the package root. Publisher documentation supports the platform procedures; repeat source, compatibility and isolation checks on the target machines.

• Nanbeige publisher repository and pinned revision
• Azure NC A100 v4 sizes
• Homebrew installation and tools
• Azure GPU driver setup
• Azure disk expansion
• Azure managed disk prices
• Azure disk billing
• Azure VM states and billing
• Azure bandwidth pricing
• Azure public retail-price API
• Pinned llama.cpp source
• mlx-lm
• MLX quantization
• vLLM offline inference
• Safetensors format

Rates are the recorded East US retail inputs, not a fresh account quote. Refresh rates and provider availability before a new operation. The operation-duration disk calculation is a separate planning allocation for this tutorial.