21 NAND Docs 中文

A large model on a small computer

No FPGA: run binary and ternary models with 1.7 billion parameters on the CPU of a small ARM computer. This page has the measured speeds and how we scale them to a board.

Summary

The models come from Prism ML's Bonsai series (Apache-2.0).

Our own runtime, s6rt

ItemResult
Machinex86 desktop, AVX2, 4 threads
Speed33–34 tokens per second (7.5 with the generic C version, no SIMD)
Does it match?Logits over 24 steps on the same prompt are byte for byte identical to the generic C version, the aarch64 NEON version and the 10-24 golden reference
Date2026-10-30

Measured on x86 (llama.cpp)

llama.cpp, CPU only, generating 64 tokens. One x86 desktop (Ryzen AI Max+ 395). Unit: tok/s.

ModelFile1 thread2 threads4 threadsPeak memory
Bonsai-1.7B Q1_0231 MiB17.431.259.9408 MB
Same, no SIMD (generic scalar path)231 MiB1.492.965.88—
Bonsai-4B Q1_0540 MiB7.36—25.8731 MB
Bonsai-8B Q1_01.07 GiB3.97—14.61,295 MB
Ternary-Bonsai-1.7B PQ2_0 (PrismML branch)436 MiB16.9534.061.0930 MB

How we scale to a board

For each case, compute two ceilings and take the smaller one.

Bandwidth ceiling = effective memory bandwidth ÷ bytes read per token. Each generated token reads all the weights once, so the bytes read are about the file size.

BoardTheoretical bandwidthEffective, as assumed
A55 board (LPDDR4)8–10 GB/s60%, so 5–6 GB/s
JH7110About 11 GB/s2–4 GB/s (weak memory access, to be checked on real hardware)

Compute ceiling:

  1. Use llvm-mca (-mcpu=cortex-a55) to count how many cycles the compute loop takes per weight.
  2. Cycles per token = number of weights × cycles per weight, plus 8% (attention, normalization, activation quantization).
  3. A quad-core A55 at 1.8 GHz with 0.85 parallel efficiency gives about 6.1 billion cycles per second.
  4. tok/s = cycles per second ÷ cycles per token. At a 2.0 GHz clock, multiply by 1.1.

Example, ternary 1.7B:

Compute kernelCycles per weightCycles per tokentok/s
PrismML original implementation0.67About 1.25 billion4.9
Our slice kernel, one row, 4 groups0.41About 760 million8.0
Our slice kernel, 4 rows at once0.385About 720 million8.5

There are 1.72 billion weights. Each token reads 0.46 GB, so at 5–6 GB/s the bandwidth ceiling is 11–13 tok/s. Further optimization would be limited by memory, not by compute.

Quad-core A55 estimate

1.8–2.0 GHz, NEON, LPDDR4. Unit: tok/s.

ModelStock llama.cppBandwidth ceilingOur kernel (first version / fully scheduled)Memory
Bonsai-1.7B Q1_04–621–256–7 / about 120.4 GB, fine on a 1 GB board
Bonsai-4B Q1_01.7–2.69–112.6–3 / about 50.73 GB, safe on a 2 GB board
Bonsai-8B Q1_00.8–1.34.7–5.61.3–1.5 / about 2.51.3 GB, does not fit on a 1 GB board
Ternary-Bonsai-1.7B PQ2_04.911–138–8.5About 0.6 GB (0.93 GB measured on the branch)

On the A55, our binary kernel is only about 1.2 times as fast: the A55 can issue only one 128-bit NEON instruction per cycle, and the llama.cpp path is already close to that floor. The ternary slice kernel needs only one AND plus 1/16 of an SDOT per weight, so it is about 1.7 times as fast.

JH7110 estimate

4 × U74 1.5 GHz, RV64GC + Zba/Zbb, no vector extension. Unit: tok/s.

ModelStock llama.cpp (generic scalar)Bandwidth ceilingOur Zbb cpop kernel (estimate)
Bonsai-1.7B Q1_00.3–0.48–164–6
Bonsai-4B Q1_00.13–0.173.5–71.7–2.6
Bonsai-8B Q1_00.07–0.11.7–3.50.8–1.3

The s6rt runtime we wrote

How much reading TBW1 saves over converting GGUF on the fly (measured on x86, estimated for the A55):

ItemGGUF, converted on the flyTBW1, read directly
Load (layout conversion)1.06–1.35 s0.000 s
Time to first heartbeat1.22–1.31 s0.09–0.17 s
Peak memory948 MB507 MB
Conversion saved on the A55—About 2–3 s

A cold start reads about 460 MB from storage on both paths. At 40–90 MB/s, a microSD card takes 5–12 s, and that is the largest part.

Not done yet