这是旧页,不再更新。回首页 · Old page, no longer updated.
21 NAND
Driver
TapeOut · BNB Chain
impossible LLM · v2

impossible LLM v2 — 5,142,875 个 NAND 门的语言模型a language model made of 5,142,875 NAND gates

它读最近 6 个 MiniCPM5 token 和 4 个随机位,一次求值输出下一个 token;这一页在你的浏览器里逐门计算,没有后端。 v2 相比 v1:不再掉进 “the 2008 , the 2008” 这类死循环(采样时 0%),在对话、网页、中文文本上的一致率分别是 v1 的 4.1、2.1、53 倍。 它仍然写不出连贯的句子,也没有上链。It reads the last 6 MiniCPM5 tokens plus 4 random bits and outputs the next token in a single evaluation; this page computes it gate by gate in your browser, with no backend. Compared with v1, v2 no longer falls into loops like “the 2008 , the 2008” (0% when sampling), and its agreement with the teacher on chat, web and Chinese text is 4.1×, 2.1× and 53× that of v1. It still can't write coherent sentences, and it is not on-chain.

续写Continuation

正在读取网表(压缩后 7.9 MB,解开 36 MB)和分词器(约 2 MB)…Loading the netlist (7.9 MB compressed, 36 MB unpacked) and the tokenizer (about 2 MB)…

每个 token:取文本最后 6 个 token(102 位)+ 4 个随机位 → 5,142,875 个门 → 13 位编码 → token。随机位来自浏览器的 crypto.getRandomValues;勾选「贪心」后恒为 0,输出只由文本决定。Per token: the last 6 tokens of the text (102 bits) + 4 random bits → 5,142,875 gates → 13-bit code → token. The random bits come from the browser's crypto.getRandomValues; with “Greedy” checked they are always 0, so the output depends only on the text.

模型Model

门数Gates5,142,875 NAND · 0 LATCH · 记录 36,000,125 字节36,000,125 bytes of records
接口Interface106 位输入:6 × 17 位 token id(最老在低位)+ 4 个随机位 · 13 位输出(在 8,192 个 token 里的序号)106 input bits: 6 × 17-bit token ids (oldest in the low bits) + 4 random bits · 13 output bits (an index into 8,192 tokens)
教师TeacherMiniCPM5-2B BF16 的 argmax,每段先喂 1,024 个 token 上下文argmax of MiniCPM5-2B (BF16), with 1,024 tokens of context fed in before each segment
数据Data英文维基 1,000 万 · 网页 fineweb-edu 800 万 · 对话 smoltalk 800 万 · 中文维基 600 万 token;每个来源另留 20 万 token 做测试English Wikipedia 10M · web (fineweb-edu) 8M · chat (smoltalk) 8M · Chinese Wikipedia 6M tokens; another 200K tokens per source held out for testing
结构Structure6 层回退查表:看最近 1…6 个 token 的表项 52,751 / 140,400 / 79,659 / 14,780 / 5,078 / 1,049 个,长的优先;都查不到时按 16 格分布输出6-level back-off lookup: 52,751 / 140,400 / 79,659 / 14,780 / 5,078 / 1,049 entries keyed on the last 1…6 tokens, longest match first; if none match, it outputs from a 16-slot distribution
采样Sampling每个表项存教师在该上下文下最常给出的 3 个答案及比例;随机位按比例选(量化到 1/16)。随机位全 0 就是贪心Each entry stores the teacher's 3 most frequent answers in that context and their shares; the random bits pick one in proportion (quantized to 1/16). All-zero random bits means greedy
选表项Entry selection只收比退回下一层能多猜对的上下文,门数按「每门多猜对几次」分配到各层Only contexts that guess right more often than backing off to the next level are kept; gates are allocated across levels by “extra correct guesses per gate”
浏览器耗时Browser time每个 token 约 10 ms(node 实测 9.6 ms)about 10 ms per token (9.6 ms measured in node)
上链On-chain没有流片。按容器成本模型,一步约 116 亿 gas ≈ 170 个区块,实际上不可能上链执行(模型只在 5,300 门以内实测过,这里外推了约 1,000 倍)Not taped out. Under the container cost model one step costs about 11.6 billion gas ≈ 170 blocks, so running it on-chain is practically impossible (the cost model was only measured up to 5,300 gates; this extrapolates about 1,000×)

与 v1 对比(同一批测试文本)Compared with v1 (same test text)

和教师一致的比例Agreement with teacherv1v2 贪心greedyv2 采样sampled
英文维基English Wikipedia0.2350.3090.233
对话Chat0.0740.3030.226
网页Web0.1160.2410.177
中文维基Chinese Wikipedia0.0040.2190.174
MiniCPM5-2B 自身itself约 0.49(维基,v1 测得)about 0.49 (Wikipedia, measured for v1)
80 条提示各续写 48 个 token80 prompts, 48 tokens eachv1v2 贪心greedyv2 采样sampled
卡进短循环的比例Stuck in a short loop100%3.7%0%
平均不同 token 数Mean distinct tokens7.524.334.5

「采样」一致率更低是预期的:采样按比例选答案,不总选最常见的那个;换来的是不重复。Lower agreement when “sampled” is expected: sampling picks answers in proportion rather than always the most common one; what it buys is no repetition.

怎么验证的How it was verified

  • 网表和查表语义逐位一致:全部 293,717 个表项 × 16 种随机位、20 万条随机输入、四个来源各约 20 万条测试窗口,0 处不一致(C 位并行求值器)。The netlist matches the lookup-table semantics bit for bit: all 293,717 entries × 16 random-bit values, 200K random inputs, and about 200K test windows from each of the four sources, with 0 mismatches (bit-parallel C evaluator).
  • 上面的一致率是用网表本身的输出算的,和建表程序的数字逐位相同。The agreement rates above are computed from the netlist's own outputs and are identical to the table builder's numbers.
  • 这一页用的 JavaScript 求值器在 4,000 条输入上与参考语义 0 处不一致;分词和打包与离线流程在 200 条参考提示上一致。The JavaScript evaluator on this page has 0 mismatches against the reference semantics on 4,000 inputs; tokenization and packing match the offline pipeline on 200 reference prompts.

它做不到什么What it can't do

相关页面:Pi 工作台(大脑换成本网表) · v1 存档(54,147 门) · 源码在 tapeout 仓库 onestep_lm/v2/(建表 v2_build.py、分块综合 v2_chunk_synth.py、验证 v2_verify.py + nleval.c)。Related: Pi workbench (with this netlist as its brain) · v1 archive (54,147 gates) · Source is in the tapeout repo at onestep_lm/v2/ (table builder v2_build.py, chunked synthesis v2_chunk_synth.py, verification v2_verify.py + nleval.c).