AI Card Bao NAS.AICARDBAO.COM

Industry News

Inkling 2 Mini running locally and Claude Sonnet 5.5: 210 tokens per second is the number on the 96GB machine. Others only measured 83 on the same machine.

210 tokens per second, measured on an M5 Ultra Mac Studio with 96GB of RAM. The test was conducted by EXO Labs, and the results were adjusted together with Thinking Machines, the developer of the model.

The same machine, with another person and another runtime, the number is 83.4.

The latter result comes from a public oMLX test, which is also a 96GB M5 Ultra. When the context is 4k, it reaches 83.4 tokens per second. When the context is pulled to 32k, it drops to 56.2. After running the programming agent for a few rounds, the context easily exceeded 32k, so the daily feel is closer to 56 and far away from 210. In the past two days, the saying that "local small model applications are 1.8 times faster and 11 times cheaper than Claude Sonnet 5.5" has been widely spread, and the source is EXO's post. Before making a purchasing decision with it, there are several calibers that need to be aligned.

Six lines on the speedometer

Inkling 2 Mini was released on November 20th, the weight is hung on Hugging Face, about 20GB, Apache 2.0 protocol. It has only been released for a few days, but many people have already run it on their own machines, and the results are far from satisfactory.

machine Operation mode Decoding speed (token/second) Who measured it?
96GB M5 Ultra Mac Studio EXO tuned version 210 EXO Labs self-test
96GB M5 Ultra oMLX, 4k/32k context 83.4 / 56.2 open beta
48GB M5 Max MacBook Pro EXO tuned version 125 EXO Labs self-test
48GB M5 Max MacBook Pro MTPLX three-level quantization package 103.8 to 128.8 MTPLX Release Notes
48GB M5 Pro Original MLX weights 24.0 to 24.8 Community actual test
24GB M4 Pro Part of the computing power falls on the CPU 11.6 (median) third party review

The two lines of M5 Max match up, which means that the 125 range can be reproduced by installing a ready-made bag. The 210 on the M5 Ultra is currently only reported by EXO.

One letter difference, five times difference.

With the same 48GB memory, the M5 Max reaches 125, while the M5 Pro only reaches 24.8. The memory that can hold the model is just the threshold. The speed depends on the memory bandwidth and the number of GPU cores. "The 48GB MacBook Pro also has 125" is often missing the three letters Max in the paraphrase, and then it leads to another conclusion.

Three quantization packages, the fastest one presses the hardest

MTPLX offers three packages for this model. Bare Speed ​​compresses the attention layer, output head and word embedding to 4-bit, with a volume of 16.0GB, up to 128.8; Optimized Speed ​​leaves the attention layer and output head at 8-bit, 17.2GB, up to 123.8; Optimized Quality does not change the official weight, 20.2GB, up to 103.8.

To get the advertised speed, you have to accept the most aggressive compression. Keeping the original weights, the speed is reduced by 19%. Which level of precision EXO’s 210 uses is not stated in the post.

There is another parameter that is easy to miss. Inkling 2 Mini has a total parameter of 26.4B, a mixed expert structure, and each token only activates 3.4B, which looks very light. But it is a cyclic architecture. The middle 12 layers among the 24 layers share weights. Each token has to pass three times. The official caliber is that each token actually passes through 5.4B of weights. Using 3.4B to estimate its speed on your machine will overestimate it by about 60%.

What exactly does that weather app do?

“1.8 times faster” is compared to the total time it takes to build a native weather application from scratch on Mac Studio, and “11 times cheaper” is based on the API price. The two figures are not under the same set of conditions: the speed is compared with the best local hardware, but the cost is replaced by the cloud price. If you really run it locally, the token fee is zero, and the money that should be calculated is the cost of that machine.

In terms of pricing, Thinking Machines’ own interface has an input of $0.15 and an output of $0.50 per million tokens, while the ones listed on OpenRouter are $0.075 and $0.30. Sonnet 5.5 is $2 and $10. Just looking at the output unit price, the difference is 20 times.

But Sonnet 5.5 has five inference gears, and the bill follows the gear. When Artificial Analysis runs its set of intelligence indexes, the max level spits out an average of 193,000 output tokens per question, and the cost of a single question is US$10.02; the low level single question only costs US$0.41. The same agency still wrote in its review on September 28 that a single question was US$7.60. The two numbers did not match, and neither one would change the magnitude. There is also caching on the input side, and the hit price is 5% of the list price, which is $0.10.

The low-end Sonnet 5.5 has an index of 26 points. Inkling 2 Mini is 20 points.

This set of numbers is almost never mentioned in various reports. Adjust the Sonnet to the most economical setting, and it is still a bit higher than the Inkling 2 Mini, with a question costing just over 40 cents; setting it to the maximum is around 55 points, which is no longer the same level. For weather application questions, both the 20-point and 55-point models were completed, so in the end only speed remained. If you change a question that cannot be solved in 20 points, there is no point in doing it 1.8 times faster.

The cloud is not slow either. The output speed of Sonnet 5.5 in xhigh mode is 143 tokens per second, and in medium mode it is 110.6. Put together with the M5 Ultra's 83.4 in the independent test, the local speed does not take advantage. Thinking Machines' own hosting service running Inkling 2 Mini is 280 per second, faster than any Mac.

The hurdles that both sides cannot overcome

The official model card lists three shortcomings of Inkling 2 Mini: it generates seemingly reasonable error content, occasionally does not follow instructions strictly, and performance declines in multiple rounds of long conversations. Other reviews mentioned that its reasoning process may go into a loop and be unable to get out. There are two opinions on the context length. The model card can write up to 1M, and the Tinker document can write 256K. The release announcement did not explain how much memory a 1M window will consume locally. Its self-reported DeepSWE v1.1 score is 48.2%, and Vals Index v2.2 is 46.5%. Both are run by the company itself, and there is no third-party retest for the time being.

The issues with Sonnet 5.5 center on money and control. The high level is very good at writing. The bill for the same question can be 24 times that of the low level. Which level to open is often decided by the agent framework for you. The upper limit of output is 128,000 tokens, and long documents generated will be truncated. All code and data pass through Anthropic's servers, and projects with data that do not leave the intranet are directly out.

It cannot be used if the network is disconnected.

Don’t bother with machines below 32GB.

The official recommended minimum memory limit is 32GB, and Ollama's support range also starts from 32GB Mac. Unsloth has released a GGUF version that can run on 16GB. Someone has run it on the 24GB M4 Pro, with a median of 11.6 tokens per second. This speed is too slow for code completion.

If you have an M5 Max or M5 Ultra and start with 48GB of memory, you can use it as a second model that can be used offline: modify small functions, build scaffolding, and process code that cannot be sent out. The speed of over 100 is enough for you. Cross-file troubleshooting and complex reconstruction tasks are still left to Sonnet 5.5, even if you only open the low file.

In order to run it, I added a 96GB Mac Studio. This account is not easy to calculate at the moment. Based on the low-level single question price of 0.41 US dollars, 4100 US dollars can run 10,000 questions. You can use this number to compare the quotation of that machine.

Related comparisons:"Gemini 4 Hydrogen and Claude Sonnet 5.5: 57% and 70.6% are not the same paper at all" | "Muse Ember, Claude Haiku 5.5, GPT-6 Luna: the same price, the difference in the account of each question is almost three times" | "Hunyuan Hy4 flash, DeepSeek V4.1 Flash 1017. GLM-5.3-Flash: Tencent said it was eight times cheaper, but it was only four times cheaper at 3 p.m. Beijing time.

Related