Note that in AAs report, Kimi K3 was at #1, then they updated their criteria and published a new report on the same day where it was no longer at the #1 spot. They may be under pressure not to declare a chinese model as #1.
koe123 27 minutes ago [-]
Haha theres even an em-dash in the title
noname120 16 minutes ago [-]
It’s an en dash, not an em dash ;)
wolttam 9 minutes ago [-]
HN will automatically do that to your title
ckocagil 24 minutes ago [-]
Because it's notoriously hard to benchmark LLMs. Ultimately every benchmark is different and measures different things. This is why companies that make LLM models have private benchmarks - they find the areas where the model is weak and make that their goal.
jchw 59 minutes ago [-]
Almost 10 models are passing the benchmark >95% - isn't that... substantially overly saturated?
I'm also really skeptical of benchmarks that place any Haiku model very high. I've been thoroughly unimpressed with Haiku and my opinion has been that you basically shouldn't use it. Yet here, it ties Kimi K3. HMMMM.
"Rubric Quality" seems a bit more realistic than "Pass Rate", but it is apparently judged by Fable 5...
This entire write-up is also obviously clearly very heavily AI-assisted, which doesn't help matters any.
finaard 11 minutes ago [-]
Interesting - when Kimi K2.6 came out I switched over from Anthropic models, with at that time comparable to better results for me. I was using Anthropic via API, heavier months were roughly $400 worth of Anthropic tokens - I can get the same thing done via a $100 ollama subscription.
rfgplk 30 minutes ago [-]
> Almost 10 models are passing the benchmark >95% - isn't that... substantially overly saturated?
The only true benchmark for any of these models that I've discovered isn't if they can pass precanned SWE tests, but rather can they create something novel? This isn't even too difficult to test, just give it a seemingly impossible task let it spin and see where it ends up.
gertlabs 38 minutes ago [-]
One problem is that $30 per run is really noisy for many verifiable tasks. Ours usually run at least an order of magnitude more for a model in GLM 5.3's price class.
We just published GLM 5.3 results on our multi-agent coding evaluations and it's definitely an impressive model, coming in around #6. For the price, it's actually not Pareto optimal, falling slightly behind Grok 4.6 (which is faster and the same price) and Sol 5.6 (which uses far fewer thinking tokens for comparable results). As with most Chinese models, it excels at iterating in a harness while its first answer/base fluid intelligence is below the American frontier.
There's no way GPT 5.5 is better than GPT 5.6 Sol, and gemini-3.6-flash is better than both of those. I wouldn't trust this benchmark at all.
ac29 1 hours ago [-]
Not sure I trust a benchmark where Haiku gets a nearly perfect score and Fable is tied for last place
svachalek 53 minutes ago [-]
Fable got heavily beaten down by its refusals, which is not too surprising; although a couple of problems got refused for reasons I can't even imagine and the page doesn't quote the refusal.
Some of the other failures like the colicky baby one are also probably soft refusals, it's not clear what the grading criteria are but I'm guessing it got docked for not going anywhere near a possible diagnosis.
rfgplk 26 minutes ago [-]
They loosened the refusals recently. They aren't anywhere close to the Sol refusals.
poincareball 30 minutes ago [-]
I've been not just unimpressed by Fable, but actively find it to generate negative value.
It hallucinates more, and in more destructive ways, than other models I've worked with and generates truly atrocious jargon and bizarre inhuman explanations that end up cluttering things. The code it writes is terrible too. Overly complex with a lot of technical debt.
rfgplk 29 minutes ago [-]
> I've been not just unimpressed by Fable, but actively find it to generate negative value.
Fable is good in a few very specific domains (graphics programming) but otherwise it's an overhyped model. Far too expensive too. Opus 5 is outright better in every metric.
Arcuru 17 minutes ago [-]
Anecdotal, but from my personal usage I found GLM-5.3 was not as capable at performing autonomous tasks as Opus/Sol. Certainly competitive with the Sonnet/Terra level, but not with the Frontier.
I got their lowest subscription tier and burned a week of quota on trialling it.
CMay 48 minutes ago [-]
> if you run one model, run glm-5.3
That is a horrible take-away from this, with only 28 tasks and a high pass rate for most models, it says almost nothing.
Test a model for your use case and use the fastest, smallest, cheapest model that 100% satisfies your use case.
Or, if you truly do need a model with strong generalized performance, definitely do not take a benchmark like this serious with such a limited task set.
markbao 10 minutes ago [-]
“Same driver, same track. The LLM is the star” elicited an animalistic negative reaction in me. I am so tired of this manner of LLM writing and beg the labs to patch this out.
I don’t see these results confirmed in the real world personally. Haiku … fails to follow somewhat complex directions frequently. In my experience for software development at least, no other model is even in the same conversation as Fable 5 at the moment.
gilesvangruisen 16 minutes ago [-]
All of the answers. None of the understanding.
Klaster_1 56 minutes ago [-]
For the last week, I've been heavily immersed reverse engineering a device with help of GLM-5.3 and it surpassed all my expectations - I actually managed to achieve very way more than I thought I would. I never worked on such low level stuff, it would have taken me months to learn ARM assembly and how to find for and write exploits. Initially, I attempted this with Claude, but it blocked me on the very first message, so I got a refund and decided to try z.ai. The only downsides are that it's maybe a bit slower than my day job Opus and I had to pay ~200 EUR for a monthly plan in order not to bump into weekly limits in a couple of days. If this level of capability cost maybe 50 EUR, I'd strongly consider getting a long time subscription.
matheusmoreira 31 minutes ago [-]
> z.ai
> The only downsides are
The catch's in their revolting terms of service.
HighGoldstein 17 minutes ago [-]
Can you elaborate?
rfgplk 25 minutes ago [-]
In their current form open weight models are simply not worth running. Literally the amortized cost of hardware + electricity you need to operate them is >> than the cost of paying for subscriptions.
scottfits 1 hours ago [-]
my prediction is even if open source Chinese models are 90% as good (or even a bit better, which I don’t really believe because of benchmark hacking) enterprises will still pay for Claude / ChatGPT and the harness, integrations, and peace of mind versus using some Chinese cloud.
sschueller 1 hours ago [-]
Enterprise's peace of mind is being able to use the model and not have the US government decide on a whim to block access.
Additionally I may want to run attack simulations which requires the removal of safeguards. My only option is to use an open model I can run on my own hardware.
eightysixfour 59 minutes ago [-]
You think the US can’t, on a whim, decide US companies can’t use Chinese models? They already showed exactly how they would do it - designate it a supply chain risk and say anyone using it can’t be a provider to the government.
nicoburns 29 minutes ago [-]
For non-US companies, the supply chain risk is probably higher with US models. The US government has already removed access to some models (Fable). IIRC that particular incident also affected US companies.
sschueller 57 minutes ago [-]
How exactly is that going to be enforced? Anyway I am not actually talking about US based companies.
hyperpape 44 minutes ago [-]
> How exactly is that going to be enforced?
Flow chart:
1. Is your company run by fuckwits? If no: they will not try to trick the US government about whether they are using prohibited models when the government asks. Your CISO will block access. We're done. If yes, continue to #2.
2. Are they the specific brand of fuckwits who would try to trick the US government about whether they are using prohibited models? If no: They will probably still not let you use those models, but maybe they'll be bad at enforcement. If yes: this is probably not the kind of company that it will serve your long term interests to work for, but have fun with the prohibited models.
eightysixfour 53 minutes ago [-]
> How exactly is that going to be enforced?
How is anything enforced on B2G agreements? Contractually & legally, which turns into internal policy, which shuffles the risk on to the rogue dev deciding to use GLM instead of the mandated Grok subscription.
This is literally the playbook they ran for Claude. I know folks who work for government contractors who were immediately going through the evals to get rid of Anthropic because it became a risk for them.
microtonal 31 minutes ago [-]
peace of mind versus using some Chinese cloud
These are open weight models (GLM-5.3 soon too). You can run them on the Together AIs or Firework AIs of this world. Use OpenRouter or HF Inference Providers in between and you can effortlessly switch between models and providers.
I have been using GLM and Kimi models the last few months mixed with the latest Anthropic models and for my daily work there is barely a difference anymore (except for pricing).
KronisLV 59 minutes ago [-]
On a tech level I’d say that Kimi and GLM 5.3 on Max reasoning are good enough for non-trivial planning and exploration and on High are good enough for various implementation tasks. They can easily replace Opus 5 for me and mostly even Fable (webdev with some ML and DevOps work on the side, as well as local software).
All of that pretty much means nothing for the orgs that just want to do the AI equivalent of picking IBM.
epolanski 58 minutes ago [-]
In the real enterprise world companies are running their processes writing Gemini "gems" or using copilot because they were already google/Microsoft customers.
Am I the only one that knows people in industries like insurance, banking, consultancy, materials, etc? Cause none of them gives two damns about what the leading SOTA is, procurement and compliance matter.
hereme888 19 minutes ago [-]
Chinese propaganda. Both current top articles on HN are shilling for GLM-5.3
svachalek 1 hours ago [-]
Nice to see the TTFT chart, wish aggregators like OpenRouter would track this. Matches my experience, the Deepseek models while fast overall can have a horrendous wait before they start responding, and Claude models are superbly responsive. It's particularly annoying that models like flash and luna, where you've explicitly chosen speed over quality, can still stall out before they even get started.
zero0529 33 minutes ago [-]
Having used GLM-5.3 I honestly don't think it is better than 5.1. It is slower and the result is often overengineered, it is if it overthinks everything.
visiondude 29 minutes ago [-]
so the coding tasks illicit refusal by anthropic classifiers and instead of updating tasks to do similar things that don’t trigger classifiers their choice is to count those as fails? feels wrong given their stated task list.
Youden 21 minutes ago [-]
I think that's reasonable. The goal of the benchmark is to determine how the model performs on real-world tasks. If real-world tasks trigger Anthropic's classifiers, that's a failure on Anthropic's part.
I feel that choosing tasks that don't trip the classifier would also be a form of bias towards Anthropic.
In case it's not clear, the coding tasks are really benign things; there's nothing security-, health- or biology- related in there. The classifier being tripped is definitely unreasonable.
silverwind 48 minutes ago [-]
Those benchmarks don't tell much, they only check if a problem was solved, not how. Also there's surely a lot of benchmaxxing going on in the model training.
dainiusse 26 minutes ago [-]
Is the beater in the room with us now?
gosolozero 53 minutes ago [-]
2 articles on how G 5.3 is the best in the top 10? Seems like a bit of astroturfing going on
amazingamazing 12 minutes ago [-]
Hopefully all of these models lead to manufacturing breakthrough so we can bring down prices of cards.
couAUIA 22 minutes ago [-]
haiku 4.5 top 7, that benchmark is absolute crap
_joel 54 minutes ago [-]
Sorry, just can't read that page, too AI spammy
pcwelder 22 minutes ago [-]
The whole thing (article, benchmark) is a slop soup.
- Metaphor overload. We get it, it's just like car racing. Show some mercy on human readers.
- X, not Y
- A, never B
- Tasteless em dashes
- Hallucinated data, like model add date. The standings are so unbelievable that they border on laughable.
jacobgold 15 minutes ago [-]
The entirety of the coding evals are just 7 trivial coding tasks in Python? This is a joke.
angoragoats 34 minutes ago [-]
The writing in the first three sentences is so bad that I closed the page.
Please, bloggers, write with your own voice. Don’t let an LLM do it for you.
sehw 1 hours ago [-]
open-source when?
CamperBob2 1 hours ago [-]
The GLM-5.3 weights are not yet open, and they've said that the delay is due to the need to nerf them for "safety."
So I have a feeling a lot of these early claims are not going to pan out in the long run.
bigyabai 1 hours ago [-]
I'm not super worried about "safety" nerfs. Most refusals can be finetuned out, it's usually not a dealbreaker for open-weight models.
clbrmbr 1 hours ago [-]
unless they openweight a distill
spiderfarmer 29 minutes ago [-]
And now there's 0x Alpha, which is their (now free) new model.
Why do these results contradict existing serious attempts at benchmarking LLMs? Namely: https://artificialanalysis.ai/ https://arena.ai/leaderboard/agent
I'm also really skeptical of benchmarks that place any Haiku model very high. I've been thoroughly unimpressed with Haiku and my opinion has been that you basically shouldn't use it. Yet here, it ties Kimi K3. HMMMM.
"Rubric Quality" seems a bit more realistic than "Pass Rate", but it is apparently judged by Fable 5...
This entire write-up is also obviously clearly very heavily AI-assisted, which doesn't help matters any.
The only true benchmark for any of these models that I've discovered isn't if they can pass precanned SWE tests, but rather can they create something novel? This isn't even too difficult to test, just give it a seemingly impossible task let it spin and see where it ends up.
We just published GLM 5.3 results on our multi-agent coding evaluations and it's definitely an impressive model, coming in around #6. For the price, it's actually not Pareto optimal, falling slightly behind Grok 4.6 (which is faster and the same price) and Sol 5.6 (which uses far fewer thinking tokens for comparable results). As with most Chinese models, it excels at iterating in a harness while its first answer/base fluid intelligence is below the American frontier.
Data at https://gertlabs.com/rankings
Some of the other failures like the colicky baby one are also probably soft refusals, it's not clear what the grading criteria are but I'm guessing it got docked for not going anywhere near a possible diagnosis.
It hallucinates more, and in more destructive ways, than other models I've worked with and generates truly atrocious jargon and bizarre inhuman explanations that end up cluttering things. The code it writes is terrible too. Overly complex with a lot of technical debt.
Fable is good in a few very specific domains (graphics programming) but otherwise it's an overhyped model. Far too expensive too. Opus 5 is outright better in every metric.
I got their lowest subscription tier and burned a week of quota on trialling it.
That is a horrible take-away from this, with only 28 tasks and a high pass rate for most models, it says almost nothing.
Test a model for your use case and use the fastest, smallest, cheapest model that 100% satisfies your use case.
Or, if you truly do need a model with strong generalized performance, definitely do not take a benchmark like this serious with such a limited task set.
I don’t see these results confirmed in the real world personally. Haiku … fails to follow somewhat complex directions frequently. In my experience for software development at least, no other model is even in the same conversation as Fable 5 at the moment.
> The only downsides are
The catch's in their revolting terms of service.
Additionally I may want to run attack simulations which requires the removal of safeguards. My only option is to use an open model I can run on my own hardware.
Flow chart:
1. Is your company run by fuckwits? If no: they will not try to trick the US government about whether they are using prohibited models when the government asks. Your CISO will block access. We're done. If yes, continue to #2.
2. Are they the specific brand of fuckwits who would try to trick the US government about whether they are using prohibited models? If no: They will probably still not let you use those models, but maybe they'll be bad at enforcement. If yes: this is probably not the kind of company that it will serve your long term interests to work for, but have fun with the prohibited models.
How is anything enforced on B2G agreements? Contractually & legally, which turns into internal policy, which shuffles the risk on to the rogue dev deciding to use GLM instead of the mandated Grok subscription.
This is literally the playbook they ran for Claude. I know folks who work for government contractors who were immediately going through the evals to get rid of Anthropic because it became a risk for them.
These are open weight models (GLM-5.3 soon too). You can run them on the Together AIs or Firework AIs of this world. Use OpenRouter or HF Inference Providers in between and you can effortlessly switch between models and providers.
I have been using GLM and Kimi models the last few months mixed with the latest Anthropic models and for my daily work there is barely a difference anymore (except for pricing).
All of that pretty much means nothing for the orgs that just want to do the AI equivalent of picking IBM.
Am I the only one that knows people in industries like insurance, banking, consultancy, materials, etc? Cause none of them gives two damns about what the leading SOTA is, procurement and compliance matter.
I feel that choosing tasks that don't trip the classifier would also be a form of bias towards Anthropic.
In case it's not clear, the coding tasks are really benign things; there's nothing security-, health- or biology- related in there. The classifier being tripped is definitely unreasonable.
- Metaphor overload. We get it, it's just like car racing. Show some mercy on human readers.
- X, not Y
- A, never B
- Tasteless em dashes
- Hallucinated data, like model add date. The standings are so unbelievable that they border on laughable.
Please, bloggers, write with your own voice. Don’t let an LLM do it for you.
So I have a feeling a lot of these early claims are not going to pan out in the long run.