Qwen3.8-flash-next broke the vram requirement with its new architecture (which is a preview of their 4.0 architecture). You can run a 177b parameter model on 12gb VRAM, 64gb ram and 80gb SSD at 30tok/s and the speed scales almost linearly with increases to vram.
Incidentally, this was proven at almost the exact time all the calls for “regulating local models because they’ll spread into the toasters” started.
You absolutely can have local models that run on consumer grade hardware, and the proprietary companies know it and are scared to death.
Are these really able to be as good as chat gpt? Because every time I’ve tried a model like that, it has not been able to be nearly as good. I have 48 gb memory on my machine so I can’t run this particular one though.
I would buy more ram and a whole new computer if the market wasn’t insane.
It is good enough to be useful for programming. It can do clearly defined tasks like find the root cause of a bug.
Using it for writing code is frustrating because it is slow and produces lots of tokens. Meanwhile Opus 5.5 is good enough that you can pretty much rely on it doing exactly what you say (and not what you meant), faster, more context efficient and you get a ton of usage with the cheapest subscription.
I can see this changing, though. If the open-weight models reach the level of Claude 5.5 and Anthropic stops subsidizing they might be on pretty equal footing. I don’t see how further improving the model would help, except maybe if somebody invents a different way to manage context that remembers all the important details over a long time so it actually “learns”.
I won’t pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.
As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.
I haven’t run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.
It is a q3 quantization so there is some small loss of accuracy, but from what I’ve understood those are modest losses in the worst case.
Qwen3.8-flash-next broke the vram requirement with its new architecture (which is a preview of their 4.0 architecture). You can run a 177b parameter model on 12gb VRAM, 64gb ram and 80gb SSD at 30tok/s and the speed scales almost linearly with increases to vram.
Incidentally, this was proven at almost the exact time all the calls for “regulating local models because they’ll spread into the toasters” started.
You absolutely can have local models that run on consumer grade hardware, and the proprietary companies know it and are scared to death.
Interesting.
What an excellent time to have 8gb VRAM and 32gb RAM with the current pricing -.-
Are these really able to be as good as chat gpt? Because every time I’ve tried a model like that, it has not been able to be nearly as good. I have 48 gb memory on my machine so I can’t run this particular one though.
I would buy more ram and a whole new computer if the market wasn’t insane.
It is good enough to be useful for programming. It can do clearly defined tasks like find the root cause of a bug. Using it for writing code is frustrating because it is slow and produces lots of tokens. Meanwhile Opus 5.5 is good enough that you can pretty much rely on it doing exactly what you say (and not what you meant), faster, more context efficient and you get a ton of usage with the cheapest subscription.
I can see this changing, though. If the open-weight models reach the level of Claude 5.5 and Anthropic stops subsidizing they might be on pretty equal footing. I don’t see how further improving the model would help, except maybe if somebody invents a different way to manage context that remembers all the important details over a long time so it actually “learns”.
i would not recommend using the SSD to run local LLMS,its slow
best: VRAM only
good: VRAM + RAM
bad: Swapping
I don’t think they’re insinuating running inferences from the SSD. But you need somewhere to store the weights for the GPU to load.
i am trying to say: you need the SSD to store it but not swap to it due to its slowness
I won’t pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.
As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.
I haven’t run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.
It is a q3 quantization so there is some small loss of accuracy, but from what I’ve understood those are modest losses in the worst case.