Deepseek Revolutionizes Local AI

Have you tried deepseek lately? A month or so ago I started playing with their new (then) flash model because of some social media support about running it locally. I expected a flash model to be notably less capable than a frontier model, and my experience with Chinese models generally is that they’re typically not as strong as OpenAI or Anthropic models. But I can’t express strongly enough how impressive it is.

In truth, I gave it to my son first for his Minecraft PvP project to play with, and he kept telling me it was amazing, going so far as to beg me to refill his token limit so he didn’t have to use Sol. That was a weird eye-opener, so I dug in too.

After a few days, I was floored. Not one task I gave it came back with clear bugs or gaps. Not one. It was every bit as good as Sol or Fable, it seemed, at least for every-day work. There wasn’t a single occasion where I said “this isn’t working well for this task, let me switch to a stronger model.” Since then it’s only gotten better. They dropped flash 4.1 this week, which is so good that they (almost) discontinued their pro model.

The implications of this are wild, because Flash is small enough to run on local (~expensive) hardware. You can run it on 256 GB of VRAM, which is two networked DGX Sparks, which can be found for around $4,000 each. For that $8,000 you can run 6+ concurrent 1M streams at high speed (30 tok/s) and significantly more at lower context or lower speed. The scaling is very nice.

That means an enterprise or startup could very reasonably create an inference datacenter for local models for their own SE staff, and have better controls on their data.

The evidence here is so strong that local models are going to sweep the field in the coming months. It’s going to take time because DRAM is still being bought up by datacenters so aggressively, but it will move. And when it does, it’s going to be an avalanche.

And if enterprise moves to running local models primarily, then what’s the utility of OpenAI and Anthropic? And what’s the need for all those datacenters?