I think people made some crazy magical assumptions about AI based on scifi that doesn’t apply to real life. Real things to consider and prepare for, but not likely.
Recursive exponential self improvement? Just because something knows how to code doesnt mean it can make the best ultimate next version of itself. Even physical evolution takes millions of years, and it creates mistakes and has setbacks.
We might be seeing logarithmic AI improvement today, like evolution hitting a hill that it can’t cross. We might need trillions more gigabytes of clean training data that isn’t LLM generated to hit the next level, and it might not even be worth it.
I think what the AI developers are doing by constantly promoting ai fear is linking the idea of exponential ai self improvement to it, because their biggest fear right now might be investors realizing that isn’t real, and every dollar they invest is getting less and less back.
Exponential improvement is indeed optimistic - a sigmoid curve (plateauing after a period of increase) is much more plausible, though in the computer programming case I haven’t noticed the plateau yet.
Indeed, throughout nature it’s almost all sigmoids. The trick is that sigmoids look exponential before the inflection point and it’s hard to predict when that inflection point is going to come.
Agreed… I’ve been dabbling in “smart” algorithms for 50 years, the recent (last 8-10 years) progress has been dramatically faster than the previous 40, but each new amazing field: voice transcription, language translation, computer vision object recognition, games mastery, have all rather obviously hit sigmoid-like plateaus. LLM agent software writing has been a slow-burn improvement over the past 18 months - from my perspective it seems like it’s still improving, though that also seems to be a combination of the models getting better, their built in instructions getting better, my local “memory” getting better, and me learning what to challenge it with and what’s unrealistic. A big sign for me is: something I challenged it with 12-14 months ago and got basically nowhere, I tried again last month and it’s made solid progress, delivering a lot of features it couldn’t last year - and those are a lot of features I “gave up on” 5-6 years ago, not because they were impossible, but because they were just too much annoying, time consuming work for the value they deliver to me (personally) - and now the barrier to entry for making those things happen in software is dramatically lower.
Looking at LLM benchmarks over time, there was still doubling of benchmark scores as of 2024 to mid-2025, then it inflected to more modest, incremental improvements, so sigmoid seems about right. Now we are seeing open models rapidly catching up now that the frontier has significantly slowed.
Yeah, improving on 2024 performance was a pretty low bar to clear, by mid-2025 I saw the continuing improvement and decided that even if it was marginal at the time, learning how to use it was probably worthwhile given the improvements that seemed to be coming. Those improvements definitely did come from my perspective. Much of it was in the harnesses - many things I used to have to tell models explicitly, repeatedly in fall of 2025 they started doing without explicit prompting by spring of 2026. I also think I learned what they could be expected to do well and what was a waste of time trying which made me more productive with them as well.
I recently heard that Kimi v3 has made significant progress in code quality - with some people calling it “on par” with Claude. I primarily use Claude Opus - when I tried Fable during their free preview it “felt” even better, but not enough better to shell out a lot of extra cash for personal playtime projects. Kimi isn’t as accessible under the fixed monthly price model so I’m unlikely to try it anytime soon.
Progress has slowed at the frontier and open weight is right on their heels. The general advice is that Fable and the like are best for niche use cases and to make an open weight model your daily driver because they’re almost as good and are a lot cheaper. In my case for personal use, I’m perfectly happy with Qwen 3.8 27B running locally and don’t care to deal with data collection or huge bills to get a bit stronger of a model.
To switch from Claude to Kimi in service forms available to me would be a 5x price increase for how I use Claude (milking the subscription 7 day token limit dry after 5 days pretty consistently.) Decent self-hosting solutions seem to be running around $20K+ in capital equipment and more than $20 per month in electricity costs alone.
I think people made some crazy magical assumptions about AI based on scifi that doesn’t apply to real life. Real things to consider and prepare for, but not likely.
Recursive exponential self improvement? Just because something knows how to code doesnt mean it can make the best ultimate next version of itself. Even physical evolution takes millions of years, and it creates mistakes and has setbacks.
We might be seeing logarithmic AI improvement today, like evolution hitting a hill that it can’t cross. We might need trillions more gigabytes of clean training data that isn’t LLM generated to hit the next level, and it might not even be worth it.
I think what the AI developers are doing by constantly promoting ai fear is linking the idea of exponential ai self improvement to it, because their biggest fear right now might be investors realizing that isn’t real, and every dollar they invest is getting less and less back.
Exponential improvement is indeed optimistic - a sigmoid curve (plateauing after a period of increase) is much more plausible, though in the computer programming case I haven’t noticed the plateau yet.
Indeed, throughout nature it’s almost all sigmoids. The trick is that sigmoids look exponential before the inflection point and it’s hard to predict when that inflection point is going to come.
Agreed… I’ve been dabbling in “smart” algorithms for 50 years, the recent (last 8-10 years) progress has been dramatically faster than the previous 40, but each new amazing field: voice transcription, language translation, computer vision object recognition, games mastery, have all rather obviously hit sigmoid-like plateaus. LLM agent software writing has been a slow-burn improvement over the past 18 months - from my perspective it seems like it’s still improving, though that also seems to be a combination of the models getting better, their built in instructions getting better, my local “memory” getting better, and me learning what to challenge it with and what’s unrealistic. A big sign for me is: something I challenged it with 12-14 months ago and got basically nowhere, I tried again last month and it’s made solid progress, delivering a lot of features it couldn’t last year - and those are a lot of features I “gave up on” 5-6 years ago, not because they were impossible, but because they were just too much annoying, time consuming work for the value they deliver to me (personally) - and now the barrier to entry for making those things happen in software is dramatically lower.
Looking at LLM benchmarks over time, there was still doubling of benchmark scores as of 2024 to mid-2025, then it inflected to more modest, incremental improvements, so sigmoid seems about right. Now we are seeing open models rapidly catching up now that the frontier has significantly slowed.
Yeah, improving on 2024 performance was a pretty low bar to clear, by mid-2025 I saw the continuing improvement and decided that even if it was marginal at the time, learning how to use it was probably worthwhile given the improvements that seemed to be coming. Those improvements definitely did come from my perspective. Much of it was in the harnesses - many things I used to have to tell models explicitly, repeatedly in fall of 2025 they started doing without explicit prompting by spring of 2026. I also think I learned what they could be expected to do well and what was a waste of time trying which made me more productive with them as well.
I recently heard that Kimi v3 has made significant progress in code quality - with some people calling it “on par” with Claude. I primarily use Claude Opus - when I tried Fable during their free preview it “felt” even better, but not enough better to shell out a lot of extra cash for personal playtime projects. Kimi isn’t as accessible under the fixed monthly price model so I’m unlikely to try it anytime soon.
I saw this article today summarizing a Mozilla report that open weight models are about 4 months behind the frontier: https://arstechnica.com/ai/2026/09/exclusive-open-chinese-models-close-gap-with-silicon-valleys-frontier-ai-models/
Progress has slowed at the frontier and open weight is right on their heels. The general advice is that Fable and the like are best for niche use cases and to make an open weight model your daily driver because they’re almost as good and are a lot cheaper. In my case for personal use, I’m perfectly happy with Qwen 3.8 27B running locally and don’t care to deal with data collection or huge bills to get a bit stronger of a model.
To switch from Claude to Kimi in service forms available to me would be a 5x price increase for how I use Claude (milking the subscription 7 day token limit dry after 5 days pretty consistently.) Decent self-hosting solutions seem to be running around $20K+ in capital equipment and more than $20 per month in electricity costs alone.