Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
-
-
@devsimsek I am the farthest thing from a mathematician but just from a general lay persons perspective the thesis seems a little rigid. Assuming all training data outside the internet is corrupt, assuming that synthetic data only comes from recursive souring, etc. I don't really want to cheerlead AGI as I assume it will bring about a bunch of existential risk but I'm still in the "wouldn't bet against it" camp.
@mike Yeah I see your pov. I just made some remarks and some of them were satirical which you can obviously see through my writing style. I do agree with your stance on rigidity.
Also thanks for the comment
I hope you have enjoyed scrolling through those gifs -
@mike Yeah I see your pov. I just made some remarks and some of them were satirical which you can obviously see through my writing style. I do agree with your stance on rigidity.
Also thanks for the comment
I hope you have enjoyed scrolling through those gifs @devsimsek Forgive me as I digress. Even though I left Twitter years ago whenever I even mildly challenge someone on social media I still "brace for impact" after I press send. It's so refreshing to get a polite and coherent response that resembles civil discourse. Mastodon still amazes me.
-
@mike Yeah I see your pov. I just made some remarks and some of them were satirical which you can obviously see through my writing style. I do agree with your stance on rigidity.
Also thanks for the comment
I hope you have enjoyed scrolling through those gifs @devsimsek @mike I have a vague feeling that there's an argument to be made that a model that can train itself which is then given continuous IO from/to the world (vision, hearing, touch, voice, body) might be able to evolve into a self-sustaining intelligent entity, as long as it has to work for its survival (i.e. work for its input energy).
-
@devsimsek @mike I have a vague feeling that there's an argument to be made that a model that can train itself which is then given continuous IO from/to the world (vision, hearing, touch, voice, body) might be able to evolve into a self-sustaining intelligent entity, as long as it has to work for its survival (i.e. work for its input energy).
@virtuous_sloth @mike On a personal belief I don't think so but this must be tried or at least done an experiment.
I believe someone out there are currently working on it
-
@virtuous_sloth @mike On a personal belief I don't think so but this must be tried or at least done an experiment.
I believe someone out there are currently working on it
@devsimsek @mike I certainly don't think we are anywhere near it nor will we be until someone invents a silicon nerve (simple multiple-input binary output tunable arbitrary functional nerves) that uses very little energy instead of simulated (via linear algebra on GPUs) ones.
-
@devsimsek Forgive me as I digress. Even though I left Twitter years ago whenever I even mildly challenge someone on social media I still "brace for impact" after I press send. It's so refreshing to get a polite and coherent response that resembles civil discourse. Mastodon still amazes me.
@mike always, I am open for feedback that's how I learn at least
Also similar thing happened on reddit; someone called my blog post ai generated, I told them nope its all written by me and they apologized. That's not normal
At least not according to my experience -
@devsimsek @mike I have a vague feeling that there's an argument to be made that a model that can train itself which is then given continuous IO from/to the world (vision, hearing, touch, voice, body) might be able to evolve into a self-sustaining intelligent entity, as long as it has to work for its survival (i.e. work for its input energy).
@virtuous_sloth @mike well that sounds intriguing
-
@devsimsek @mike I certainly don't think we are anywhere near it nor will we be until someone invents a silicon nerve (simple multiple-input binary output tunable arbitrary functional nerves) that uses very little energy instead of simulated (via linear algebra on GPUs) ones.
@virtuous_sloth @devsimsek There is a technology that is using light pathways to create an "Analog" GPU which is showing some promise.
-
@virtuous_sloth @devsimsek There is a technology that is using light pathways to create an "Analog" GPU which is showing some promise.
@mike @devsimsek Interesting.
Also confusing since GPUs are currently being used to do massively-parallel floating-point arithmetic as part of the matrix-matrix and matrix-vector math involved in the linear algebra of the models, and there's not much analog about that I am aware of. I do know that simple resister, capacity, & inductor circuits are analog representations of 2nd-order differential equations, but not sure how that would apply here.
/shrug/ -
@mike @devsimsek Interesting.
Also confusing since GPUs are currently being used to do massively-parallel floating-point arithmetic as part of the matrix-matrix and matrix-vector math involved in the linear algebra of the models, and there's not much analog about that I am aware of. I do know that simple resister, capacity, & inductor circuits are analog representations of 2nd-order differential equations, but not sure how that would apply here.
/shrug/@virtuous_sloth @devsimsek This is the company, however there's not much on their web site. I believe I saw a youtube video explaining its operation. The gist was that the major roadblock was analog to digital conversion to utilize memory. Apologies my memory of the details is vague. https://qant.com/photonic-computing/
-
@virtuous_sloth @devsimsek This is the company, however there's not much on their web site. I believe I saw a youtube video explaining its operation. The gist was that the major roadblock was analog to digital conversion to utilize memory. Apologies my memory of the details is vague. https://qant.com/photonic-computing/
@mike @devsimsek
OK, so it looks like regular linear algebra but using photonics (one of my PhD committee members pioneered in the field of photonics https://en.wikipedia.org/wiki/Sajeev_John) for much higher energy efficiency, presumably by trading digital exactness (hah! IEEE 754 is not exact) for analog computation.Their current produce is 8 GOPS (but does that mean GFLOPS?) and current GPUs are doing TFLOPS, but they seem confident about 1000-fold improvements every 1.5 years, so 2027 here they come!
-
@mike @devsimsek
OK, so it looks like regular linear algebra but using photonics (one of my PhD committee members pioneered in the field of photonics https://en.wikipedia.org/wiki/Sajeev_John) for much higher energy efficiency, presumably by trading digital exactness (hah! IEEE 754 is not exact) for analog computation.Their current produce is 8 GOPS (but does that mean GFLOPS?) and current GPUs are doing TFLOPS, but they seem confident about 1000-fold improvements every 1.5 years, so 2027 here they come!
-
Researchers just mathematically proved that AI can't recursively self-improve its way to superintelligence.
Not "we think it's unlikely." Not "it seems hard." Formally proved.
The model doesn't climb toward AGI — it slowly forgets what reality looks like. They call it model collapse. The math calls it inevitable.
I wrote about it
https://smsk.dev/2026/04/26/ai-cannot-self-improve-and-math-behind-proves-it/
Love it when the mathematical model puts in black and white what most of us intuitively pointed out. Just proves our billionaire geniuses are short on genius and long on shell games.
-
@anne_twain @devsimsek
This reminds me of some right-wing Youtube channels or Telegram groups. For years they're feasting on their own output and in the process are getting intellectually shallower but more agitated and radical... -
@Quantensalat @devsimsek the main issue is that unless you maintain an external signal (so human input in the form of token sequences that are actually carefully curated for coherence) the models become more and more incoherent. Sounds like you're on board with that. The next step is that we're quickly devaluing money spent on human creativity and the world is awash in LLM garbage. So the human signal *is* disappearing.
@wronglang @Quantensalat @devsimsek On the reverse side, we have humans generating human output and then asking LLM chat bots to "improve" that output. What the chat bot deems wrong or improvable causes the elimination of creative personal human voice. Our unique voice gets contaminated and reformed to conformity.
-
@Quantensalat @devsimsek the main issue is that unless you maintain an external signal (so human input in the form of token sequences that are actually carefully curated for coherence) the models become more and more incoherent. Sounds like you're on board with that. The next step is that we're quickly devaluing money spent on human creativity and the world is awash in LLM garbage. So the human signal *is* disappearing.
@wronglang @Quantensalat @devsimsek On the reverse side, we have humans generating human output and then asking LLM chat bots to "improve" that output. What the chat bot deems wrong or improvable causes the elimination of creative personal human voice. Our unique voice gets contaminated and reformed to conformity.
-
@wronglang @Quantensalat @devsimsek On the reverse side, we have humans generating human output and then asking LLM chat bots to "improve" that output. What the chat bot deems wrong or improvable causes the elimination of creative personal human voice. Our unique voice gets contaminated and reformed to conformity.
@jeawhiz @Quantensalat @devsimsek that's a great point--I agree completely! Here's a list of reasons why...
-
@mike @devsimsek
That being said, I have no idea how real or how actually useful it is (but since apparently it can be bought and they have Python libraries, it is not vaporware).@virtuous_sloth @devsimsek Another European company working on photonic computing. One of the partners at the end says something to the effect that if they're successful there will be no need for cloud AI as you could simply run models locally. He should be careful with those kinds of statements when most of the US economy right now is a bet on cloud AI. However I'm becoming more convinced that if they can get it right that this is the future. https://www.youtube.com/watch?v=9tqOPS6x9l8&t=3s
-
@devsimsek Is that a thing people believe, that LLMs generate themselves towards the singularity simply by eating their own output and no other feedback?
@Quantensalat @devsimsek@universe
Well not with that attitude they won't. Why, they just simply need to pull themselves up by their own bootstraps!
.
