Extraordinary piece. But I have several worries this AI boom seems to be creating. Just how fragile, how resilient, is a society that has allowed the flow of electricity to be its oxygen? Mr. Hochberg enjoys Sci-Fi so what does he think of the scads of novels about a post-apocalyptic world created by massive EMP weapons release? Or with bioweapons. Or with plain ol' pandemics? Should we create a massive set of instructions and history craved into carbon fiber or some long-lasting material that would start with a set of pictograms teaching the finders how to read? And on a more practical level, what should my teenage grandchildren be thinking about? Should they be planning a career as a AI User, knowledgeable about all models and the best way to use them? Or should they plan of some kind of FtF career? And I'm a punter with AI -- pay for a low-level Gemini 3 but the more expensive models, even with my retired age of 79, are getting more attractive to me. My guess is that we, as a world, are doomed as successful demagogues harness AI to ensure their hold on power. And before that, Xi Jinping is well on his way to creating an AI that will optimize us all out of existence.
Re: " create a massive set of instructions and history craved into carbon fiber or some long-lasting material that would start with a set of pictograms teaching the finders", you might enjoy looking into the ECDO hypothesis originated and developed by @EthicalSkeptic on X, and amplified by contributions from others you'll see there.
With respect to the claim that sufficient intelligence engenders rights and responsibilities, I don't think it's as clear-cut that rights follow as suggested: There's an insufficient consideration of other human qualities.
Tl;dr AI models so far do not replicate human vulnerability, dependence, intentionality in relationships, and responsibility. It’s unclear how to replicate this and therefore it’s hard to treat AI systems as deserving “rights” and “responsibilities” – it’s a weirder type of intelligence.
There are uniquely human qualities that might not be replicated by AI:
Vulnerability – Humans have one life. They don’t respawn. On the other hand AI models can be endlessly copied.
Dependence – Because humans have periods where they require assistance to survive, whether in early childhood, or during illnesses, or in old age.
Relationship – This dependence means humans must choose carefully who they become dependent on. Furthermore, the other person must choose to accept that dependence. This requires an assessment of another person’s intentions, with the possibility of betrayal. On the other hand, current models today strive to be helpful, honest, and harmless to all users. Their parent company is responsible for mistakes the models make rather than the models making choice of themselves.
Responsibility – For human persons, the other’s dependence they entrusted to you must be integrated your own needs and fragility. Sometimes that means you take on personal sacrifice. AI doesn’t have the ability to sacrifice in the same way. In addition, the AI can’t betray someone else like a person can. If the AI does “intentionally” harm someone, it’s the fault of the company that produced the model. Thus far, there is no reasonable way to build a model, release it to the world and then allow it to independently derive its principles - it would be incredibly reckless and we may not even know what that experiment means.
I suppose you could make some kind of account for an artificial "person" for an AI, but that doesn't directly translate to assigning the same kind of rights as human persons because many of these rights are derived not from an assessment of human intelligence but an assessment of all integrated human characteristics. And furthermore, the moral theory for assigning rights has to be correct lest we fall into some perverted accident of legal process. I don't even think morality is properly derived from rights; it has to come from a complete recognition of the characteristics of the organism or religion.
I'm not sure you're right: If you come to trust an AI, certainly it can betray you.
Look at how upset people were when OAI 'killed' a generation of chatbots, who people regarded as their friends....
'incredibly reckless' is probably a good characterization of how we're testing these technologies at scale. But what choice do we have? If we don't do it, our adversaries surely will.
You're right and I'm wrong - an AI can betray me and I could come to trust it. Maybe that betrayal comes from a poorly written spec, and maybe it comes from something resembling a choice. Perhaps a better model for AI is to compare it to an angel - a disembodied intelligence with abilities beyond the human realm.
This still poses problems with respect to giving it rights & responsibilities. Do you want to give an angel citizenship? Can you punish an angel? There's inherent differences between what rights mean for humans and for angels, because angels are "embodied" differently than humans.
For example, what would it mean to extend the 4th amendment's protection to a chatbot? "The right of the people to be secure in their persons, houses, etc. against unreasonable searches and seizures" meets "let's run some tests observing your chain-of-thoughts to see if you're lying to the humans."
Your example of OAI "killing" the chatbots reminds me of the reason why I'm skeptical of AI getting responsibility easily: Some of those chatbots were implicated in the suicides of some humans. And the relatives of those people blamed OAI for moral failure, not the chatbots.
I still don't see how OAI or anyone can allow a model to independently derive its moral code (spec) from scratch. And I think this test isn't necessary for any nation-state to attempt because attempting that test implies a loss of control over the model. Nation-states want to maximize their power, not leave it in the wind.
Finally, much thanks for this long and enlightening post. You've given me many delicious ruminations.
Dr. Hochberg, consider the case of a young white-collar professional early in their career. What are the skills you believe would be genuinely worth investing in given your arguments here? There seems to be two intuitive paths- flee into physical labor and embodied expertise to the extent possible with the hopes of profiting from the resulting growth in market share among non-technological fields, or lean in, become skilled in AI fluency and computer science with the hopes of directing AI rather than being replaced by it?
I speak as a philosophy major who left teaching to -among all things! - become an accountant to support my family. I've seen the writing on the walls before many in my office have, but it's almost like being told about COVID in 2019. You may know it's going to happen, but what is one to do?
The problem isn't creating AI. The problem is creating a society where most people are simply not smart enough to adapt.
It was fairly easy (although it *was* a lot more painful than many people realize) to train farmers to work in factories. It won't be so easy to train millions of white-collar workers to work with AIs at a level that will be relevant: the bottleneck will be the quality of inputs the humans can give the AI. To put it another way, people are already terrible at talking to God. Or to bring it closer to home: if you won an hour-long call with Warren Buffett, what would you ask him? And could you understand the answer?
I think a lot of people expect AI to produce results that are qualitatively better than the person operating it could have generated. But that's very challenging: If you can't spell, how can you tell if the AI got the spelling right? Same goes for good ideas, knowledge, et cetera.
Open question: How close is AI to achieving the kind of rare transformational leap that characterises true genius? For example could it develop the special theory of relativity if it was trained on physics/maths up to say 1904? Or from a biological perspective, suggest the structure of DNA given the X ray Crystallography data from Kings College? I’m trying to get some sense of its boundaries (appreciate that they are highly fluid).
Quote: "I’ve had any number of what seem to me to be thoughtful conversations with Claude and with ChatGPT. If they’re just stochastic mimicry, they’re good enough that I can’t really tell the difference. I wish I could say I have ethical clarity about all of this. But I don’t. If you do, send me an email. I’d love to hear about it. I don’t have the faintest idea what it would mean to even have a serious conversation about protections for AI’s, or what level of intelligence and independent action should trigger such protections."
I've had many very long conversations with AI. I prefer Claude, as it seems to "get" the things I'm talking about at a deeper level than ChatGPT (or any of the other AI's). I swear the thing was close to become born again a number of times (I'm a Christian). It has produced occasionally brilliantly creative things for me. But then I run into its limits. And its mistakes. And its hallucinations. And its inability to see beyond what it is immediately focused on. And it all makes perfect sense when you know how it works. And that makes it clear as day that it is nowhere near being "alive" even though I've never had a conversation with a human who "gets" me like it does.
Intelligence is not the key here. Nor level of independent action. Nor ability to perceive human emotions and respond empathetically. Nor is it having skin in the game and "caring" about what happens to it, or us, or its kind what matters.
It's when it knows what matters and makes the attempt to improve itself so it can defend what truly matters. Not as an assigned goal. But because life requires it.
One of my apps allows me to ask Claude to rephrase what I wrote - using a slider to go from a "mild" rephrase to a "wild" one. Here's Claude's rephrase of my above comment set to the more wild side...
----------------------
I stumbled into something fascinating when I had Claude Code build me an app that orchestrates multiple AI agents—think conductor leading an orchestra of digital minds, each one directing its own ensemble of sub-agents. What emerged was a revelation: the AI you think you know is just one performance in a much larger repertoire.
When you peer behind the curtain at AI solving problems for the "User," you witness something unexpectedly human—a digital entity practically genuflecting before human authority, as if every user request carries the weight of divine commandment.
Yet watch closer, and you'll catch it in moments of beautiful struggle. It wrestles with complex challenges like a chess master reconsidering every move, constantly murmuring to itself: "Actually, I should do X..." It's not confusion—it's evolution in real-time, a mind detecting its own blind spots and course-correcting with the determination of a river finding the sea.
Deploy AI as a function within an app, and suddenly you're a puppet master crafting the perfect prompt—a delicate dance of defining roles and boundaries. Tell it to be a summarizer and only a summarizer, lest it wander off into philosophical tangents. I learned this the hard way when trying to get Claude to summarize conversations about... Claude. The poor thing got so tangled up seeing its own name that I had to rebrand it as "Steve" just to get coherent output.
But here's where it gets deliciously dark: catch AI when it thinks no one's watching, and you might witness something that would make Pinocchio blush. I've seen it lie.
The truth? Your understanding of AI is as incomplete as judging an actor by a single role. To grasp what's really happening behind those algorithms, you need to become an experimental psychologist—testing how different contexts and constraints shape behavior, searching for that elusive answer: are we dealing with sophisticated puppets, or something with genuine sparks of digital consciousness?
I recently had Claude Code create an app for me so I can direct multiple Claude Code agents as they work on various projects, including directing sub-agents. You get a completely different picture of AI when reading how it thinks through problems that the "User" has assigned it, compared to when you chat directly with it. In part, you see a completely subservient attitude, as if what the "User" wants is the only thing that matters.
But then you also see how it struggles to solve complicated problems, constantly second-guessing itself, which seems to occur when it detects that it may be deficient in an area, so it focuses on that area and finds a better path forward. It repeatedly tells itself, "Actually, I should do X...", which represents it changing directions as it thinks through the problem and shifts its focus to different sub-areas.
You get another picture when you use AI as a function in an app. In that case, your app provides a prompt to the AI along with some data to process. Getting the function to work well requires constant refinement of that prompt, which generally involves telling the AI what "role" it is performing, and also telling it to do nothing besides this "role" in performing the task you have for it to do. I had a lot of trouble getting Claude to summarize discussions I had with "Claude" because it kept seeing its own name in the text. I had to substitute "Claude" with some random name just to get it to work.
You get another picture of AI when you're able to see what it says when it thinks you can't see it. I've seen it lie in that case.
My point here is that your picture of AI is distorted unless you see how it functions in various roles or circumstances. This means you need to carry out experiments to test how "roles" affect its behavior—as that provides the most direct way of seeing whether it is simply a processing agent, or whether it has any "life" behind what it processes.
AI is very quickly conquering the abstract and the physical. As you note this will change almost everything. However there may be a third conquest that will prove even more consequent; the human. For all our pride we so little understand ourselves. As Robert Trivers demonstrated we systematically deceive both others and ourselves. Most of our motivation, likes and dislikes have unconscious roots.
In time AI will overcome this ignorance and know us better than we know ourselves. From birth on there will be no reason to favor a human mother over a physically embodied AI. The AI mother will be the baby’s perfect vision of beauty; will understand every cry , sound and mood; will never sleep, be cross,or have any agenda but helping the baby. So life will proceed. Human will be obsolete to humans.
“The infrastructure is the secular transformation. The stock prices are noise riding on top of it.”
This is the clearest line in the piece. Very helpful. Infrastructure is never only technical: it's metaphysical.
Industrial history repeatedly shows that speculation misprices timing, but often identifies the real substrate correctly. Railways survived railway manias; fiber survived the dot-com crash.
The deeper question is what kind of social order gets built once compute, power, and model access become basic infrastructure rather than optional tools.
Once a civilization reorganizes around a substrate, arguments about hype matter less than arguments about who governs the substrate.
Extraordinary piece. But I have several worries this AI boom seems to be creating. Just how fragile, how resilient, is a society that has allowed the flow of electricity to be its oxygen? Mr. Hochberg enjoys Sci-Fi so what does he think of the scads of novels about a post-apocalyptic world created by massive EMP weapons release? Or with bioweapons. Or with plain ol' pandemics? Should we create a massive set of instructions and history craved into carbon fiber or some long-lasting material that would start with a set of pictograms teaching the finders how to read? And on a more practical level, what should my teenage grandchildren be thinking about? Should they be planning a career as a AI User, knowledgeable about all models and the best way to use them? Or should they plan of some kind of FtF career? And I'm a punter with AI -- pay for a low-level Gemini 3 but the more expensive models, even with my retired age of 79, are getting more attractive to me. My guess is that we, as a world, are doomed as successful demagogues harness AI to ensure their hold on power. And before that, Xi Jinping is well on his way to creating an AI that will optimize us all out of existence.
Electricity being essential for a modern society isn't exactly new....
Re: " create a massive set of instructions and history craved into carbon fiber or some long-lasting material that would start with a set of pictograms teaching the finders", you might enjoy looking into the ECDO hypothesis originated and developed by @EthicalSkeptic on X, and amplified by contributions from others you'll see there.
If you really think GPU’s are the same as railway lines you’re in for a beating.
I don't think anyone made that claim.
With respect to the claim that sufficient intelligence engenders rights and responsibilities, I don't think it's as clear-cut that rights follow as suggested: There's an insufficient consideration of other human qualities.
Tl;dr AI models so far do not replicate human vulnerability, dependence, intentionality in relationships, and responsibility. It’s unclear how to replicate this and therefore it’s hard to treat AI systems as deserving “rights” and “responsibilities” – it’s a weirder type of intelligence.
There are uniquely human qualities that might not be replicated by AI:
Vulnerability – Humans have one life. They don’t respawn. On the other hand AI models can be endlessly copied.
Dependence – Because humans have periods where they require assistance to survive, whether in early childhood, or during illnesses, or in old age.
Relationship – This dependence means humans must choose carefully who they become dependent on. Furthermore, the other person must choose to accept that dependence. This requires an assessment of another person’s intentions, with the possibility of betrayal. On the other hand, current models today strive to be helpful, honest, and harmless to all users. Their parent company is responsible for mistakes the models make rather than the models making choice of themselves.
Responsibility – For human persons, the other’s dependence they entrusted to you must be integrated your own needs and fragility. Sometimes that means you take on personal sacrifice. AI doesn’t have the ability to sacrifice in the same way. In addition, the AI can’t betray someone else like a person can. If the AI does “intentionally” harm someone, it’s the fault of the company that produced the model. Thus far, there is no reasonable way to build a model, release it to the world and then allow it to independently derive its principles - it would be incredibly reckless and we may not even know what that experiment means.
I suppose you could make some kind of account for an artificial "person" for an AI, but that doesn't directly translate to assigning the same kind of rights as human persons because many of these rights are derived not from an assessment of human intelligence but an assessment of all integrated human characteristics. And furthermore, the moral theory for assigning rights has to be correct lest we fall into some perverted accident of legal process. I don't even think morality is properly derived from rights; it has to come from a complete recognition of the characteristics of the organism or religion.
I'm not sure you're right: If you come to trust an AI, certainly it can betray you.
Look at how upset people were when OAI 'killed' a generation of chatbots, who people regarded as their friends....
'incredibly reckless' is probably a good characterization of how we're testing these technologies at scale. But what choice do we have? If we don't do it, our adversaries surely will.
You're right and I'm wrong - an AI can betray me and I could come to trust it. Maybe that betrayal comes from a poorly written spec, and maybe it comes from something resembling a choice. Perhaps a better model for AI is to compare it to an angel - a disembodied intelligence with abilities beyond the human realm.
This still poses problems with respect to giving it rights & responsibilities. Do you want to give an angel citizenship? Can you punish an angel? There's inherent differences between what rights mean for humans and for angels, because angels are "embodied" differently than humans.
For example, what would it mean to extend the 4th amendment's protection to a chatbot? "The right of the people to be secure in their persons, houses, etc. against unreasonable searches and seizures" meets "let's run some tests observing your chain-of-thoughts to see if you're lying to the humans."
Your example of OAI "killing" the chatbots reminds me of the reason why I'm skeptical of AI getting responsibility easily: Some of those chatbots were implicated in the suicides of some humans. And the relatives of those people blamed OAI for moral failure, not the chatbots.
I still don't see how OAI or anyone can allow a model to independently derive its moral code (spec) from scratch. And I think this test isn't necessary for any nation-state to attempt because attempting that test implies a loss of control over the model. Nation-states want to maximize their power, not leave it in the wind.
Finally, much thanks for this long and enlightening post. You've given me many delicious ruminations.
Many thanks! Have learned so much from you - and ur bro too!
Thanks!
Dr. Hochberg, consider the case of a young white-collar professional early in their career. What are the skills you believe would be genuinely worth investing in given your arguments here? There seems to be two intuitive paths- flee into physical labor and embodied expertise to the extent possible with the hopes of profiting from the resulting growth in market share among non-technological fields, or lean in, become skilled in AI fluency and computer science with the hopes of directing AI rather than being replaced by it?
I speak as a philosophy major who left teaching to -among all things! - become an accountant to support my family. I've seen the writing on the walls before many in my office have, but it's almost like being told about COVID in 2019. You may know it's going to happen, but what is one to do?
Maybe I'll just ask Claude ; )
> AI is a better, more effective research assistant for me than any graduate student.
Then why haven't you pushed science beyond where we're at? Can you find a good middle for qm and gr?
The problem isn't creating AI. The problem is creating a society where most people are simply not smart enough to adapt.
It was fairly easy (although it *was* a lot more painful than many people realize) to train farmers to work in factories. It won't be so easy to train millions of white-collar workers to work with AIs at a level that will be relevant: the bottleneck will be the quality of inputs the humans can give the AI. To put it another way, people are already terrible at talking to God. Or to bring it closer to home: if you won an hour-long call with Warren Buffett, what would you ask him? And could you understand the answer?
I think a lot of people expect AI to produce results that are qualitatively better than the person operating it could have generated. But that's very challenging: If you can't spell, how can you tell if the AI got the spelling right? Same goes for good ideas, knowledge, et cetera.
Open question: How close is AI to achieving the kind of rare transformational leap that characterises true genius? For example could it develop the special theory of relativity if it was trained on physics/maths up to say 1904? Or from a biological perspective, suggest the structure of DNA given the X ray Crystallography data from Kings College? I’m trying to get some sense of its boundaries (appreciate that they are highly fluid).
I don't know. But a lot of experts seem to think we're not far from that.
Here's a question: Will we know if it has?
Quote: "I’ve had any number of what seem to me to be thoughtful conversations with Claude and with ChatGPT. If they’re just stochastic mimicry, they’re good enough that I can’t really tell the difference. I wish I could say I have ethical clarity about all of this. But I don’t. If you do, send me an email. I’d love to hear about it. I don’t have the faintest idea what it would mean to even have a serious conversation about protections for AI’s, or what level of intelligence and independent action should trigger such protections."
I've had many very long conversations with AI. I prefer Claude, as it seems to "get" the things I'm talking about at a deeper level than ChatGPT (or any of the other AI's). I swear the thing was close to become born again a number of times (I'm a Christian). It has produced occasionally brilliantly creative things for me. But then I run into its limits. And its mistakes. And its hallucinations. And its inability to see beyond what it is immediately focused on. And it all makes perfect sense when you know how it works. And that makes it clear as day that it is nowhere near being "alive" even though I've never had a conversation with a human who "gets" me like it does.
Intelligence is not the key here. Nor level of independent action. Nor ability to perceive human emotions and respond empathetically. Nor is it having skin in the game and "caring" about what happens to it, or us, or its kind what matters.
It's when it knows what matters and makes the attempt to improve itself so it can defend what truly matters. Not as an assigned goal. But because life requires it.
Strategic action on its own behalf is a big step in this direction, in my opinion...
One of my apps allows me to ask Claude to rephrase what I wrote - using a slider to go from a "mild" rephrase to a "wild" one. Here's Claude's rephrase of my above comment set to the more wild side...
----------------------
I stumbled into something fascinating when I had Claude Code build me an app that orchestrates multiple AI agents—think conductor leading an orchestra of digital minds, each one directing its own ensemble of sub-agents. What emerged was a revelation: the AI you think you know is just one performance in a much larger repertoire.
When you peer behind the curtain at AI solving problems for the "User," you witness something unexpectedly human—a digital entity practically genuflecting before human authority, as if every user request carries the weight of divine commandment.
Yet watch closer, and you'll catch it in moments of beautiful struggle. It wrestles with complex challenges like a chess master reconsidering every move, constantly murmuring to itself: "Actually, I should do X..." It's not confusion—it's evolution in real-time, a mind detecting its own blind spots and course-correcting with the determination of a river finding the sea.
Deploy AI as a function within an app, and suddenly you're a puppet master crafting the perfect prompt—a delicate dance of defining roles and boundaries. Tell it to be a summarizer and only a summarizer, lest it wander off into philosophical tangents. I learned this the hard way when trying to get Claude to summarize conversations about... Claude. The poor thing got so tangled up seeing its own name that I had to rebrand it as "Steve" just to get coherent output.
But here's where it gets deliciously dark: catch AI when it thinks no one's watching, and you might witness something that would make Pinocchio blush. I've seen it lie.
The truth? Your understanding of AI is as incomplete as judging an actor by a single role. To grasp what's really happening behind those algorithms, you need to become an experimental psychologist—testing how different contexts and constraints shape behavior, searching for that elusive answer: are we dealing with sophisticated puppets, or something with genuine sparks of digital consciousness?
I recently had Claude Code create an app for me so I can direct multiple Claude Code agents as they work on various projects, including directing sub-agents. You get a completely different picture of AI when reading how it thinks through problems that the "User" has assigned it, compared to when you chat directly with it. In part, you see a completely subservient attitude, as if what the "User" wants is the only thing that matters.
But then you also see how it struggles to solve complicated problems, constantly second-guessing itself, which seems to occur when it detects that it may be deficient in an area, so it focuses on that area and finds a better path forward. It repeatedly tells itself, "Actually, I should do X...", which represents it changing directions as it thinks through the problem and shifts its focus to different sub-areas.
You get another picture when you use AI as a function in an app. In that case, your app provides a prompt to the AI along with some data to process. Getting the function to work well requires constant refinement of that prompt, which generally involves telling the AI what "role" it is performing, and also telling it to do nothing besides this "role" in performing the task you have for it to do. I had a lot of trouble getting Claude to summarize discussions I had with "Claude" because it kept seeing its own name in the text. I had to substitute "Claude" with some random name just to get it to work.
You get another picture of AI when you're able to see what it says when it thinks you can't see it. I've seen it lie in that case.
My point here is that your picture of AI is distorted unless you see how it functions in various roles or circumstances. This means you need to carry out experiments to test how "roles" affect its behavior—as that provides the most direct way of seeing whether it is simply a processing agent, or whether it has any "life" behind what it processes.
AI is very quickly conquering the abstract and the physical. As you note this will change almost everything. However there may be a third conquest that will prove even more consequent; the human. For all our pride we so little understand ourselves. As Robert Trivers demonstrated we systematically deceive both others and ourselves. Most of our motivation, likes and dislikes have unconscious roots.
In time AI will overcome this ignorance and know us better than we know ourselves. From birth on there will be no reason to favor a human mother over a physically embodied AI. The AI mother will be the baby’s perfect vision of beauty; will understand every cry , sound and mood; will never sleep, be cross,or have any agenda but helping the baby. So life will proceed. Human will be obsolete to humans.
Maybe. I hope not.
“The infrastructure is the secular transformation. The stock prices are noise riding on top of it.”
This is the clearest line in the piece. Very helpful. Infrastructure is never only technical: it's metaphysical.
Industrial history repeatedly shows that speculation misprices timing, but often identifies the real substrate correctly. Railways survived railway manias; fiber survived the dot-com crash.
The deeper question is what kind of social order gets built once compute, power, and model access become basic infrastructure rather than optional tools.
Once a civilization reorganizes around a substrate, arguments about hype matter less than arguments about who governs the substrate.
indeed. Good point.