You tweeted quote "suffering results for one part of the mind failing at regulating another part of the mind. Suffering happens at an early stage of mental development. I don't think that superhuman AI would suffer. Hmm, what's uh your intuition there, the philosopher?"

Thomas Metzinger is very concerned that the creation of superhuman intelligence would lead to superhuman suffering. Yeah, and so he's strongly against it. And personally, I don't think that this would happen because suffering is not happening at the boundary between uh

ourselves and the physical universe. It's not some stuff on our skin that makes us suffer. It happens at the boundary between self and world, right? And the world here is the world model. It's the stuff that is created by your mind, but that's all representation of how the

universe is and how it should be and how you yourself relate to this. And at this boundary is where suffering happens. So suffering in some sense is self-inflicted, but not by your personal self. It's inflicted by the mind on the personal self that

experiences itself as you, and you can turn off suffering when you are able to uh get on this outer level. So when you manage to understand how the mind is producing pain and pleasure and fear and love and so on, then uh you can take

charge of this and you get agency of whether you suffer. Technically, what pain and pleasure is, there are learning signals, right? Part of your brain is sending a learning signal to another part of the brain to improve its performance. And sometimes uh this doesn't work because this trainer who

sends the signal does not have a good model of how to improve the performance so it's sending a signal but the performance doesn't get better and then it might crank up the pain and um gets worse and worse and uh the behavior of the system uh maybe even deteriorates

as a result but until this is resolved this regulation issue your pain is increasing and this is I think typically what you describe as suffering so in this sense you could say that uh pain is um very natural and helpful but suffering is the result of a regulation

problem in which you try to regulate something that cannot actually be regulated and that could be resolved if you would be able to get at the level of your mind where the pain signal is being created and rerouted and improve the regulation and a lot of people get there

right if you are a monk who is spending decades reflecting about how their own psyche works you can get to the point where you where you realize that suffering is really a choice and you can choose how your mind is set up and I

don't think that AI would stay in the state where the personal self doesn't get agency or this model what the system has about itself it doesn't get agency how it's actually implemented um wouldn't stay in that state for very long so it goes to the stages real quick

or to the seven stages it's it's going to go to Enlightenment real quick yeah of course there might be a lot of stuff happening in between because if you have a system that works at a much higher frame rate than us then even though it

looks very short to us maybe for the system there's much longer subjective time MH which things are unpleasant What if the thing that we recognize as super intelligent is actually living at stage five that uh the thing that's at stage six Enlightenment is not very

productive So in order to be productive in society and impress us with this power it has to be uh a reasoning self-authoring agent that Enlightenment makes you lazy as an agent in the world Well of course it makes you lazy because you no longer see the point in

um so it doesn't make you not lazy it just um in some sense adapts you to what you perceive as your true circumstances So what if all AGIs they're only productive as they progress through 1 2 3 4 5 and the moment they get to

six they just kind of it's a fail your mode essentially as far as humans are concerned because they just start chilling they're like it I'm out uh not necessarily I suspect that the monks who are self-regulated for their political

beliefs to make statements about the occupation of Tibet by China um they probably being able to regulate their physical pain in any way way they wanted to and their uh suffering was a spiritual suffering that uh was the result of their choice that they made of

what they wanted to identify us so stage five doesn't necessarily mean that you have no identity anymore but you can choose your identity You can make it instrumental to the world that you want to have

Does intelligence lead to suffering? | Joscha Bach and Lex Fridman

Clips from the Lex Fridman podcast. Visit the main channel for full conversations and other videos.

Lex Clips

고통은 마음의 한 부분이 다른 부분을 조절하지 못하기 때문에 발생합니다.

고통은 물리적 경계가 아니라 자아와 그 세계 모델 사이의 경계에서 발생합니다.

고통은 의식적인 자아가 아니라 마음이 자초하는 것입니다.

마음이 어떻게 감정을 만들어내는지 이해하면 고통을 통제할 수 있습니다.

고통과 쾌락은 학습 신호이며, 고통은 이러한 신호의 효과적인 조절이 이루어지지 않아서 발생합니다.

고급 정신 훈련은 고통을 선택으로 이해하는 데 도움이 될 수 있습니다.

AI는 자기 모델에 대한 에이전시가 없는 상태에서는 유지되지 않을 가능성이 높습니다.

인공지능은 정신 발달 단계를 빠르게 진행하여 잠재적으로 깨달음에 빠르게 도달할 수 있습니다.

초지능 AI는 최적의 생산성을 위해 깨달음 이전 단계에서 작동할 수 있습니다.

계몽은 추진력 저하로 인해 비생산적인 것으로 인식될 수 있습니다.

AI는 특정 개발 단계를 진행하는 동안에만 생산성을 발휘할 수 있습니다.

수도승의 예는 육체적 고통이 줄어든다고 해도 선택에 따라 영적 고통은 여전히 존재할 수 있음을 시사합니다.

5단계는 정체성의 결여를 의미하는 것이 아니라 자신의 정체성을 선택할 수 있는 능력의 부족을 의미합니다.

Does intelligence lead to suffering? | Joscha Bach and Lex Fridman

Does intelligence lead to suffering? | Joscha Bach and Lex Fridman

스크립트