The fundamental reason for this is simple. Humans are prone to cognitive dissonance. Meaning, we do absurd things to avoid painful thoughts. And anything that questions our sense of identity, is a painful thought.
So if my self-image is, "I've advanced our understanding of the fundamental nature of reality," then the idea that my contributions weren't useful becomes painful. So we avoid thinking it, challenge people who question our past contributions, and so on.
The natural result of this cognitive dissonance is a feeling of undue certainty in our speculations. After all certainty is merely a belief that one idea is easy to believe and its opposites are hard to believe. We imagine that our certitudes are based on fact. But they more easily arise from cognitive biases.
And this is how a group of intelligent and usually rational people descend into theology whose internal contradictions can't be acknowledged.
And reinforces my general below-the-line (layperson) fear about the state of physics today (as reinforced ofc by the likes of Sabine Hossenfelder & Eric Weinstein).
I've been working on how to formulate that idea clearly for a while. It is a problem that goes well beyond physics. For example I believe that the same cognitive error is behind the fact that experts do significantly worse than chance in actually predicting the world, and the more certain the expert sounds, the less likely they are to be right. See https://www.amazon.com/Expert-Political-Judgment-Good-Know/d... for data demonstrating that fact.
Depressingly, this means that we consistently put public policy in the hands of people who are demonstrably incompetent.
>> I've been working on how to formulate that idea clearly for a while. It is a problem that goes well beyond physics.
It's a really fundamental thing in psychology. The solution is something like the destruction of the ego, and many people who push hard enough to be a PhD tend toward larger ego to start with. Meditation and practicing martial arts can help. Apparently psychedelics can as well.
It's a real pain because if you try to tell someone their ego is preventing them from seeing things clearly... Well that's going to trigger the same problem. So yes, it's good to find ways to articulate the message so it can get through to those that suffer from it the most.
This of course still leaves us with an identity. For that I've found that gratitude can help us deal with pain. And so targeted gratitude can help us avoid cognitive dissonance when we otherwise would be overrun by it.
Sadly, neither skill is widely taught in our society.
> The problem is that we need an ego to be healthy. Attempts to destroy it can wind up compromising your mental health.
You only need to destroy it temporarily. When you do it using certain tools or techniques, it will reconstitute by itself once the effect of the tool or technique has passed.
This temporary ego death can open your eyes without creating a permanent void where your ego used to be.
Anecdotally I've seen such claims, but have seen mixed results as well. I've also encountered people who clearly had an ego about how little ego they had. Yes, it was just as ridiculous as it sounds.
I've never seen anything that looks like solid research on the topic.
Given infinite outcomes, experts in a field without a theory of the world that can be used to calculate the future will always perform worse than chance.
Experts of engineering perform better than non experts. However the field of political behavior (or economy) is difficult. The only way to know what's going to happen is wait for it to happen.
Sometimes you know more or less what's going to happen but not the details or the exact outcome. That's enough to make plans.
Examples: at the beginning of 2024 we average persons knew that Putin would win the Russian elections no matter what. We average persons also knew that either Trump or Biden would win the American ones but we didn't know whom. We have to wait. Then surprise, it became either Trump or Harris.
Maybe there are people around the world or even the USA that wonder why Obama don't run for president instead of Harris. They are not experts of the rules of the competition.
So the question is, do the experts predictions are consistently worse than the predictions of any randomly picked person?
Experts who felt certain about their ideas did worse than chance, and worse than simplistic models.
Experts who quantified uncertainty and tried multiple theories did better than both chance or simplistic models.
Sadly, the experts who felt certain presented themselves with confidence and got higher paying jobs.
Normal people were not in the data set reported. He's since done more research on good predictions. You can read Superforecasting by the same author for more.
To extend this, group dynamics can come into play too.
I once worked at a startup that developed fancy new tec. The group dynamic there was that critical thinking absolutely did not exist. The reason was probably that they accepted only people in their circle, that had the same burning positive attitude towards the idea.
This can become a self reinforcing circle, because critically thinking people will leave at some point. (Like sabine did in physics).
> So if my self-image is, "I've advanced our understanding of the fundamental nature of reality," then the idea that my contributions weren't useful becomes painful.
Only if one believes the logical fallacy that the dependent steps of a process of elimination weren't useful.
Even if you believe that they are useful, you're also not going to wind up as a hero in the history books. And so people wind up acting in the same way.
Besides, the argument that all of the bad ideas contributed to discovering the right one, is as strong as the empirical argument that white chairs are evidence that all ravens are black. Logically you're right. Discovering the right idea requires disproving all of the wrong ones. Similarly "all ravens are black" is logically the same as its contrapositive, "all non-black things are not ravens". It's just that you've just decided to focus on a search space that is so much bigger, that each data point in it becomes much less important.
Eh, in many ways the problem is a sunk cost fallacy type issue.
If someone is later in their career and looking at having to throw away all that time - time they will never recover - it takes someone really special to just do it.
And by really special I mean ‘kinda suicidal sometimes’.
Okay. Give it to us. The painful, cognitively harmonant thought which questions our sense of identity but brings further understanding. This idea which all the scientists who have worked on this for decades have avoided thinking about.
This is where climate skepticism comes from by the way. Even climate skeptics will acknowledge that climate scientists are well educated, they don’t deny science as a process of truth seeking, the problem lies in the incentives.
There’s a lot of prestige and grant money that comes with insisting climate change is true.
There’s a lot of political power that gets ceded to the people in charge if we “just accept that we’re in a crisis and us elite are the only ones that can stop it”.
I believe climate change is real and human caused, but many of the claims and doomsday speak feel like self interested humans following their incentives beyond the scientific truth
What are you talking about? I spent years munging climate datasets from various research institutes around the world. The upshot: it doesn't look good for humanity. I seriously don't understand how a neutral third party can walk away from this climate work and think, 'Nothing to see here folks.' The denialism and willful ignorance of the potential catastrophic consequences is something I find terribly disheartening. Mark my words - the temperature records that keep getting broken year after year are going to keep getting broken. Entire towns going up in flames and cities being wrecked by increasingly more powerful hurricanes will be the new normal. <sigh>
Its hard to claim its a narrative created out of grant money incentives when the first people to come to these conclusions and make an acurate predictive model was the oils companies who have tried to deny their own conclusions ever since.
So if my self-image is, "I've advanced our understanding of the fundamental nature of reality," then the idea that my contributions weren't useful becomes painful. So we avoid thinking it, challenge people who question our past contributions, and so on.
The natural result of this cognitive dissonance is a feeling of undue certainty in our speculations. After all certainty is merely a belief that one idea is easy to believe and its opposites are hard to believe. We imagine that our certitudes are based on fact. But they more easily arise from cognitive biases.
And this is how a group of intelligent and usually rational people descend into theology whose internal contradictions can't be acknowledged.