> My experience in CS is that the replicability of experimental results is embarrassingly bad

I think that's true for low-tier/low-impact CS papers, but the difference is that literally nobody cares about those papers. The high-tier stuff is easily verifiable (like Tensorflow or whatever) and nobody is writing articles about 5% improvements against a benchmark in some obscure niche scheduling and planning domain.

Outside academic CS people are more scientific about experimental results because it has concrete implications on revenue or spend... but they aren't getting the results from conference papers.

The ML part of CS is in a sense funding replication: there are a decent number of papers whose only premise is "we compare 5 recent papers on this benchmark (and fill a couple pages with discussion)" or "we made a new benchmark, here's how commonly cited papers compare (plus a couple pages of discussion)"

Outside the high-profile cases it seems accepted norm that papers perform far worse when scored against somebody else's benchmark. The real measure of quality is how big the gap is.

I'm talking about papers in top PL and Security venues.

No. Psychology is the worst. It is encouraging that an insurgent group is taking replication on but it feels like only about 20% of the field.

Some results do replicate. Big Five personality traits, general classification of mental illness, mainstream IQ results, and human perception and performance. All psychology is not false and there are interesting things to learn from the field.

But the severity and frequency of fraud and non replication is worst in psychology. So much so that half of what you learned in Psych 101 ten years ago does not replicate.

The big challenge for the field is that humans know a lot of psychology. We can track our own thoughts and we are constantly interacting with people and trying to understand their psychology. A biologist who studies ants intensely for 5 years is maybe only one of 100 people have ever watched them that carefully. They'll find lots of new stuff.

Psychology doesn't have powerful techniques than that for determining new truths about humans. It's still mostly give a questionnaire or put people in weird situations.

So there are a lot of researchers hunting around for original ideas that are undiscoverable with our current tech. Some of them are bound to give into the temptation to just make up an interesting result with manufactured data. With p hacking they might not even be committing fraud, they are desperate for a positive result, when the get one they stop looking and publish.

So I would still guess in 2023 that about 80% of "new" discoveries published in journals won't replicate. That rises to 95% with journal publications that are published as university pr and reported in the media.

For coverage of what to remember to unlearn Rolf Degen is great: https://twitter.com/DegenRolf

Popular examples that do not replicate (mostly from Rolf):

Repressed childhood memories Social media harms Search bubbles and echo chambers Women prefer masculine men when fertile Priming (you see a fight in the hallway and then don't cooperate later) Power posing (puff out your chest to feel more confident) Watching eyes make people more honest (in Kahneman and Malcolm Gladwell)

Notice that a lot of these are interesting, and you want them to be true. That's not enough to make it so.

As a bonus, an interesting interview with Daniel Kahneman: https://www.edge.org/adversarial-collaboration-daniel-kahnem...

Other fields have a fairly firm scientific foundation. Whereas it is routinely pointed out by psychologists and psychiatrists that psych has no such thing. The field struggles to authoritatively explain the first thing about human thinking.

Adding to complication in the psych field is the widely observed phenomenon that it draws students with psych problems. Like those who would be willing to fabricate data, for example.

But hey, at least it isn't sociology.

There are resilient exceptions in academia in general, but soft science fields have led the way in the decay of academic standards.

Psychology really isn't the worst field out there. They do at least talk about their problems and claim to care about them, which is better than most fields. They actually do try to replicate each other's results sometimes and publish it when they fail. I used to believe psychology must be uniquely bad, these days I think it's just uniquely honest (!).

There are dozens of well known fields that are built on epistemological quicksand yet which point-blank refuse to admit to or talk about their problems. Because they have a culture of deny deny deny, even when the evidence is overwhelming, they actually get less attention because why bother doing a nice writeup of fraud if you know the result will just be stonewalling? It's better to focus on fields where there might be some actual response, a chance of improvement, no matter how minor.

Isn't that psychiatry? Psychologists aren't necessarily licensed medical doctors who can prescribe.

We could maybe make the conversation about the social penetration of ideas coming out of various disciplines and how invalid research can harm policy or medical practice, but that'd be a completely different discussion than "psychology is utter garbage as a field."

Because no one writes software that can lead to death.

Therac-25.

You can easily kill people with software. It has happened.