What does this methodology measure anyway?

156 points•timpera•about 6 hours ago•59 comments•

59 comments

Magi604about 4 hours ago
Every time I come across an article about cancer, I'm reminded of that website that catalogued all the things that The Daily Mail said either caused or prevented cancer.

https://web.archive.org/web/20200221175201/http://kill-or-cu...

Too bad it's down, but here's an archive of it.

BobbyTables2about 3 hours ago
Watch out for dihydrogen monoxide… It has been found in 100% of all cancer cases but the government pumps it into our houses anyway…
tossandthrow7 minutes ago
This "joke" usually comes up in discussions like this.

Though it is kind of wasted. Dihydrogen monoxide would be the chemical compound without any salts, minerals or other additives we normally get from water.

It would likely not be beneficial for you to drink it as opposed to water.

Machaabout 2 hours ago
Put to song by comedian Russell Howard:

https://youtu.be/q3chJN9DCGg

kreelmanabout 4 hours ago
Is this whole article (the article in this Hacker News post, to be clear) an example of

   "Correlation is not causation"  ?
christina9737 minutes ago
No, because no correlation was proved. It’s just a flawed methodology.
tjwebbnorfolkabout 3 hours ago
The strength of the correlation might be the only useful single variable we can have in discussions like this.

I'm starting to think that "Correlation is not causation" is a kind of slippery slope toward a world where nobody can ever prove the cause of anything. At the same time, just about everything is at least a mild carcinogen including food, the sun, air particulates, etc. But knowing that everything causes cancer isn't useful information. Knowing the strength of the correlations of each of these things is actually useful.

Without correlation, what are we left with?

hatthewabout 4 hours ago
Did I miss it, or do the authors not bother to explain how the original methodology is wrong? Do we just take their word for it?
amlutoabout 2 hours ago
Not that I can see. They claim to have reverse engineered the methodology, but they have not obviously published what they came up with. They also have no explanation as to why they can find positive correlations with all kinds of locations but no negative correlations with anything.
barney54about 4 hours ago
Do you think Costco really is associated with 2.2 million deaths?
hatthewabout 4 hours ago
No, but how do I know that the authors used the same methodology? The most they say about it is "we developed our own methodology by guessing what they did, and we changed stuff around until the results from our methodology matched their results", which is very unconvincing to me.
shoobiedooabout 3 hours ago
Does choking to death on their hotdogs count? It's on my todo list
neoleftyabout 4 hours ago
They didn't! They mentioned receiving "eight lines of code" and, at the end of the article, write:

> It is especially telling that no matter what landmark we applied to the methodology, we have yet to get a negative result. There is, in fact, a real chance that you simply cannot get a negative result from this method.

My guess: The code is not shareable. Maybe its method is laughably disprovable?

dekhnabout 5 hours ago
Cancer risk associations studies are not normally described as "proving" anything. The article being criticized uses 'risk' and 'association'. There is a long history of argument around these sorts of studies, there was a previous series of these around people living near power plants (where it seems like SES, not plant proximity, was the strongest explanatory variable, see https://www.aps.org/archives/publications/apsnews/200710/ele... ). I've seen similar "living near a freeway causes cancer" arguments. There was a massive court case by flight attendants, who have a higher rate of cancer than the regular population.

In reading the criticism, I noticed the authors keep using the term "prove" and they also keep trying to come up with mechanisms ("refueling of the plant"). Even the argument about plant worker exposure compared to people living far away doesn't completely work, because the plant workers are taking all sorts of precautions to minimize exposure to radiation, but there are still mechanisms where something could go out the cooling stacks and deliver something harmful downwind.

The biophysics of cancer causation is entirely nontrivial and looking for the actual sources of the cancer risk is challenging, and watching physics people argue with epidemiologists gets old quickly (my field is biophysics, and I've had a few physics people insist that non-ionizing radiation couldn't possibly cause cancer, "because it doesn't damage DNA". Unfortunately, that argument isn't good, because it presupposes a mechanism (DNA damage due to radiation); we know now that non-ionizing radiation causes cellular heating, stress response, and more, which are all associated (based mostly on in vitro studies) with increased rate of cancer.

Note the authors (and the institute they work for) have vested interest: Dr. Adam Stein is the Director of the Nuclear Energy Innovation program at the Breakthrough Institute, where his work centers on the technology, regulation, economics, and risk governance of advanced nuclear energy.

Deric Tilson is a Senior Nuclear Energy Innovation Analyst at The Breakthrough Institute, where he focuses on advancing nuclear energy as a critical pathway to a carbon-free and energy-abundant future.

codeonlineabout 4 hours ago
I thought the papers author's nearly ten year employment at `Petrofac` is interesting and probably warrents a mention.

> Petrofac designs, builds, manages and maintains oil, gas, refining, petrochemicals and renewable energy infrastructure.

https://hsph.harvard.edu/profile/yazan-alwadi/

https://www.linkedin.com/in/yazan-alwadi-ab1b112b/

dekhnabout 4 hours ago
he was a construction engineer, not a policymaker.
toast0about 4 hours ago
> there was a previous series of these around people living near power plants (where it seems like SES, not plant proximity, was the strongest explanatory variable, see https://www.aps.org/archives/publications/apsnews/200710/ele... ). I've seen similar "living near a freeway causes cancer" arguments.

Social Economic Status is certainly associated with all sorts of things, but being near a freeway means being near brake dust (which used to have asbestos), being near all sorts of tailpipe nasties, being near tire dust, etc. I'd be surprised if it was all explained by money.

dekhnabout 4 hours ago
See- you immediately went to a mechanism that justified the result. It's all too easy to convince yourself something is true because you can see a pathway- yet that pathway might not matter.
Revanche1367about 4 hours ago
Besides the bias question (which is possible and apparently likely for both sides), I realize this is a touchy subject for many but the tone of this article is rather more irritated than it really needs to be. They even mention that the authors of the Harvard paper said the study wasn't meant to show causality, but the very next section heading is "Ridiculous things you can 'prove' caused cancer mortality," which suggests that is what the authors of the original paper were trying to do. If you're going to accuse them of being deceptive, at least be open about it instead of using this sort of passive-aggressive approach.

Also, they say:

"Over the last several months, we have replicated the results of these papers. The authors supplied us with eight lines of code and answered a couple of questions about the covariates, which did not replicate the results. Most of our replication was done through first principles combined with trial and error. Once we were reasonably close to the results of the first national study on cancer mortality, we took the methodology and applied it to numerous other landmarks."

I'm no expert in experiment design, but unless I missed something, I have qualms about calling this method a true "replication."

barney54about 4 hours ago
Do Tilson and Stein have more or less of a vested interest than the people who wrote the studies they are criticizing? It’s not obvious to me that they are more (or less) conflicted.
dekhnabout 4 hours ago
I could go on a long rant on how sociologists, pyschologists, and epidemiologists all play games with data to support their own pet theories and causes, but I won't.

To answer your question: without other data, generally I would expect a person who works for an industry supporting institute to have greater vested interest than academics working at a university- academics mainly just want to get more funding for their individual research, while the industry is dealing with multi-billion-dollar industries and they get paid well to write articles like this. But now that I look carefully, I don't think their institute is funded directly by the nuclear power/plant industry.

msteffenabout 4 hours ago
My understanding was that they take issue with the word "attributable". They don't attempt to propose a mechanism by which living near Costco causes cancer (the most strongly associated of the landmarks they measured). Rather, the implication is that it doesn't. Like I think it's a lot of words arguing that "correlation doesn't prove causation" could be restated "association doesn't prove attribution."
dekhnabout 4 hours ago
Attributable risk is a specific term in epi, the FDA uses it, and it doesn't imply causality.
aljgzabout 3 hours ago
I hoped to see a systematic explanation of what the correlations show. As long as we don't have a viable explanation, there is something interesting there to discover. It can be that the statistical analysis is fundamentally flawed. Pointing out a flaw that tricked several serious researchers is progress. If it was because of some other factor (let's assume there are many more elderly people living around those neighborhoods, can't think of a better example now).
verteuabout 3 hours ago
> Over the last several months, we have replicated the results of these papers. The authors supplied us with eight lines of code and answered a couple of questions about the covariates, which did not replicate the results. Most of our replication was done through first principles combined with trial and error. Once we were reasonably close to the results of the first national study on cancer mortality, we took the methodology and applied it to numerous other landmarks.

Any details of the methodology? Is it possible some mistakes were made during this "trial and error"?

edit: Some aggregated data is available from https://pmc.ncbi.nlm.nih.gov/articles/PMC12929679/ , but far from enough to reproduce: https://media.springernature.com/original/springer-static/es...

Read the full thread on Hacker News →

Related stories