I see a lot of reliance on containers to make code always available, and I have the same misgivings as you do. It'll work a few years into the future, but what happens once packages aren't compatible with each other/the base container is upgraded/etc.

I've already seen this with older bioinformatics code, which is on old repositories that aren't running anymore, or are very unreliable (but weren't at the time that the code was written). And I'm talking about code that's "only" 15 years old; people will be going to these papers for implementation details long after that point.

Zenodo seems like a good step in the right direction. It should be available as long as CERN is going, shouldn't it? And by that stage it should be "too big to fail".

I’d suggest data management needs to be a well paid career in academia, with senior level influence to enable long term open access to data.

I was on a grant from the Wellcome Trust putting an epidemiology database online. This remained available until the PI moved on and their successor was decidedly less public spirited, so sadly it’s not available anymore. What can I do about that when I’m seen as a technician? I have over 8000 citations from my database work but there’s really no career in academia for me.

I am also in academia and regardless, factually this is not true at all for data, not even remotely (less than like 10% of journals even have data availability policies which are recommendations, and in practice only a small percentage of papers actually make anything available), unless by "publicly available" you mean "available to some academics or academic labs after an often tedious and slow approval process requiring an academic email and various signed agreements". Maybe what you are saying is true in some very specific domains (e.g. machine learning research), but in general what you are saying here is IMO wildly out of touch with present realities in the vast majority of fields, but especially those involving human subjects.

> While I have some sympathy with a lot of your bitterness, this statement is insulting silliness that a quick look at the list of Nobel Prizes in physiology and medicine would prove wrong. Almost all major breakthroughs in the applied sphere stem from decades of basic research that happened just because it interested someone.

Nobel Prizes are so rare they don't speak at all to the generalizations I am making here. Also, much medical academic research is arguably successful because it is in fact ultimately industry-funded or tied to industry. It is of course though highly dependent on the academic subfield, for sure, and I was painting with a broad brush.

If I had to narrow things, STEM academic research isn't so bad, so long as we exclude social science from STEM. Much social science research needs to be defunded ASAP. And I'm not claiming industry research doesn't also have warped incentives. But, on balance, I'd wager outside of pure math/physics and certain more algorithmic/pure domains in comp sci, the smartest people today are going to choose (and be found in) industry, not academia.

The important part here is, as you say, to light candles and insist on rigor. Coincidentally, history tells us that that also gets us further. So by pure memetic selection, this strategy should win

Yes, for the most part the fixes have to be in terms of funding and incentives. Funding needs to be more careful, and more careful funding can be a carrot rather than a stick here.

Re: incentives, IMO we clearly need a stick: there need to be harsh negative consequences for engaging in degenerate research programs and methods that have clearly been shown to result in pathological or cargo-cult science. Null-hypothesis significance testing is one clear practice that needs to go, but building entire fields on phony / meaningless uncalibrated metrics (think: a lot of self-report instruments that are never properly calibrated to objective outcomes or real-world behaviours and/or consequences, with results being reported only as standardized effect sizes) are another more pernicious practice permeating far too many fields. Ideological bias also needs to have funding consequences. Replication issues are still only surface problems in many fields, where the research would still all be worthless even if it replicated 100% perfectly.

> Your analysis completely ignores the physical and biological sciences and the humanities

I admitted later to painting with a broad brush, and yes, it is always hard to generalize and cover everything fairly. But IMO humanities has serious ideological and methodological rigor problems as well, and is overdue for disciplining. I would tend to have stronger positive feelings toward the biological sciences generally, yes. Yes, the social sciences are the major source of the problem (in part because they are so bad they tarnish the reputation of all academia).

> These fields have gotten a lot better over the past decade in the wake of the replication crisis. Preregistration, publishing all code and data, reporting null findings, replicating results, etc. are becoming the norm.

IMO "a lot better" is subjective, and I don't see those things as being the norm yet (beyond as lip-service), and the rate is far too slow. I agree we'll get there eventually, but I am worried about the loss of public trust and thus the production of real knowledge if we don't try a bit harder at this. Plus, globally, countries like China do seem to be more willing to actively crack down on research misconduct, at least in the past years, and it might not be unrelated to them increasingly pulling ahead technologically in many areas.

Everything can be prevented given sufficient knowledge. That's not the point. The point is how to prevent as much as possible.

This has all been profoundly obvious for at least well over a decade or even two now. A consequence has been that too many serious people are driven away from academia and research, to the detriment of science generally.

I've no idea what to do about all this, because people have voiced obvious and easy solutions for decades, but they are all routinely ignored.

> tech and industry, where producing junk actually has real negative economic and personal consequences

In academia, you can just endlessly produce low-quality garbage, and basically make a career out of this. In industry, things more often eventually at least have to work and survive contact with reality. Academia mostly lacks this basic check.

The scientific method should be the north star, sure. Much of what is happening in academia is cargo-cult / degenerate / pathological science though.

> TL;DR I claim that modern science is synonymous with open source software

That's a strong statement that's not supported by the arguments and IMO misguided.

I don't have a problem with "open", but rather with "software".

Both science and software deal with models, however the focus is quite different. I suspect you conflate theory with models.

The goal of science is to produce and test theories — that's an inductive/abductive process. A model, regardless of whether it's reified into mathematical formulas or software, is a means of making a theory operational enough that its consequences can be derived and confronted with observations.

Software often starts downstream of this: it's a reification of theories, models, algorithms, or findings that are the result of research. Of course software can also be used as part of the research process itself. The distinction is roughly the familiar one between research and development.

> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".