A Question about Robot Scientist Methods
April 13, 2022
I read “Robot Scientist Eve Finds That Less Than One Third of Scientific Results Are Reproducible.” The write up makes a big deal that Eve (he, her, it, them) examined in a semi automated way 12,000 research papers. From that set 74 were “found” to be super special. Now of the 74, 22 were “found” to be reproducible. I think I am supposed to say, “Wow, that’s amazing.”
I am not ready to be amazed because one question arose:
Can Eve’s (her, her, it, them) results be replicated. What about papers about Shakespeare, what about high energy physics, and what about SAIL Snorkel papers?
Answers, anyone.
I have zero doubt that peer reviewed, often wild and crazy research results were from one of these categories:
- Statistics 101 filtered through the sampling, analytic, and shaping methods embraced by the researcher or researchers.
- A blend of some real life data with synthetic data generated by a method prized at a prestigious research university.
- A collection of disparate data smoothed until suitable for a senior researcher to output a useful research finding.
Why are data from researchers off the track? I believe the quest for grants, tenure, pay back to advisors, or just a desire to be famous at a conference attended by people who are into the arcane research field for which the studies are generated.
I want to point out that one third being sort of reproducible is a much better score than the data output from blue chip and mid tier consulting firms about mobile phone usage, cyber crime systems, and the number of computers sold in the last three month period. Much of that information is from the University of the Imagination. My hunch is that quite a few super duper scholars have a degree in marketing or maybe an MBA.
Stephen E Arnold, April 13, 2022
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
