AI: Empty Calories Fuel Smart Software
June 1, 2020
Fast food is a wonder for some people. Zip into McDo, grab two Big Macs, fries, and a chocolate shake. (Is there milk in those?) Bang, zoom, hit the meeting room. After a few years of this accelerated approach to fine dining, the consequences are becoming evident.
The artificial intelligence sector is the digital equivalent of fast food. Limited menu of algorithms, a dash of big data, and massive advertising and marketing hype.
The hype calories would amaze Jenny Craig, a vendor of healthy and delicious meals. Smart software that can recognize faces — maybe half the faces on a good day. Smart software kicking a Go master in the knee was thrilling. Career ending, sure. But it’s an AI victory. Plus, there’s the virtue signaling about AI’s contributions to beating Covid 19. How is that working out? Oh, right. More time, more data needed, more money, and more marketing, Zoom presentations, and tweets. Yes, tweets.
Fast food marketing for smart software has been easy and fun to gobble down. VCs love this stuff. But heart burn? Yes, heart burn. Why?
The fact that artificial intelligence has not made that much progress if the information in “Eye-Catching Advances in Some AI fields Are Not Real.”
The write up states:
But some of the improvement comes from tweaks rather than the core innovations their inventors claim—and some of the gains may not exist at all, says Davis Blalock, a computer science graduate student at the Massachusetts Institute of Technology (MIT). Blalock and his colleagues compared dozens of approaches to improving neural networks—software architectures that loosely mimic the brain. “Fifty papers in,” he says, “it became clear that it wasn’t obvious what the state of the art even was.”
The report, if accurate from MIT, yep, the outfit that took money from everyone’s favorite influencer Jeffrey Epstein, says:
The researchers evaluated 81 pruning algorithms, programs that make neural networks more efficient by trimming unneeded connections. All claimed superiority in slightly different ways. But they were rarely compared properly—and when the researchers tried to evaluate them side by side, there was no clear evidence of performance improvements over a 10-year period.
And the extra side of fries? Check this statement:
Researchers are waking up to the signs of shaky progress across many subfields of AI. A 2019 meta-analysis of information retrieval algorithms used in search engines concluded the “high-water mark … was actually set in 2009.” Another study in 2019 reproduced seven neural network recommendation systems, of the kind used by media streaming services. It found that six failed to outperform much simpler, nonneural algorithms developed years before, when the earlier techniques were fine-tuned, revealing “phantom progress” in the field. In another paper posted on arXiv in March, Kevin Musgrave, a computer scientist at Cornell University, took a look at loss functions, the part of an algorithm that mathematically specifies its objective. Musgrave compared a dozen of them on equal footing, in a task involving image retrieval, and found that, contrary to their developers’ claims, accuracy had not improved since 2006. “There’s always been these waves of hype,” Musgrave says.
Oh, oh. Science Magazine’s write up reports:
Guttag says there’s also a disincentive for inventors of an algorithm to thoroughly compare its performance with others—only to find that their breakthrough is not what they thought it was. “There’s a risk to comparing too carefully.” It’s also hard work: AI researchers use different data sets, tuning methods, performance metrics, and baselines. “It’s just not really feasible to do all the apples-to-apples comparisons.”
The conclusion to the write up is a truism:
Researchers point out that even if new methods aren’t fundamentally better than old ones, the tweaks they implement can be applied to their forebears. And every once in a while, a new algorithm will be an actual breakthrough. “It’s almost like a venture capital portfolio,” Blalock says, “where some of the businesses are not really working, but some are working spectacularly well.”
When progress is slow but the market is hungry, what does an AI expert do? McDonald’s is investing about $200 million to boost marketing. Sounds like a plan. The Big Mac is more than a burger.
Stephen E Arnold, June 1, 2020
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
