Is Deep Learning About to Hit a Wall?
August 13, 2020
Given the complexity of deep learning computations, it should be no surprise the technology uses an abundance of computing power. After analyzing analyzed over a thousand arXiv research papers and other sources, a team of researchers has to determined just how much power all this image classification, object detection, question answering, named entity recognition, and machine translation take now and, in theory, will take in the future. Researchers from MIT, the MIT-IBM Watson AI Lab, Underwood International College, and the University of Brasilia contributed to the research. Interesting Engineering summarizes the results in, “Deep Learning Reaching Computational Limits, Warns New MIT Study.” Reporter Loukia Papadopoulos writes:
“They did so by conducting two separate analyses of computational requirements: (1) Computation per network pass (the number of floating-point operations required for a single pass in the network), and (2) Hardware burden (the computational capability of the hardware used to train the model). The researchers found that just three years of algorithmic improvement was equivalent to a 10 times increase in computing power. They concluded that if progress continues along the same lines, deep learning‘s computational requirements will quickly become technically, economically, and environmentally prohibitive. However, all is not lost.
“‘Despite this, we find that the actual computational burden of deep learning models is scaling more rapidly than (known) lower bounds from theory, suggesting that substantial improvements might be possible,’ wrote the coauthors. The researchers found that there are deep learning improvements at the algorithmic level taking place all the time. Some of these include hardware accelerators, field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). Time will tell whether deep learning will become more efficient or be replaced altogether.”
We wonder, if deep learning is replaced by something more efficient, what would that something look like? More marketing?
Cynthia Murrell, August 13, 2020
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
