CFO Surprises: Making Smart Software Smarter
April 27, 2020
The Cost of Training NLP Models is a useful summary. However, the write up leaves out some significant costs.
The focus of the paper is a:
review the cost of training large-scale language models, and the drivers of these costs.
The cost factors discussed include:
- The paradox of compute costs going down yet the cost of processing data goes up—a lot. The reason is that more data are needed and more data can be crunched more quickly. Zoom go the costs.
- The unknown unknowns associated with processing the appropriate amount of data to make the models work as well as they can
- The wide use of statistical models which have a voracious appetite for training data.
These are valid points. However, the costs of training include other factors, and these are significant as well; for example:
- The directs and indirects associated with creating training sets
- The personnel costs required to assess and define retraining and the information assembly required for that retraining
- The costs of normalizing training corpuses.
More research into the costs of smart software training and tuning is required.
Stephen E Arnold, April 28, 2020
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
