Big Data and Small Data: Tesla and the Gray Lady
February 15, 2013
I don’t want to let this anecdote slip by without capturing it. The addled goose remains quietly in Harrod’s Creek, Kentucky. I ignore the bleats of public relations professionals who “assure me” that unwanted email to me is not spam. You can track the exploits of this PR outfit yourself at North of Nine Communications.
Nope, a more interesting New York style tussle is underway and it concerns what I call small data. In our pulse pounding world of Big Data with IBM Watson tackling cancer, I find small data interesting.
Here’s the story as I understand it. A big newspaper, not too far from the North of Nine outfit, collected some small data about the performance of an electric car. I don’t know about you, but these electric cars stop running when the batteries are exhausted. No big surprise.
The car maker performed various tests and analyses and presented small data to explain that the big newspaper’s small data were incorrect or maybe just out of round. I don’t know, and I don’t care. If you are curious about the status of the dust up, read “Tesla CEO Reveals Evidence against New York Times’ Damning Review in Blog Post.”
My point is that when two outfits cannot agree on small data which presumably both outfits have scrutinized closely, what confidence should you, gentle reader, have in the outputs of Big Data systems. These are based on methods which most folks, including the addled goose, do not understand. Forget the data’s integrity. Let’s just assume that Big Data works like a bulldozer and smoothes out the imperfections.
Well, data and methods don’t smooth out anything. The choices made and the interpretation make a difference for both small data and Big Data.
My point: If you cannot get the small data right, how can we have confidence that Big Data’s outputs, methods, systems, and processes are right? I can’t and won’t. Spats about small data are amusing but the spats illuminate the cloud of craziness which blankets some interesting activities.
Stephen E Arnold, February 15, 2013