When It Comes to AI, Who Is Wrong? Sorry, Who Is Right? Who Is on First? I Don’t Know.
July 15, 2022
When I read revelations about alleged issues with smart software I think about the famous Abbott & Costello routine “Who’s on First?” If you are not familiar with this comedy classic you can find a version at this link.
I read “30% of Google’s Emotions Dataset is Mislabeled.” Who says, “I don’t know.”
The write up asserts:
Last year, Google released their “GoEmotions” dataset: a human-labeled dataset of 58K Reddit comments categorized according to 27 emotions. The problem? A whopping 30% of the dataset is severely mislabeled! (We tried training a model on the dataset ourselves, but noticed deep quality issues. So we took 1000 random comments, asked Surgers whether the original emotion was reasonably accurate, and found strong errors in 308 of them.) How are you supposed to train and evaluate machine learning models when your data is so wrong?
Who? Surgers. What? Yes, what’s in charge of synthetic data? What? Yes. I don’t know. Okay, I don’t know what’s going on in this write up.
Wow.
The article contains some examples of humans mislabeling data. Today these labels are metadata, not index terms or classification codes. Metadata. “I never metadata I didn’t like.” Really.
The article in my opinion is actually pro-Google. Why?
Why’s responsible for Google and its goal of eliminating as many humans from a process once deemed appropriate for subject matter experts. SMEs are too expensive and slow for today’s metadata mavens.
What’s the fix? Synthetic data which relies only a a few humans and eventually (one theorizes) no humans at all. Really? Yes, Really works with Snorkel-type technology.
I enjoyed this statement from the cited article:
If you want to deploy ML models that work in the real world, it’s time for a focus on high-quality datasets over bigger models – just listen, after all, to Andrew Ng’s focus on data-centric AI. Hopefully Google learns this too! Otherwise those big, beautiful traps may get censored into oblivion, and all the rich nuances of language and humor with it…
What? Yeah, I know. What’s in charge of synthetic data. The idea is for Google whopper approach to smart software resolves these issues and others as well. What’s “high quality”? I bet you didn’t know quality requires Google scoring algorithms. What? In the manager’s seat.
Stephen E Arnold, July 15, 2022
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
