Disrupting Neural Nets: Adversarial Has a More Friendly Spin Than Weaponized
August 28, 2019
In my lecture about manipulation of algorithms, I review several methods for pumping false signals into a data set in order to skew outputs.
The basic idea is that if an entity generates content pulses which are semantically or otherwise related in a way the smart software “counts”, then the outputs are altered.
A good review of some of these flaws in neural network classifiers appears in “How Reliable Are Neural Networks Classifiers Against Unforeseen Adversarial Attacks.”
DarkCyber noted this statement in the write up:
attackers could target autonomous vehicles by using stickers or paint to create an adversarial stop sign that the vehicle would interpret as a ‘yield’ or other sign. A confused car on a busy day is a potential catastrophe packed in a 2000 pound metal box.
Dramatic, yes. Far fetched? Not too much.
Providing weaponized data objects to smart software can screw up the works. Examples range from adversarial clothing, discussed in the DarkCyber video program for August 27, 2019, to the wonky predictions that Google makes when displaying personalized ads.
The article reviews an expensive and time consuming method for minimizing the probability of weaponized data mucking up the outputs.
The problem, of course, is that smart software is supposed to handle the tricky, expensive, and slow process of assembling and refining a training set of data. Talk about smart software is really cheap. Delivering systems which operate in the real world is another kettle of what appear to be fish as determined by a vector’s norm.
The Analytics India article is neither broad nor deep. It does raise awareness of the rather interesting challenges which lurk within smart software.
Understanding how smart software can get off base and drift into LaLa Land begins with identifying the problem.
Smart software cannot learn and discriminate with the type of accuracy many people assume is delivered. Humans assume a system output is 99 percent accurate; for example, Is it raining?
The reality is that adversarial inputs can reduce the accuracy rate significantly.
On good days, smart software can hit 85 to 90 percent accuracy. That’s good enough unless a self driving car hits you. But with adversarial or weaponized data, that accuracy rate can drop below the 65 percent level which most of the systems DarkCyber has tested can reliably achieve.
To sum up, smart software makes mistakes. Weaponized data input into a smart software can increase the likelihood of an error.
The methods can be used in commercial and military theaters.
Neither humans nor software can prevent this from happening on a consistent basis.
So what? Yes, that’s a good question.
Stephen E Arnold, August 29. 2019
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
