Facebook: Search Images by the Objects They Contain
July 3, 2017
Has Facebook attained the holy grail of image search? Tech Crunch reports, “Facebook’s AI Unlocks the Ability to Search Photos by What’s in Them.” I imagine this will be helpful to law enforcement.
A platform Facebook originally implemented to help the visually impaired, Lumos (built on top of FBLearner Flow), is now being applied to search functionality across the social network. With this tool, one can search using keywords that describe things in the desired image, rather than relying on tags and captions. Writer John Mannes describes how this works:
Facebook trained an ever-fashionable deep neural network on tens of millions of photos. Facebook’s fortunate in this respect because its platform is already host to billions of captioned images. The model essentially matches search descriptors to features pulled from photos with some degree of probability. After matching terms to images, the model ranks its output using information from both the images and the original search. Facebook also added in weights to prioritize diversity in photo results so you don’t end up with 50 pics of the same thing with small changes in zoom and angle. In practice, all of this should produce more satisfying and relevant results.
Facebook expects to extrapolate this technology to the wealth of videos it continues to amass. This could be helpful to a user searching for personal videos, of course, but just consider the marketing potential. The article continues:
Pulling content from photos and videos provides an original vector to improve targeting. Eventually it would be nice to see a fully integrated system where one could pull information, say searching a dress you really liked in a video, and relate it back to something on Marketplace or even connect you directly with an ad-partner to improve customer experiences while keeping revenue growth afloat.
Mannes reminds us Facebook is operating amidst fierce competition in this area. Pinterest, for example, enables users to search images by the objects they contain. Google may be the furthest along, though; that inventive company has developed its own image captioning model that boasts an accuracy rate of over 90% when either identifying objects or classifying actions within images.
Cynthia Murrell, July 3, 2017
Comments
One Response to “Facebook: Search Images by the Objects They Contain”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
It’s an amazing post in favor of all the web people; they will take benefit from it I am sure.|