Spark: Another Open Source Game Changer
September 24, 2015
Gentle reader, I know that knowledge about Spark is as widespread as information about the woes of the Philadelphia Eagles. My understanding of Spark is that is is an open source engine for large scale data processing. It is faster than Hadoop. It is easy to use. It is flexible enough to allow the intrepid Spark aficionado the combine structured query language, streaming, and analytics in one software system. Spark runs “everywhere.” For more about Spark, see this Apache project page.
Spark is one of the next big things, poised to ignite innovation, consulting revenues, innovations, and vendor repositionings.
I approached “Game-Changing Real-time Uses for Apache Spark” in order to learn how Spark can change the game for real time data and information work. Game changing means that old school outfits are going to lose because the new game has new rules, new players, and new everything.
The write up identified these ways Spark will change some quite significant markets:
- Credit card fraud detection
- Network security
- Genomic sequencing
- Real time ad processing
- Medical
My goodness, Spark will become the number one enabling technology for some very problematic market spaces.
Let’s look at what Spark will do to real time ad processing. The write up reports:
One advertising firm uses Spark, on MapR-DB, to build a real-time ad targeting platform. The system looks at user data and decides which ads to show users on the Internet based on demographic data. Since advertising is so time-sensitive, advertisers have to move fast if they want to capture mindshare. Spark Streaming is one way to help them do that.
What strikes me is that Spark requires programmers, software engineering, and then integration of different components. If an error manifests itself, the Spark solution may require those who embrace it to perform some old fashioned work.
In a sense, the game hasn’t changed at all. Open source software reduces license fees and provides a developer with some freedom from license restrictions. On the other hand, the difficult task of getting a complex system to work as intended remains.
My hunch is that Spark is an interesting open source project. The consultants and start ups see Spark as an opportunity. The game changing nature of Spark is potential energy, not a sure thing.
Stephen E Arnold, September 23, 2015
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
