LucidWorks Bet on Spark. Now What?
August 8, 2016
Many clear night ago, Lucid Imagination offered an open source enterprise search solution. Presidents came. Presidents went. Lucid Imagination morphed into LucidWorks. I promptly referred to the company in this way: Lucid works, really?
The firm embraced Spark and did a not-unexpected pirouette into a Big Data outfit. I know. I know. Lucid Imagination is a company anchored in key word search, but this is the 21st century. Pirouettes are better than mere pivots, so Big Data it is.
I read “Big Data Brawlers: 4 Challengers to Spark” and the write up triggered some thoughts about LucidWorks. Really.
The point of the story is to identify four open source solutions which do what Spark allegedly does so darned well. Each of these challengers:
- Scales
- Handles Big Data (whatever that means)
- Exploits cheap memory so there are no slug like disc writes
- Does the old school batch processing thing.
What are the “challengers” to Spark? Here are the contenders:
- Apache Apex. Once proprietary, now open source, the software does micro batching for almost, sort of real time functions
- Heron. Another real time solution with spouts and bolts. Excited?
- Apache Flink. This is an open source library with a one two punch: It does the Flink stuff and the Spark stuff.
- Onyx. This is a distributed computation system which will appeal to the Java folks.
What do these Spark alternatives have to do with LucidWorks, really? I think there is going to be one major impact. LucidWorks will have to spend or invest in supporting whatever becomes the next big thing. Recommind hit a glass ceiling with its business model. LucidWorks may be bumping into the open source sky light. Instead of being stopped, LucidWorks has to keep investing to keep pace with what the community driven folks generate with little thought to the impact on companies trying to earn a living with open source.
Stephen E Arnold, August 8, 2016
Comments
One Response to “LucidWorks Bet on Spark. Now What?”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
Spark is also open source of course, although Lucidworks have combined it with Lucene, Solr and proprietary code to build their closed-source Fusion product.