Lucene and Integrated Log Data
February 5, 2010
You may find “Into the Cloud: How Search Unlocks Log Metadata to Visualize Your Business Process” interesting if you are an open source technology maven. The idea is that different applications generate log files. When these log files are aggregated, the information that can be searched reveals insights about a business, customers, system issues, etc. The participants are Boomi and Lucid Imagination. Boomi is the “integration cloud company”. You can get more information at www.boomi.com. Lucid Imagination is the company that creates a build of Lucene and Solr that is current, complete, and ready to install. Lucid sells engineering services, and I have a hunch some services will be required to deliver unlocked log data.
After listening to the program, I had several questions:
First, the notion of integrating log files is a good one but I wondered how long it takes to suck big log files, determine deltas, and then update the indexes.
The second question pivots on the usefulness of search for log file analysis. In my experience, we have had to jump through hoops to concatenate certain query results, perform sub queries, and then crunch data. The bigger the log files, the more work these steps were.
Listen to the podcast. The idea is interesting, and I think the market uptake on this idea will be the proof of the pudding.
Stephen E Arnold, February 5, 2010
No one paid me to listen to the podcast or write this article. Too bad. I will report this failure to get paid to the Department of Labor. Too bad I am not a child. I could report myself for unfair practices.
Comments
One Response to “Lucene and Integrated Log Data”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
[…] Arnold asks in a blog post on his useful site Beyond Search: …the notion of integrating log files is a good one but I […]