I spoke with Mr. Doscher when he was working for the Dassault Exalead organization. When he shifted to Lucid Imagination, I spoke with him about his views of open source search. After that brief initial conversation, I met again with Mr. Doscher and probed into his views about the impact open source search is having on traditional for-fee, proprietary search systems.

When I asked about the shift from proprietary search systems to open source search, he told me:

Today organizations need the flexibility to adapt and make changes. A proprietary solution may not permit the licensee to make enhancements. If a change is made, the proprietary search vendor may “own” the fix and will add that innovation to its core product. The licensee who created the fix gets nothing and may have had to pay for the right to innovate. As corporate information technology struggles to keep up with escalating business information demands and an ever increasing mountain of growing content of all types, open source search provides a cost effective and efficient way to develop applications to address the challenges and opportunities in today’s enterprise.

Mr. Doscher has strong views about how licensees of enterprise search systems have learned about costs, the time required to deploy a system, and the effort needed to keep a search system up and running. I asked him about Lucid Imagination’s approach to a search engagement. He said:

Our approach to an engagement is to listen to what our customers need, prepare an action plan, and then deliver. In a sense, our approach is the type of involvement that many software companies have stepped away from. We have an enthusiastic group of engineers and professionals who work with clients to meet their needs.

The full text of the interview appears on the ArnoldIT.com Web site. For more information about Lucid Imagination’s open source search system, you will want to explore the company’s Web site and its blog. In addition, an interview with one of the founders of Lucid Imagination, Marc Krellenstein, and with Eric Gries, a former executive at Lucid Imagination, is available in the Beyond Search archives.
Stephen E Arnold, April 16, 2012

Sponsored by Pandia.com

IKANOW, founded in 2010, is focused on the intersection of analytics and big data. The company’s motto is “The power to act.” The company’s approach combines cutting-edge analytic methods and open source technology to produce innovative ways to distill meaning from digital data and information. Unlike traditional key word search, IKANOW delivers solutions which reveal insights.

We learned about IKANOW when we were ramping up our new open source analytics information service “TheTrendPoint.com”. As we were sifting through the companies offering open source analytics, IKANOW stood out.

David B. Camarata told me:

IKANOW was created from observations and lessons learned made by my colleagues and I after working hand in hand with Department of Defense and intelligence community over the past decades. What we learned from hundreds of projects was that new ways of thought were required to solve the problems and challenges, which were coming with increasing speed and from many different angles. After a very successful and rewarding career in building tightly woven integrated systems, I became aware of the value of next-generation analytics. I wanted to build a company around the core idea of “the power to act.” I don’t want to spend hours trying to think up key words that will unlock the information in a system. I don’t know what information is in the system so how can I know what there. That’s when I hit on the idea of “I can know.”

With university computer science departments and entrepreneurs around the world jumping into the analytics market, the question becomes, “What sets IKANOW apart?” Mr. Camarata said:

We have some breakthroughs, but I want to be careful in how I explain our approach. Our research activities have been focused on creating simple and scalable ways to harvest, unify and expose unstructured and structured information. The breakthroughs we have achieved yield what we call “enhanced reasoning.” The core of our approach pivots on our methods for addressing the three legs of the decision stool. There is the platform which must be scalable, extensible, and performant. Next, we have to be able to harvest and enrich data and information. In their raw form, data lacks context and may be meaningless. A range of numerical methods and semantic techniques are required to figure out what significance something has. Finally, we created a robust application programming interface so developers and our licensees can tap into the platform. At the heart of the technology is a flexible semi-structured NoSQL data model that can be easily tailored to specific business problems.

You can read the complete, exclusive interview at the ArnoldIT.com Web site in its Search Wizards Speak feature. The interview with Mr. Camarata is now available. For more information about IKANOW, navigate to the company’s Web site at www.ikanow.com. My view is that IKANOW is definitely “performant.”

Stephen E Arnold, April 9, 2012

Sponsored by Pandia.com

I interviewed Dr. McIsaac on February 15, 2012, and the information I gathered struck me as important and highly relevant to enterprise search. With users of enterprise search systems expressing significant dissatisfaction with incumbent systems, licensing a new search engine should make people happy. Well, it does not. Traditional methods of figuring out what makes a user of a findability purr does not work particularly well.

Dr. McIsaac’s method may provide a solution. I have put the full text of our conversation on my Search Wizards Speak subsite. You can access the interview at this link.

Here’s an example of the outputs her research approach:

Xyte provides a system science model and a structure that identifies the way people function intellectually by classifying their cognitive information-processing capabilities presented as a logical system of relationships among various human capabilities. In simple terms, Xyte’s system provides a structure based on a proven theory which is predictive of the way individuals think and process information and then act consistently. No other system is predictive.

I was able to gather some information about the type of findings she delivers to clients. She told me:

Xyte surveyed each of what we call cognofiles or behavioral sets of the Xyting Insight™ system about usage of social media to determine who is most apt to use these computer applications. Not all consumers are eager to use Facebook or use it consistently. According a recent Xyte study, 27 percent of the population has never logged in to Facebook and another 20 percent only log in once or more a month. Data from our study suggest that some of the broad generalizations about social media, particularly as a replacement for a search system or face-to-face interaction are inaccurate. That means that advertisers are missing 47 percent of the population. However, 38 percent do log in daily and 16 percent do log in several times a week. The population segment that most frequents Facebook has a number of characteristics; for example, showing great compassion for others, wanting to be emotionally connected with others, having a natural intuition about people and how to relate to them, adapting well to change, embracing technology such as the Internet, and enjoying gossip and messages delivered in story form and liking to read and write.

For more information about Xyte, navigate to www.xyte.com. The complete interview is at http://www.arnoldit.com/search-wizards-speak/xyte.html

Stephen E Arnold, February 21, 2012

Sponsored by Pandia.com

“Social Media Analytics Expert Interview Series: Part 2” is conducted by the Chief Editor of Text Analytics news, Ezra Steinberg. The interview panel includes: Meta Brown, General Manager of Analytics at LinguaSys; Christine Campbell, Director of Marketing at Socialware; and Pirouz Nilforoush, President & Co-Founder of NetShelter Technology Media. All three interviewees will be speaking at the Social Media Analytics Summit in San Francisco in April. The interview sheds some light on customer interaction; some helpful questions and responses from the interview follow:

 “USM:  What do you believe the average consumer thinks about companies’ social media listening initiatives?

Nilforoush (Netshelter): I think the average consumer is confused as to why different brands are initiating conversations with them online that can resemble advertising or spam. Brands need to focus their efforts around engaging their top influencers, rather than trying to engage with every single person that has something to say about their brand. It is not a scalable model for the brands and can be annoying for the end user. Instead, brands should focus their efforts on the people that have the biggest impact on their brand. These influencers will do the work for brands on their own and impact the masses.

USM:  What would you tell someone who is thinking about employing social media analytics for their company?

Brown (LinguaSys): Start with just one narrow project tied to a specific business problem. Choose something where you feel confident that quick improvement is possible. Plan carefully – what’s the path from data collection to analytics to action to returns? Give yourself the best opportunity to succeed – don’t begin until you have made a plan that gives you a way to demonstrate measurable value for your investment in social analytics!

The interview focuses on planning for implementation of social media analytics and consumer’s thoughts on the topic. Many organizations would benefit by considering the opinions and thoughts provided by these leaders in social media. The full interview can be found here and can give insight on building relationships via social media and what to anticipate during the process.

Andrea Hayden, February 02, 2012

PolySpot is a provider of open search solutions. The company offers a robust and innovative architecture which supports search-centric applications accessible from any device connected to a client’s network.

I was interested in Mr. Andre’s view of PolySpot. The search and content processing sector is in transition, and the role of open source solutions continues to gain traction. He told me:

PolySpot’s agile framework, its use of open source technology like Lucene, and a focus on putting information in the business work flow. Olivier Lefassy, David Fischer – our CTO – and I had designed some interesting ideas, and I was eager to fine tune these elements into a business model that would propel PolySpot over the hurdles which cause many enterprise information solutions to fail.

With open source making in roads at IBM and other major technology providers, I asked about Mr. Andre’s involvement in the “communities” which play an important role in the sector. He told me:

When I was board member at Talend, a very successful French initiative in the ETL [extract, transform, load] segment from inception in 2006 to December 2010, I came to understand the potential of open source software. PolySpot gives me a chance to leverage my knowledge about fast growth, high potential companies, open source software, and the “big data” opportunity around us. I think you can say that data management and information are woven throughout my business fabric.

The PolySpot approach boasts a robust framework. I asked what PolySpot has constructed around Lucene, the open source search system:

We build the connectors I mentioned before and a connector software development kit. We engineered out proprietary transformation and enrichment platform (that’s the Sense Builder components) which adds intelligence to raw information. We also developed a very innovative end to end administration console enabling to design and maintain search applications with no particular technical skill, this eases Lucene and Solr configuration but also amplifies the search functionalities provided by Solr. Last, we have added display modules, information views, and graphical user interfaces. These can easily be customized. To make it brief, PolySpot delivers the first end-to-end packaged search infrastructure over Lucene and SOLR core technologies.

After seeing several demonstrations of client deployments, I was impressed with the PolySpot technology. To learn more about PolySpot’s solutions and technical approach, navigate to www.polyspot.com. The full text of the interview with Mr. Andre is located in the ArnoldIT’s series Search Wizards Speak at this link.

Stephen E Arnold, December 13, 2011

Sponsored by Pandia.com, publishers of The New Landscape of Enterprise Search

You can find the interview at this link. The interview carries the date of September 27, 2011. Our error. At age 67, I lose my pen several times a day.

Stephen E Arnold, December 11, 2011

In October 2011, I had a chance to talk informally with Dr. Jerry Lucas, an expert in telecommunications and the owner of the Telestrategies conference series. I was quite interested in his views about content processing. His interest spans text and the large volumes of information that accrue in modern telecommunications systems. One theme which threaded thought his observations was the large volume of data that is now available in digital form. I don’t want to denigrate the commercial services who chit chat about “big data” for figuring out which soap detergent is perceived as having a “smiley face” on the brand. I do want to point out that the Telestrategies’ conferences are designed for law enforcement agencies, intelligence professionals, and practitioners who either work as advisors to agencies or as product developers. Put that your AtomicPR water balloon, insert XML tags, and spam the connected world.

The full text of the interview with him appears as part of the Search Wizards Speak series, which is the largest, free collection of first person narratives about information retrieval. The full text of my conversation with him is at this link. The master index for the series is available on the Beyond Search Web site at Wizards Index.

I wanted to highlight two points Dr. Lucas made in our discussion.

First, I asked him, “What is your view of the challenges flows of digital information pose to government professionals working in law enforcement or the intelligence community?” He told me:

First and foremost are the lack of updated laws creating new lawful interception mandates. In the US the last technical mandate law passed by Congress was the Communications Assistance for Law Enforcement Act. CALEA was passed in 1994 and enacted in 1995. The key players providing today’s communication services used by bad guys—specifically, Apple, Facebook, Google, Second Life, Skype, etc.–are not covered by CALEA mandates nor any other interception assistance laws. These companies have to respond to court orders but these companies don’t have to deploy any infrastructure features to assist law enforcement. I think this is a challenge which must be resolved. A second big challenge law enforcement and intelligence professionals is the lack of educational and budget support by their senior management. As you know, today’s senior management developed professionally in their careers depending on voice calls and e-mail messaging as their prime electronic communications tools. Today many senior managers still make phone calls and send e-mails during working hours and likely watch TV during off hours. So here is my point. To understand what’s needed to police a community you have to live in that community. I call this Policing 101. But Today’s senior managers usually don’t live in the Facebook, Twitter, Skype, Second Life and other cyber space environments as part of their every day activities.

Second, I asked Dr. Lucas, “You have a unique vantage point on some quite interesting technologies. If you were to advise a developer at a large firm specializing in digital information analysis, what would be the three most important features the company should include in their next product release?”

I know you are aware of the phenomenal requirements regarding data privacy or who can access data in a law enforcement agency. Privacy policies and safeguards for open source search in an enterprise can be very lax with regard to a law enforcement agency. Data gathered on a bad guy from a communications service provides under a court order is not fair game for searching over time by law enforcement professional. Those data may have to be erased over time and more. So compliance is an essential characteristic of many products and service. Second, product feature to consider is interoperability with legacy lawful intercept and intelligence gathering products. Interoperability is very important. So called “fork lifts” are rare events in this space and no one in this space wants to see an additional screen introduced in the central monitoring center. I want to emphasize that the user interface must be simple and shouldn’t require the user to be highly computer literate.

I found that the Apple influence is extending beyond the iPhone and iPad crowd. For more of Dr. Lucas’ insights and views, point your browser to the Dr. Jerry Lucas Interview. Information about Telestrategies is located at www.ISSWorldTraining.com.

Stephen E Arnold, November 14, 2011

Sponsored by Pandia.com

OpenSearchServer project started in a French B2B media group in 2007. The company was looking for a search solution. The project emerged when a group of engineers became frustrated with the commercial search solutions.

Raphael Perez said:

Because no available solution was available at a decent price or offering all wished features, decision were made to create an in-house solution as an open source project based on Lucene. Emmanuel Keller, then the chief information officer, led the projects, and after two years of work more than 12 applications were installed and providing high value results. In December 2009, Emmanuel purchased the rights of the solution and formed a company to develop the community and offer them high level professional services, support, community management. It was the start of the story.

Today OpenSearchServer is one of a number of firms using Lucene as a component of a commercial enterprise search solution. One of the value adds his engineering team has crafted is a Classifier. He told me:

One a major module our Enterprise customers appreciate is called Classifier. It brings a very innovative set of features for applications with automatic classifications, matching and that are very appreciated in many businesses. Offering this module helps us to bring a nice differentiation for customers. Also we offer log reporting tools and a SOAP Web service.

The firm has a number of clients, including a rich media firm, an investment firm, and the vehicle information vendor ETAI, an Infopro group company.

You can read the full text of the exclusive interview in the ArnoldIT.com enterprise search service, Search Wizards Speak. Search Wizards Speak is the largest collection of first person narratives about search, content processing, and analytics available without charge. There are more than 50 interviews available in the series.

For more information about OpenSearchServer, navigate to the company’s Web site.

Stephen E Arnold, October 31, 2011

Sponsored by Pandia.com

Dr. Ric Upton will be leading Digital Reasoning’s Washington, DC area office and team in this exciting time for the company. Their product Synthesys is exactly what analysts require in this era of ever-amassing data.

While many other firms offering intelligence software focus on an aspect of entity extraction, Synthesys provides analysts with a comprehensive package for automating the interpretation of big data when the work of search and content processing systems has been undone.

In an exclusive  Arnoldit.com interview, Upton revealed how Digital Reasoning deals with such high volumes of real time information. He said:

[O]ur processing and analytics often have to complement these high volume data flows. We do this in part through judicious use of cloud-based processing augmented by intelligent methods of processing and storing data as it becomes available so that we can avoid the need to perform batch processing or redundant processing of previously-captured data.

The real value is their focus on content centric analytics instead of using statistical algorithms to analyze structured data. Essentially, they decipher the subtext and implicit meanings of content that doesn’t have to be well-structured. The real feat in this is that Digital Reasoning can automate this analysis without any data preparation.

Without Digital Reasoning’s systematic interpretation of data, analysts and clients would actually have to spend hours upon hours of time reading and comprehending content.

Upton shared the reasons why clients have typically used their software:

Our ability to automate understanding is critical to customers with concerns about time, accuracy, completeness, or even the ability to leverage the massive amount of data they have generated.

Serving as an intermediary between the raw data and analysts in the business process, this software has the capabilities to understand the subtleties of the human language. Synthesys can understand the underlying messages in the context of the content’s medium—whether it is a blog, a tweet, or an SMS.

In the interview, Upton sheds insight into how this rich entity extraction manifests itself:

We don’t just extract a name, we can develop and create a persona – the sum of what a person is called, where they have been and when, their relationships with other persona, their behaviors over time, etc.

Digital Reasoning is already looking towards the future, which forecasts that other media such as video and audio sources hold clout as data. As they work on developing methods to analyze these structures, competitors’ opportunities to dominate this field dwindle away.

Megan Feil, October 5, 2011

Sponsored by Pandia.com

John Steinhauer, vice president of technology, Search Technologies

John Steinhauer

On August 8, I spoke with John Steinhauer,  vice president of technology of Search Technologies. Before joining Search Technologies, Mr. Steinhauer was the director of product management at Convera. He attended Boston University and the University of Chicago. At Search Technologies, Mr. Steinhauer is Responsible for the day-to-day direction of all technical and customer delivery operations. He manages a growing team of more than 75 engineers and project managers. Mr. Steinhauer is one of the most experienced project directors in the enterprise search space, having been involved with hundreds of sophisticated search implementations for commercial and government clients. The full text of the interview appears below.

What’s your role at Search Technologies?

Search Technologies is an IT services provider focused on search engines. Working with search engines is essentially all we do. We’re technology independent and work with most of the leading vendors, and with open source. The things we do with search engines covers a broad spectrum – from helping companies in need of some expert resources to deliver a project on time, to fully inclusive development projects where we analyze, architect, develop and implement a new search-based solution for a customer, and then provide a fully managed service to administer and maintain the application. If required, we can also host it for the customer, at one of our hosting facilities or in the cloud.

My title is VP, Technology and I am one of the three original founders of the company and have been in the search engine business full-time since 1997. I am responsible for the technical organization, comprised of 70+ people, including Professional Services, Engineering, and Technical Support.

From your point of view, what do customers value most about your services?

We bring hard-won experience to customer projects and a deep knowledge of what works and where the difficult issues lie. Our partners, the major search vendors, sometimes find it difficult to be pragmatic, even where they have their own implementation departments, because their primary focus is their software licensing business. That’s not a criticism. As with most enterprise software sectors, license fees pay for all of the valuable research & development that the vendors put in to keep the industry moving forward. But it does mean that in a typical services engagement, less emphasis is put on the need for implementation planning, and ongoing processes to maintain and fine-tune the search application. We focus only on those elements, and this benefits both customers, who get more from their investment, and search engine partners who end up with happier customers.

In your role as VP of Technology, what achievements are you most proud of?

I’m proud that we have built a company with happy customers, happy employees, and good profits. I’m also proud that we’ve delivered some massively complex projects on time and on budget, even after others have tried and failed. It is gratifying that we have ongoing, multi-year relationships with household names such as the US Government Printing Office, Library of Congress, Comcast, the BBC, and Yellowpages.com.

But our primary achievement is probably the level of expertise of our personnel, along with the methodologies and best practices they use that are now embedded into our company culture. When we engage with customers, we bring experience and proven methodologies with us. That mitigates risks and saves money for customers.

Do you recommend search engines to customers?

Occasionally, but only after conducting what we call an “Assessment”. We start from first principles and understand the customer’s circumstances; business needs, data sets, user requirements, infrastructure, existing licensing arrangements, etc. Based on a full knowledge of those issues, we offer independent advice and product recommendations including, where appropriate, open source alternatives.

So you also work with customers who have already chosen a search engine?

This is our primary business. Often, our initial engagement with a customer is to solve a problem; they’ve acquired a software license, spent significant time and money on implementation and are having technical problems and/or trouble meeting their deadlines and budgets. Problems include poor relevancy, performance and scaling issues, security issues, data complexity issues, etc. Probably 70% of our customers first engaged with us by asking us to look at a narrow problem and solve it. Once they discover what we can do and how cost effective we are, they typically expand the scope into implementation of the full solution. We help people to implement best practices to reduce complexity and ownership cost, while dramatically improving the quality of the search service.

So, what’s your secret sauce?

With search projects, usually the secret sauce is that there is no secret sauce. Success is down to hard work and execution at the detail level.

What makes Search Technologies unique?

Sure. If there is any secret to building great search applications, it is usually in showing greater respect for the data and how best to process and enhance it to enable sophisticated search features to work effectively through the front end. That and just experience from hundreds of search application development projects. When a customer hires a Search Technologies Engineer to participate in their project, they are not just getting a well-trained, hard working and hugely experienced individual who writes good code, they are getting access to 80+ technical colleagues in the background with more than 40,000 person-days experience on search projects. We’re great at sharing experiences and best practices – we’ve worked hard at that since the beginning. Also, our staff turnover is really low. People who like working with search engines like it here, and they tend to stick around. That huge body of experience is our differentiation.

So you’re pure services, no software of your own?

In customer engagements we’re pure services. That’s our business. But as a company of largely technical people, of course we’ve developed software along the way. But we do so for the purposes of making our implementation services more efficient, and our support and maintenance services more reliable and sustainable.

Where is the search engine industry heading?

There are now two 800 pound Gorillas in the market, called Microsoft and Google. That’s a big difference from the somewhat fractious market that existed for 10 years ago. That will certainly make it harder for smaller vendors to find oxygen. But at the same time, these very large companies have their own agendas for what features and platforms matter for them and their customers. They will not attempt to be all things to all prospective customers in the same way that smaller hungrier vendors have. In theory this should leave gaps for either products or services companies to fill where specific and relatively sophisticated capabilities are required. We see those requirements all over the place.

Open source (primarily SOLR/Lucene) is making major inroads too. We are seeing a lot of large companies move in this direction.

So is innovation dead?

Not at all. Actually we see lots of companies doing really cool and innovative things with search. Many people have been operating on the assumption that search software would reach a sort of commodity state. Analysts have predicted this for years, that once all the hard problems had been solved, then all search engines would have equivalent capabilities and compete on price. What we’re seeing is very different from that. People are realizing that these problems can’t just be solved and then packaged into an off the shelf solution.

Instead the software vendors are putting a ring fence around the core search functionality and then letting integrators and smart customers go from there. With search, there are now some firmly established basics: Platforms need good indexing pipelines, relevancy algorithms that can be tweaked to suit the audience, navigation options based on metadata, readable, insightful results summaries. But that’s just the starting point for great search.

Here’s an example we’ve been involved with recently. Auto-completion functions have been around for years. You start the search clue, the system suggests what you’re looking for, to help you complete it more quickly. We’ve recently implemented some innovative new ways of doing this, working with a customer who has a specific business need. This includes relevancy ranking and tweaking of auto-completions suggestions, and the inclusion of industry jargon. Influencing search behavior in this way not only helps the customer to provide a very efficient search service, it also supports business goals by promoting particular products and services in context. Think of it as a form or relevancy tuning, but applicable to the search clue and not just the results. These are small tweaks that can have a big impact on the customer’s bottom line.

Another big innovation is SaaS models for search applications. This has also been talked about for years, but is really just now coming into focus in practical ways that customers can leverage.

I understand that your business is growing. Where are you heading and what might Search Technologies look like in a couple of years?

Perhaps the most pleasing thing of all for me personally, is that a lot of our growth, which is averaging 20%+ year on year, comes from perpetuating existing relationships with customers. This speaks well for customer satisfaction levels. We’ve just renewed our Microsoft GOLD partner status, and as a part of that, we conduct a customer satisfaction survey and share the results with Microsoft. The returns this year have been really great. So one of the places we are heading is to build ever longer, deeper relationships with companies for who search is a critical application. We initially engaged with all of our largest customers by providing a few consultant-days of search expertise and implementation services. Today, we provide these same customers with turnkey design and implementation, hosting services, and “hands-off” managed services where all the customer does is use the search application and focus on their core business. This model works really well. Through our experience and focus on search we can run search systems very efficiently and provide a consistently excellent search experience to the customer’s user community. In the future we’ll do a lot more of this.

Finally, tell me something about yourself

I grew up in Michigan, have lived in Chicago, Boston, DC, London and now in San Diego. The best thing about that is I can ride my bike to work most mornings year round. I have two boys (4 years old and 6 months old), neither of whom have the slightest clue what a Michigan winter entails. I expect that will continue for the foreseeable future.

Don C Anderson, August 29, 2011

Sponsored by Search Technologies