IBM has been a strong consumer and integrator of open source search solutions. Watson, the game show winner, used Lucene with IBM wrapper software to keep the folks in Jeopardy post production on their toes.

A screen shot of the Vivisimo Velocity system displaying search results for the RAND organization. Notice the folders in the left hand panel. The interface reveals Vivisimo’s roots in traditional search and retrieval. The federating function operates behind the scenes. The newest versions of Velocity permit a user to annotate a search hit so the system will boost it in subsequent queries if the comment is positive. A negative rating on a result suppresses that result.

I learned that IBM allegedly purchased Vivisimo, a company which I have covered in my various monographs about search and content processing. Forbes ran a story which was at odds with my understanding of what the Vivisimo technology actually does. Here’s the Forbes’ title: “IBM To Buy Vivisimo; Expands Bet On Big Data Analytics.” Notice the phrase “big data analytics.”

Why do I point out the “big data” buzzword? The reasons include:

• Vivisimo has a clustering method which takes search results and groups them, placing similar results identified by the method in “folders”
• Vivisimo has a federating method which, like Bright Planet’s and Deep Web Technologies’, takes a user’s query and sends the query to two or more indexing systems, retrieves the results, and displays them to the user
• Vivisimo has a clever de-duplication method which makes the results list present one item. This is important when one encounters a news story which appears on multiple Web sites.

According to the write up in Forbes, a “real” news outfit:

IBM this morning said it has agreed to acquire Vivisimo, a Pittsburgh-based provider of big data access and analysis tools.

Okay, but in Beyond Search we have documented that Vivisimo followed this trajectory in its sales and marketing efforts since the company opened for business in 2000. In fact, the Wikipedia write up about Vivisimo says this:

Vivisimo is a privately held enterprise search software company in Pittsburgh that develops and sells software products to improve search on the web and in enterprises. The focus of Vivisimo’s research thus far has been the concept of clustering search results based on topic: for example, dividing the results of a search for “cell” into groups like “biology,” “battery,” and “prison.” This process allows users to intuitively narrow their search results to a particular category or browse through related fields of information, and seeks to avoid the “overload” problem of sorting through too many results.

Recently Vivisimo was pushing the concept of “information optimization.” I am not sure what this phrase means. Perhaps the phrase is a synonym for “big data.” Seems a stretch to me. Vivisimo also, like MarkLogic, morphed into an enterprise search vendor and then, like Coveo, changed instantly into a vendor of customer support systems. Along the way, Vivisimo presented itself to some US government agencies as a systems integrator. A couple of the large scale deployments of the Vivisimo system lit up my radar as darned exciting. Sorry, no details in a free blog post.

Now let’s think about what IBM may be doing with a clustering and federating system? I think IBM could fix up the lousy clustering functions in products such as FileNet. I think IBM could use Vivisimo’s technology to improve the search function on the IBM.com Web site. I think IBM could use Vivisimo technology to refine the outputs of Lucene search system which is deployed in a number of IBM products.

However, I do not get the “big data” angle. No, not at all. Search systems which make sense of flows of data from social media, intelligence monitoring systems, and various types of specialized unstructured information systems come from outfits like Digital Reasoning, Ikanow, Palantir, and a handful of other firms.

My view is that IBM is getting back into the walled garden business in an effort to sell specialized functions to customers who are not happy with IBM’s existing line up of solutions. It is possible that IBM can bolt Vivisimo to Cognos, but SPSS already contains some text processing capabilities. Anyone remember Clementine? The i2 Group’s technology is not really “big data.”

I have someone monitoring news of this deal. If anything substantive turns up, we will update this post. For now, the IBM Vivisimo deal seems to be about “big data.” If true, the assertion is one that does not match what I know about Vivisimo’s systems and methods. We are often wide of the mark here in Harrod’s Creek, but I am struggling to see how Vivisimo contributes in the same way that Digital Reasoning or even DataStax does in the search sector.

The BMW dealers in Pittsburgh will be happy. Hopefully one of the founders will be relieved of manual script editing duties now that IBM is the boss.

Stephen E Arnold, April 25, 2012

Sponsored by PolySpot

“Access control to the data is a big difference between Enterprise search and the other 2 search types.  On the Web, everybody is allowed to see the data. On your desktop you are allowed to see all data, because you are the owner. Web and desktop search can index all the data without to take access control into account.”

In an enterprise, access control is very important. But we prefer to spend more time finding than searching. To get the results you want, you need the right solution and the right search structure and support.

Access control is not an obstacle for Mindbreeze. Their search technology maintains user rights while searching all company-relevant information within the enterprise and in the cloud.

Sara Wood, November 29, 2011

The defense and intelligence communities recognize the need to improve information sharing as a way to achieve true all-source analysis and deliver timely, objective, and actionable intelligence to our senior decision makers and war fighters,’ says Bob Carter, vice president and general manager, federal, of Vivisimo. ‘In an era where spending cuts are being made to improve efficiencies, Vivisimo helps streamline operations and ultimately costs by allowing analysts significantly better access, processing and sharing of critical data necessary to the defense of the U.S.

Assembling the myriad of data gathered from around the globe into useful information is one of today’s biggest challenges for the intelligence community. Though the government often travels behind the curve in tech fields, it seems to be stepping up in this area.

Cynthia Murrell May 12, 2011

Now X1 Technologies has appointed a new leader for their eDiscovery division. X1 Technologies Appoints John Patzakis as President of eDiscovery, citing his extensive background in eDiscovery and corporate compliance as well as his knowledge of the law.

“I am pleased to welcome someone as accomplished as John to the X1 team,” said John Waller, CEO of X1 Technologies. “John’s background as a senior software executive coupled with his deep understanding of compliance and discovery law make him a perfect fit to lead our efforts in the eDiscovery market.”

X1’s eDiscovery Search Suite allows users to search data stored in over 500 different files types and applications. This allows for quick retrieval of electronically stored information (ESI) for early case assessment. X1’s support of social media applications will be released this quarter. In Patzakis, X1 has found a leader with the experience and skill to push them forward in the eDiscovery sector.

Emily Rae Aldridge, February 243, 2011

Judging from the torrent of posts from “experts”, the big guns of search are saying, “We told you so.” The trigger for this outburst of criticism is the New York Times’s write up about JC Penny. You can try this link, but I expect that it and its SEO crunchy headline will go dark shortly. (Yep, the NYT is in the SEO game too.)

Everyone from AOL news to blog-o-rama wizards are reviling Google for not figuring out how to stop folks from gaming the system. Sigh.

I am not sure how many years ago I wrote the “search sucks” article for Searcher Magazine. My position was clear long before the JC Penny affair and the slowly growing awareness that search is anything BUT objective.

Source: http://www.brianjamesnyc.com/blog/?p=157

In the good old days, database bias was set forth in the editorial policies for online files. You could disagree with what we selected for ABI/INFORM, but we made an effort to explain what we selected, why we selected certain items for the file, and how the decision affected assignment of index terms and classification codes. The point was that we were explaining the mechanism for making a database which we hoped would be useful. We were successful, and we tried to avoid the silliness of claiming comprehensive coverage. We had an editorial policy, and we shaped our work to that policy. Most people in 1980 did not know much about online. I am willing to risk this statement: I don’t think too many people in 2011 know about online and Web indexing. In the absence of knowledge, some remarkable actions occur.

You don’t know what you don’t know or the unknown unknowns. Source: http://dealbreaker.com/donald-rumsfeld/

Flash forward to the Web. Most users assume incorrectly that a search engine is objective. Baloney. Just as we set an editorial policy for ABI/INFORM each crawler and content processing system has similar decisions beneath it.

The difference is that at ABI/INFORM we explained our bias. The modern Web and enterprise search engines don’t. If a system tries to explain what it does, most of the failed Web masters, English majors working as consultants, and unemployed lawyers turned search experts just don’t care.

Search and content processing are complicated businesses, and the appetite for the gory details about certain issues are of zero interest to most professionals. Here’s a quick list of “decisions” that must be made for a basic search engine:

• How deep will we crawl? Most engines set a limit. No one, not even Google, has the time or money to follow every link.
• How frequently will we update? Most search engines have to allocate resources in order to get a reasonable index refresh. Sites that get zero traffic don’t get updated too often. Sites that are sprawling and deep may get three of four levels of indexing. The rest? Forget it.
• What will we index? Most people perceive the various Web search systems as indexing the entire Web. Baloney. Bing.com makes decisions about what to index and when, and I find that it favors certain verticals and trendy topics. Google does a bit better, but there are bluebirds, canaries, and sparrows. Bluebirds get indexed thoroughly and frequently. See Google News for an example. For Google’s Uncle Sam, a different schedule applies. In between, there are lots of sites and lots of factors at play, not the least of which is money.
• What is on the stop list? Yep, a list can kill index pointers, making the site invisible.
• When will we revisit a site with slow response time?
• What actions do we take when a site is owned by a key stakeholder?

Is it possible to spoof Google? Sure. The JC Penny example is a good one, but I find examples of Google’s bumbling every day. I get auto generated pages of baloney. I get links to 404 errors at the Health & Human Services Web site. I find examples of content in the index and not in the cache. I find in most results lists totally useless links.

Run a query for “information optimization”. What do you get for this meaningless phrase? You get a link to Vivisimo which uses “information optimization” instead of “search done right”, its 2007 catchphrase. You get links to Hewlett Packard, a blog about information optimization, and baloney about search engine optimization. The problem is that the phrase is essentially meaningless. I think it has been crafted to make it easier to locate outfits in the fuzz business.

Google falls for this joyfully. Google even lists the mind bogglingly expensive Google Search Appliance as associated with “information optimization” via another meaningless phrase, “knowledge management.”

The fact of the matter is that as the Web content diffuses and becomes more voluminous, the opportunities to play tricks increases. And the Web search engines continue to make their decisions behind the scenes. Google is proud of the fact that it keeps its method secret. In The Google Legacy, I summarized about 100 factors in use in 2004 and 2005. Each “factor” is a form of editorial policy.

The content of indexes is never objective. Whether one looks at a commercial database or a Web index, decisions inform the scope, depth, and approach of what’s available to a user.

SEO or search engine optimization is just a variant of indexing. I personally find SEO in general and SEO experts in particular annoying at best. I prefer that Web sites have content about a subject. That content can be casual like the information in this Beyond Search blog. It can be weaponized like the information on some government and political sites. It can be wacky like the humor sites. SEO injects links and words that are designed to fool the Web indexing systems.

So what we have is bias in the indexing systems. We have bias in the content. We have bias in the tagging and linking.

So now everyone is horrified that free Web search systems are not “objective”.

Give me a break.

The whole search sector is not objective. The algorithms execute but no one knows the weights, thresholds, or tweaks under the hood. Believe me. There is a lot of fiddling that must be done. In the first index of the US Federal government using the Inktomi system, considerable time was spent removing certain content from the index. No one paid much attention in Year 2000, and I don’t too many people pay much attention to the contents and content scrubbing in most indexes. Where did that education policy go on the company Intranet site? Answer: the technical team was told to delete the pointers. Good bye information. Few notice. Filtering, cleaning, and scrubbing are routine in each of the component of a search system. One vice president wanted his Web site’s content updated in near real time. No problem. The Railway Retirement Board? Well, every six months will probably do the trick.

Two different users of the same search system can derive different results via their own search methods. The “smart” systems can display different search results for different users. Web content creators can manipulate the Web indexing systems. Heck, the Web indexing systems can manipulate the results to their benefit or the benefit of their bottom line.

There is no free lunch for those who don’t know what they don’t know.

Fact: There are no objective search results. Just my view from Harrod’s Creek. Don’t believe me. Track down a person with a master’s degree in library science and ask that individual to run sample queries for you and then analyze the results.

Run the same queries on commercial systems and on free Web search system. No system have content congruence. No relevance method delivers exactly the same results unless a human intervenes. To get information, one must use multiple search systems, multiple sources, and talk to humans. The notion of “getting an answer” is popular. The problem is that the answer may be wrong, biased, or an ad. Most Web users are happy with whatever pops up. Intellectual laziness? Too much work? Convenience? Who knows.

The reason there is a degree in information science is a reaction to a need to understand provenance, precision, recall, editorial policies, and indexing.

English majors, lawyers, lousy Web masters, and money crazed SEO experts are different in many interesting ways. Their view of information often contrasts sharply with that of a person with deep experience in library and information science.

Web search has to generate revenue and only be “good enough.” Forget the superlatives and deal with the bias inherent in search, content processing, and indexing.

Stephen E Arnold, January 13, 2011

Freebie and not lunch

If you are interested in visual search, you can check out Yometa.com. This is a federating search system which taps results from Bing, Google, and Yahoo. A query for “Stephen E Arnold” returned this display.

http://www.yometa.com

Yometa displays the most relevant search results based on a combination of the three search engines ranking determined by the Yometa algorithm.

The company developed its approach based on research that showed that 97 percent of search results by the three search engines(Google, Yahoo and Bing) are different and there is only three percent overlap. The visual interface allows users to see results of Google, Bing and Yahoo individually and in various combinations. Users can see any combination of search results from Bing, Yahoo and Google in one screen and is displayed in a visual interface. The search results are displayed in a Venn Diagram, the results closer to the middle are more relevant.

For more information navigate to www.yometa.com/about/ .

Stephen E Arnold, February 7, 2011

Freebie

Let’s not get ahead of ourselves. A family friendly search was the focus of The Point (Top 5% of the Internet), developed by Beyond Search’s Stephen E. Arnold, his son, Erik S. Arnold, and business partner, Chris Kitze in 1993. the Point service sold to Lycos in 1996, and, alas,  Lycos lost its way. Now, a 17-yr-old idea is back, proving The Point was right on target almost two decades ago.

Brett Quinn, August 11, 2010

Freebie

Deep Web Technology provides the software and systems that fuel Science.gov, a US government search and retrieval project. Science.gov taps into a wide range of data and information related to science and technology. The invention of the Deep Web method was an outgrowth of Dr. Lederman’s experience in providing a user with access to a broad range of structured and unstructured data. In my various reports on enterprise and special purpose search, I have given Dr. Lederman’s method high marks, and I even let him buy me a taco in a restaurant in Santa Fe, after I finished a lecture at Los Alamos. Dr. Lederman contributed at Los Alamos prior to founding Deep Web as I recall.

The deal brings Dr. Lederman’s federation technology to the SwetsWise Searcher. This service will be powered by Deep Web Technology. SwetsWise is designed to help librarians and their users meet the challenge of searching and finding relevant results from the ever-increasing catalog of content available online. The search system simplifies access to an organization’s diverse and valuable resources, along with the open Web content users are accustomed to searching. SWETS will deliver search results through the Deep Web ranking engine, providing incremental results for fast response times, scalability and flexibility. SwetsWise Searcher performs a rapid parallel search of all available sources or selected sources in real-time, ensuring fresh information and that documents are retrieved the minute they are published into a collection’s database. A simple search box to cover all sources can be integrated into any web page, blog or Intranet homepage.

A happy quack to Deep Web Technology. No more tacos in Santa Fe. I want a nuked burrito, a nod to our friends up the road.

Stephen E Arnold, February 14, 2010

No one paid me to write this. I do have a promise of a taco in Santa Fe, which I have just rejected. I will report this to the Food & Drug Administration.

Other new features include:

• E-reputation tools
• Metasearch functions
• Support for anonymous search
• Support for French, English and Dutch language.

If you have not explored the Kartoo service, give it a whirl.

Stephen Arnold, October 13, 2009, published because I like the French

Jessica Bratcher, October 8, 2009