Calais: Free Semantic Tagger
April 22, 2008
If you want to see how cloud-based software can perform rich metatagging, you will want to give the free Calais service a whirl. Navigate to the Calais Gallery and scroll down to the Capability Demonstrations and select the Calais Document Viewer. If you don’t see the link, click here.
Now cut a document and paste it into the window. The system will display this type of result:
The tags the ClearForest system automatically identifies are highlighted. The left-hand column of the display shows the types of tags identified; for example, city, company, person, etc. A single click opens a drop down list of what the system found. Worked well and it worked quickly with no “false drops” in the sample document. Performance showed some latency, but that’s not unusual with a cloud-based service and some fancy text crunching taking place on remote servers.
More about Calais
For now Calais is working to build a community to extend the Semantic Web. Without tools like Calais, the Semantic Web is likely to remain a great idea that failed because people don’t want to do tagging. When tagging is done, it’s lousy. I’m supposed to know how to index, and the tags for my Web log are pretty miserable. The reasons may be broader than just my own approach. First, indexing to be useful must use a body of terms that the average user can hit upon and remembered. So neologisms are out and weird jargon won’t work at all. Second, writing for a Web site or a Web log like this one is supposed to be disciplined, but it’s not. I have other research work that commands my primary attention. The Web log, while important, comes second, maybe third on some busy days. Finally, I’m not sure what I will write about. I react to information people send me in email, stories in my RSS reader, and comments made–often off the cuff–on a phone call. It is difficult for me to create a controlled term list because I’m not sure what the topics will be. Therefore, lousy tagging.
Calais asserts that its technologies can address my three failings and probably yours as well. You can download developer tools, upload content to Calais, or use the functions on the Calais Web site. Reuters-ClearForest has posted some useful documentation about Calais here. If your a bit nerdy, you can do some integration of Calais and your application. The best way to get a sense of what’s possible is to explore the sample applications on the Calais Web site.
More about ClearForest
ClearForest was founded by text mining guru Ronen Feldman. You can get the inside scoop on this wizard’s approach to squeezing information from text in his 2006 book, The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data (Cambridge University Press ISBN 13: 9780521836579).
The ClearForest technology performs “discovery”; that is, the system processes text and identifies important information. The company found a ready market wherever executives wanted to find the “hidden” information in text. I recall attending a presentation by Dr. Feldman in which he showed the ClearForest system processing auto warranty data in the written comments from customer support reps and owners who sent email about their vehicles.
The ClearForest system processed these comments and displayed important discoveries in easy-to-understand reports. One example concerned a flawed component that the ClearForest system pinpointed as one that was causing problems previously overlooked by the automobile manufacturer. The kicker to this example was that the manufacturer was able to make a change to the affected component and take pre emptive action to save significant amounts of warranty cost and avoid customer complaints.
The earlier versions of the ClearForest system made use of rules. Some of these required hand-tweaking or a ClearForest-adept programmer to set up, tune, and deploy. Over the years, ClearForest like other companies looking for ways to leverage “smart” software, more automation has been injected into the ClearForest system.
Text Processing Vendors as Shark Bait
April 22, 2008
I’m nervous around attorneys or, as the English say, solicitors. The Content Wrangler offers a useful discussion of why legal eagles are circling content processing and text mining vendors. Please, read the original post “Automated Intelligent Document Classification, Data Extraction and Search Tools for Legal Pros”.
The essay appears to be a contribution from A2iA. The tag paragraph for the essay says that A2iA “is the worldwide leading developer of natural handwriting recognition, Intelligent Word Recognition (IWR) and Intelligent Character Recognition (ICR) technologies and products for the payment, mail, document and forms processing markets”. Note: the company’s Web site plays music, so you may want to mute your speakers before navigating to the landing page.
The key point in the essay, from my point of view, is:
Any automated solution must be at least as accurate and error-free as the manual processes it replaces. As accuracy is equally important to successful discovery, due diligence, investigatory or redaction efforts, this is an important advantage of DocumentReaders [nb, A2iA’s product] automated solution.
A defendent facing a life sentence would probably agree that accuracy is sometimes helpful. On the other hand, some legal professionals may disagree under certain circumstances. Speed is also important. Overworked lawyers need ways to pack more billable hours into the work day.
Recommind, Stratify (now a unit of Iron Mountain), and other content processing vendors have found ways to tap into the rivers of money that flow through the mahogany corridors of major law firms. I’m open minded about A2iA. You will want to check out systems available from Brainware and ZyLAB, both companies who made the final 24 in the Beyond Search round up.
I did not include the company in Beyond Search, nor do I have an entry in my list of companies in the text and content processing business. I have added A2iA, however. If you want a useful run down of why counselors have a thirst for systems that can make sense of large quantities of text, tuck this discussion in your files.
My research suggests that legal matters generate so much information that even well-padded clients are reluctant to pay for real-live people to read, analyze, and annotate paper. Digital systems, therefore, are not the first choice, but the only choice due to quantity and costs.
Vendors peddling to law firms may want to check their shark cage. Flawed technology can bite back in interesting ways.
Stephen Arnold, April 22, 2008
Nstein: Joining Other Search Vendors in the Infomercial Parade
April 21, 2008
Nstein is a company that “delivers a complete lifecycle of solutions tailored for content-driven organizations to define and build a unique online brand experience that leverages assets, increases online revenues and meets customer expectations.” Now it is in the for-fee conference business.
Nstein has joined the parade of vendors running infomercial conferences. The idea is that a vendor hosts a conference, arranges for appropriate speakers, and runs a two- or three-day promotion. This type of infomercial makes sense for vendors for two very good reasons. The money once allocated to independent conference organizers for a booth and maybe a slot on a crowded program gets focused on one idea–the informercial owners’ product.
The second reason is perhaps even more pragmatic–control. An independent conference organizer can put on a panel a critic of a technology. There may be a rare annoyed customer. Even worse, a loud mouth in the audience can ask a difficult question and then make a vendor look like a contestant on “Are You Smarter than a Third Grader” who doesn’t know the answer.
In the good old days, vendors of search systems had user groups. A vendor would invite users to a location. The vendor would talk, and the users would trade ideas, some code snippets, and voice their concerns to management. Well, let me tell you that these became less folksy when the founder-owners of the companies were replaced by smart lawyers and with-it MBAs. The user groups are nowhere to be found in enterprise search and related disciplines. You can still locate a user group in the library world, and one company–Cuadra Associates where owner-founder-wizard Carlos Cuadra–is still running the company.
We are now in the midst of a boomlet of controlled-message conferences. There are many examples; for instance:
The Oracle conferences do double duty. These explain the Oracle way and provide a showcase for specific Oracle products. The Oracle SES10g–Oracle’s search and content processing system loaded with security options–has not had the PR muscle this year. The Oracle shows are combinations of technical evangelism for products like SES10g which has a low profile in some sectors, unabashed sales efforts, and Oracle fun.
Microsoft’s shows are good, but the attendees are sloshing with the particular Kool-Aid distributed by each Microsoft event. At the last Microsoft event I attended, I had to buy an extra duffle to cart home the free goodies, the hard copy documents, baseball caps, and T shirts (pretty good quality at that).
For search and retrieval, Fast Search & Transfer has its FastForward Conference, a Web log, and a stream of publicity that imprints the name of the company and hustles up attendees. This year’s bash attracted more than 1,500 attendees, a number of semi-objective speakers, and a handful of investor types who sniffed for a deal.
Endeca has announced that it will host its search conference this year “to encourage openness, collaboration, innovation among Endeca Developers, Partners, and Owners”. You can read about the 2007 conference here. I wasn’t able to come up with a link to the 2008 program, but I know it’s on the Endeca Web site. Glitch in Google, I suppose.
Not surprisingly, Nstein–a company that has repositioned itself from metatagging system to a broader media / marketing / content management / search solution–in in the game. The conference is “Innovation Leaders Summit 2008”. I am certain you will want to attend and learn about innovation from May 14 to 16, 2008. You can read more here and sign up. The cost is $1,195. (Turn down your speakers, this landing page plays sound. I assume that’s an innovation. There’s also a video on the page, another innovation.)
The agenda includes a track to help you chart a Google strategy. Judging from the program, it appears that there is a wide sweep to the program. Topics span search engine optimization (SEO), federated search, and workflow solutions are on tap. There’s a strong Canadian flavor to the preliminary list of speakers. And, there are Nstein executives on the program as well. When I visited the site (briefly due to the music playing), I saw mostly Nstein executives. The implication is that Nstein has mastered these diverse topics. Most search and content processing senior managers are competent in the any and all topics related to search and content processing, so relying on the firm’s executives makes sense in the sponsored-conference setting.
One plus is that conference will be held in “old” Montréal, a venue which has some wonderful restaurants and a number of interesting shops.
There is a downside to these infomercial conferences, and it’s one that is now having impacts that are not discussed in the professional journals or on specialist Web logs.
First, the messages at these conferences are shaped and channeled. The rough edges of an independent conference include talks that may be critical of a technology or a particular trend. A vendor-owned and vendor-operated conferences reduces the likelihood that the potentially disruptive questioning of a vendor once characteristic of traditional user groups is reduced. Unhappy customers and competitors don’t get to the podium.
Another consequence of vendor-owned conferences is that these are slowly sucking attendees from broader search-related conferences. I don’t want to identify any of these events, but attendance at some search-centric events is beginning to erode. I’m not talking about the SEO conferences. These are thriving because a Web site that’s not “in Google” doesn’t exist for all practical purposes.
The conferences are the more generalized looks at search and content processing. The vendor-sponsored shows are more uptown, and, therefore, pull attendees from the more traditional venues. If traditional conference organizers can’t revivify their offerings, we may lose some important oppotunities to hear objective albeit uneven presentations about search and content processing. Imagine watching TV and seeing only infomercials. As terrible as TV is, a diet of infomercials would present an odd view of reality. No presidential election but quite a few epoxy putty and thigh slimming messages.
The killer, of course, is money. When vendors run their own shindig, the conference companies take a financial hit. Fewer attendees and fewer exhibitors could mean the end of some useful, specialist programs.
I’m still a fan of the old-fashioned, let-the-customer-speak user groups. That shows how out of touch I am. I also like the objective shows. Speakers at these shows aren’t operating within guidelines the vendor lays down.
Stephen Arnold, April 22, 2008
Search Vendors: Sniper Firefights Break Out
April 21, 2008
Yesterday, I posted an email containing a statement by a publicly-traded company’s intent to replace its incumbent search system. You can read the full text of this document, which I have verified as originating with a reputable company on the West Coast of the US. I deleted the references to the vendor whose search system is getting the boot. I also redacted the name of the company reaching its boiling point and the name of the hapless information technology manager who was responsible for the acquisition of the incumbent system.
Just three days before I received this email from the aggrieved licensee of a blue-chip search system, I spoke on the telephone with a leading European investment bank’s lead analyst for a publicly-traded company with a presence in search and retrieval. That call probed my knowledge of customers using the publicly-traded firm’s search system. I don’t have too much detail (what the analysts whiz kids call “color”) about expensive systems that are a pain in the neck. What I know I keep to myself. I was interested in why the zingy young MBA was calling me in my underground bunker in Harrod’s Creek, Kentucky. The reason was that the investment bank had heard that some high profile licensees of this publicly-traded company’s search system were going bonkers over costs, erratic search results, and performance. This is a hat trick of sorts, and I slithered out of the call.
Today I spoke with my partner in Washington, DC. She told me, “I have worked with most of the big guys. None of this stuff works without work–a lot of work. So what’s new?”
I guess not much when it comes to enterprise search (what I call Intranet search or behind-the-firewall search). What is new is that the public airing of complaints seems to be ratcheting upwards. A few years ago, an organization would assume that cost overruns, grousing users, and system flakiness was a problem anywhere except the search vendor.
Not today. Some licensees are savvier, and several licensees are not too shy about telling the world, “Hey, this stuff is a major problem. It doesn’t work as advertised.”
Contravellation: An Old Strategy Might Resurface
I am not sure if you are familiar with the word contravellation. Popular among the war college set, the term refers to a fortification set up to protect a besieging force from attack by the defenders of the besieged place. In short, a contravellation is a defensive shield designed to protect one party from another. The Romans apparently had an appetite for contravellations, using them keep their enemies from slipping out of a besieged town. The besieged wall themselves in. A contravellation makes sure no one gets away.
A Civil War Contravellation. It would fence me in.
My hunch is that we are about to see a number of defensive fortifications erected by search vendors to prevent licensees from escaping. Let me be clear. This is not a problem of one vendor. This is a problem created by many different vendors. This is not a problem that appeared overnight. The pot has been boiling for years, a decade or more in some cases.
The worsening economy makes an expensive search system more than a casual expense. Users and some information technology professionals have a much deeper understanding of what a search system can do. Some vendors make an attempt to “lock in” a licensee with a “perpetual license”, deferred payment schedule, bundles of maintenance and support available only with a multi-year deal, and other techniques. Others assume that an investment in a “platform” cannot be easily discarded. The sheer scope and complexity of an information processing system puts a licensee under a state of siege.
The vendors’ contravellation will be designed to prevent a licensee from breaking an agreement.
Interview with Sinequa’s Reveals Wins from Autonomy and Google
April 21, 2008
In an interview published on April 21, 2008, on the Web log “Beyond Search”, Jean Ferré, the managing director of Paris-based Sinequa, reveals that his firm has won two contracts from search giants Autonomy and Google. The company has a solid following in Europe, including deals with the French media company Le Monde and Radio France International. The full text is on ArnoldIT.com’s Web site. http://www.arnoldit.com/search-wizards-speak/sinequa.html
Sinequa’s offers technology that allows licensees to “connect to knowledge”, not just key word search and retrieval. The system includes “a unique combination of statistical, structured, linguistic, and semantic indexes,” says Mr. Ferré.
The company’s solution, according to Mr. Ferré, “falls in between a “search toaster” and a box of technical parts you assemble. We resolve the complexity of exhaustive secured connectivity, profile based interface and yet best in class relevancy but delivers much faster at a much lower cost and complexity.”
The complete interview is available on the ArnoldIT.com Web site’s “Search Wizards Speak” service. More information is available at http://www.sinequa.com.
Stephen Arnold, April 21, 2008
Sinequa’s Jean Ferré Interviewed
April 21, 2008
Sinequa, based in Paris, provides search and content processing systems that straddle traditional search, business intelligent, and data management. The company has a strong customer base, primarily outside the United States. I reacquainted myself with the company at the International Online Meeting in London, England, in December 2007. Curious about the new features in the system, I was successful in getting the firm’s managing director to speak with me.
The positioning of the company is different from some search vendor’s approach. Mr. Ferré said:
We are a search-and-retrieval system focused on the enterprise promoting our “Connect to Knowledge™” approach. What’s different is that our technology is a self-contained packaged delivered in two formats: First, we offer a flagship solution called Sinequa CS. I’m delighted to say that our sales doubled in 2007. Sinequa CS consists of a full fledged packaged platform including connectivity, navigation and obviously the core engine deployed in a large number of enterprises such as Bouygues, Arkema, MBDA, the French Army, EADS, Eurocopter, LCF Rothschild, the French Police, etc. Second, we have what we call the OEM offer (original equipment manufacture license). Another software company licenses our technology an uses it in their enterprise system. Some OEMs embed our technology in enterprise applications, Web sites, or inside Intranets.
The complexity of search systems has been the subject of some discussion. Mr. Ferré told Beyond Search:
I think Sinequa falls in between a “search toaster” and a box of technical parts you assemble. We resolve the complexity of exhaustive secured connectivity, profile based interface and yet best in class relevancy but delivers much faster at a much lower cost and complexity…. We are now offering a turnkey deployment for enterprise content. If the client wants to search and process information in file systems, relational databases, Microsoft SharePoint, the Web crawling, RSS and enterprise content management–no problem. We can have the company up and running in four days. As an example; we recently were chosen in replacement of Autonomy by one of the largest global IT integrator for its worldwide internal search. We had to compete with what the IT director wanted–Google. We won this important contract…
You can read the complete interview on the ArnoldIT.com Web site. This interview is part of the exclusive series “Search Wizards Speak”, which allows you to learn first hand about some of the most interesting companies in the behind-the-firewall (enterprise search or Intranet search) market.
Stephen Arnold, April 21, 2008
Search Frustration: A Licensee on the Offensive
April 21, 2008
I received an email from a financial wizard on the West Coast on Friday. Here’s what I received, and I apologize for removing the names of the sender, the company, and the vendor. My attorney suggested that he needed a new BMW when I asked him about revealing details. I wisely concluded that removing the track back information would reduce my legal fees. He doesn’t need a new car, and I don’t need certified mail from the bad boys referenced in this letter.
The document redacted is:
ANYCO, Inc is a publicly traded company (Nasdaq: XXXX) that helps businesses achieve a competitive advantage by delivering timely and actionable sales opportunities and information. ANYCO offers unparalleled coverage of government purchasing activity in addition to commercial and residential projects in development for markets such as architecture and engineering, IT/telecom, business consulting services, operations and maintenance, and transportation. We gather, process, categorize and deliver up to 5000 new opportunities per day, including all associated text records and documents (.doc, .pdf, .xls, etc.) to provide our clients with up to date lead and market information.
We are actively seeking vendors interested in receiving an RFP to replace our current implementation of INCUMBENT SEARCH SYSTEM. We are looking for a comprehensive, scalable, fault tolerant leading edge search solution for both internal and client facing use. We need to be able to index and search our entire database and document store, comprising over 10 years of government procurement information (almost 1 TB of data and documents), and are interested in advanced search features to enhance our customer experience and productivity.
The RFP will be distributed on April 28th, with an aggressive timeline for responses and vendor engagements. We intend to select a vendor and begin work on the project by May 26th.
If your company would be interested in participating in this RFP, please respond with a point-of-contact.
Thank you,
SYSTEM ADMINISTRATOR, WEST COAST, USA
The only background I have for this letter is that it was sent to vendors of enterprise (what I call behind-the-firewall or Intranet) search systems. The letter was verified by both recipients as legitimate and not a spoof. The search vendor whose system is being replaced is a high-profile vendor. After reading this a couple of times, I concluded, perhaps incorrectly because I have battle scars inflicted by search system vendors who threaten me if I blow the whistle on some of the more exciting implementations I am paid to fix:
- The person writing this letter is not a happy camper. In fact, the writer’s naming the vendor is evidence that relationships between this licensee and the vendor have reached their nadir. Mentioning the vendor could be construed as an actionable offense.
- The timelines make obvious that the situation is in fast-cycle remediation mode. The incumbent system is getting tossed out, and a new system is coming within 30 days. That’s not much time to short list, procure, and schedule a “rip and replace” solution.
- The technical issues appear to be the ability to handle a modest amount of text and functions beyond key word retrieval.
One email from one licensee is interesting. In the context of the research data I summarize in my new study for the Gilbane Group and the Sinequa research data, more color is accreting about dissatisfaction with search and retrieval. Have you had a less than positive experience? If so, send along the details. I will anonymize them and summarize each example. As examples accumulate, the survey data may be amplified by squawks from licensees who perceive themselves as having been given search sizzle, not search steak.
Stephen Arnold, April 21, 2008
Indexing Dynamic Databased Content
April 20, 2008
In the last week, there’s been considerable discussion of what is now called “deep Web” content. The idea is that some content requires the user to enter a query. The system processes the querey and generates a search result from a database. This function is easier to illustrate than explain in words.
Look at the screen shot below. I have navigated to Southwest Airlines Web page and entered a query for flights from Louisville, Kentucky, to Baltimore, Maryland.
Here’s what the system shows me:
If you do a search on Google, Live.com, or Yahoo, you won’t see the specific listing of flights shown below:
Xerox Factspotter: Thingfinder’s Second Cousin
April 20, 2008
A long time ago in a research park far, far away, Xerox PARC (Palo Alto Research Center) developed text processing systems. Xerox PARC spun out a bundle of this content processing technology as Inxight Software. For about a decade, Inxight chugged along, winning accolades from the spookeratti in Washington, DC’s intelligence community. Business Objects, a disrupter of the business intelligence space, bought Inxight Software. The deal rippled the fabric of the likes of SAS Institute, a company licening Inxight’s technology for its data mining systems. Then SAP bought business objects. Along the way, start ups like PowerSet used some of Xerox’s technology to build a whizzy search start up.
Amidst this slow flowing river of deals, Xerox is back. This time “the document company” has Factspotter. Now Factspotter, like most search and text processing systems, is not newly-sprung from Xerox’s idea hathchery in Grenoble, France. The research team at XRCE, an forgettable acronym for Xerox Research Centre Europe.
I learned about Factspotter in early 2007. I dug through my files and unearthed this description of the invention from the Xerox news release:
Unlike traditional enterprise search tools, FactSpotter looks not only for the keywords contained in a query but also the context of the document those words contain. For example, if searching for documents that reference Angelina Jolie, FactSpotter will also return results where the pronoun “she” is used instead of Jolie’s full name. The “smart” search engine can comb through almost any document regardless of the language, location, format or type; take advantage of the way humans think, speak and ask questions; and discriminate the results highlighting just a handful of relevant answers instead of returning thousands of unrelated responses.
I haven’t been tracking Xerox’s “inventions” or its document processing business until the IBM InfoPrint entity popped into being in 2007. Then in January 2008, Hewlett Packard paid $1.2 billion for the Exstream Software operation in Lexington, Kentucky. When these document processing developments took place, I wondered what had happened to Xerox, “the document company”. After that thought, Xerox drifted off my radar–until today, April 12.
Someone emailed me a snippet of text from IT Reseller. The key points, which I have edited for easier readability,are:
[Factspotter’s] novel interface means users can express their queries naturally instead of forcing them to adapt their questions to the logic of computers. Traditional systems, on the other hand, split a query into isolated words and return only documents that contain exactly those words in exactly that order, And [Factspotter] takes into account the context of the entire document instead of just a cluster of nearby words. And [Factspotter] introduces the concept of “relation,” searching within and across sentences and paragraphs. It recognizes abstract concepts, like “people” or “building,” and will retrieve all the words that fit within that category.
Xerox’s marketing mavens were dead on in 2007. The only issue is that I have is that it’s on the Xerox Web site, but not anywhere else. If you know the fate of Thingfinder’s second cousin, write me at seaky2000 @ yahoo.com.
Stephen Arnold, April 20, 2008
Fast Search: It’s Pretty Easy but Training Helps
April 19, 2008
A news release arrived in my mailbox with the headline, “FAST University Receives IACET Accreditation”. Please, read the full release here before it becomes difficult to find. (PR “news” can disappear with little warning.) The course originator is Fast Search & Transfer, the object of Microsoft’s $1.2 affection. Fast Search like Autonomy bills itself as a “leader” and “award winner” in enterprise search. There’s no accreditation program for search vendors, so any vendor can don the laurels of a leader.
The key point in the news release, from my point of view, is:
FAST University, the company’s award-winning education program, is now an International Association of Continuing Education and Training (IACET) Authorized Provider able to grant Continuing Education Units (CEUs) for customers, partners and employees who successfully complete FAST University instructor-led classes and self-paced eLearning.
I was not aware that Fast Search had a university. Other vendors of enterprise search systems (what I prefer to call behind-the-firewall search) should consider this type of program for three reasons:
- Licensees are unaware of the complexity of the building block solutions that some vendors offer. Without significant experience with these modular systems, training–regardless of type–is useful if only to help the licensee have a better sense of when to pay for additional professional support.
- Vendors have training programs, but these are in step with the times; that is, somewhat disorganized. At the recent Buying and Selling eContent conference, a panel of buyers told vendors that organized, professional training is sorely needed. No comments were made about online training, but that may be better than indifferent in-person training.
- For-fee training can contribute to the bottom line. Once a course has been converted to online form, the single course can be shown without the vendor having to pay humans to teach. Training has shifted from an expected service as it was in the Dark Ages circa 1970 to a source of revenue.
One final point may be worth noting. Google has emphasized that its Google Search Appliance is much simpler to configure, deploy, maintain, and customize than other vendors’ search systems. Having had the pleasure of configuring search appliances and modular search systems, Google may be on the right track with its simplicity argument.
I recall waiting several days for a specialist to deal with a problem with a modular search system which the licensee’s engineers, the search specialists from a third-party consulting firm, and the vendor’s on-site engineer could not figure out. That system was not just complex; it was convoluted with unknown, undocumented dependencies.
Maybe these online university programs are a way to solve the problem I couldn’t. Simplicity, Google-style, may be the wave of the future. Now that such vendors as Coveo, Exalead, ISYS Search Software, Siderean Software, and Vivisimo use the “s” word in their marketing, search that requires a “college” course may be a sign that complexity has become a way to sell training in order to sell more search options to an educated licensee? My experience suggests simplicity in search pays dividends. Complexity can create an insatiable appetite for resources.
Stephen Arnold, April 20, 2008