Short Honk: High Value Podcast about Solr
August 4, 2012
If you are interested in Lucene/Solr and have a long commute, you will want to check out Episode 187 of the IEEE’s Software Engineering Podcast. You can find the podcast on iTunes. Grant Ingersoll, one of Lucid Imagination’s experts in open source search and a committer on the Apache Lucene/Solr project, reviews the origins of Lucene, explains the features of Solr, and covers a range of important, hard to get search information. According to IEEE, the podcast offers a:
dive into the architecture of the Solr search engine. The architecture portion of the interview covers the Lucene full-text index, including the text ingestion process, how indexes are built, and how the search engine ranks search results. Grant also explains some of the key differences between a search engine and a relational database, and why both have a place within modern application architectures.
One of the highlights of the podcast is Mr. Ingersoll’s explanation of vector space indexing. Even a high school brush with trigonometry is sufficient to make this important subject fascinating. Highly recommended.
Stephen E Arnold, August 4, 2012
Sponsored by Augmentext
Research and Development Innovation: A New Study from a Search Vendor
August 3, 2012
I received message from LinkedIn about a news item called “What Are the Keys to Innovation in R&D?” I followed the links and learned that the “study” was sponsored by Coveo, a search vendor based in Canada. You can access similar information about the study by navigating to the blog post “New Study: The Keys to Innovation for R&D Organizations – Their Own, Unused Knowledge.” (You will also want to reference the news release about the study as well. It is on the Coveo News and Events page.
Engineers need access to the drawings and those data behind the component or subsystem manufactured by their employer. Text based search systems cannot handle this type of specialized data without some additional work or the use of third party systems. A happy quack to PRLog: http://www.prlog.org/10416296-mechanical-design-drawing-services.jpg
The main of the study, as I interpret it, is marketing Coveo as a tool to facilitate knowledge management. Even though I write a monthly column for the print and online publication KMWorld, I do not have a definition of knowledge management with which I am comfortable. The years I spent at Booz, Allen & Hamilton taught me that management is darned tough to define. Management as a practice is even more difficult to do well. Managing research and development is one of the more difficult tasks a CEO must handle. Not even Google has an answer. Google is now buying companies to have a future, not inventing its future with existing staff.
The unhappy state of many search and content processing companies is evidence that those with technological expertise may not be able to generate consistent and growing revenues. Innovation in search has become a matter of jazzing up interfaces and turning up the marketing volume. The $10 billion paid for Autonomy, the top dog in the search and content processing space, triggered grousing by Hewlett Packard’s top executives. Disappointing revenues may have contributed to the departure of some high profile Autonomy Corporation executives. Not even the HP way can make traditional search technology pay off as expected, hoped, and needed. Search vendors are having a tough time growing fast enough to stay ahead of spiking technical and support costs.
When I studied for a year at the Jesuit-run Duquesne University, I encountered Dr. Frances J. Chivers. The venerable PhD was an expert in epistemology with a deep appreciation for the lively St. Augustine and the comedian Johann Gottlieb Fichte. I was indexing medieval Latin sermons. I had to take “required” courses in “knowledge.” In the mid 1960s, there were not too many computer science departments in the text indexing game, so I assume that Duquesne’s administrators believed that sticking me in the epistemology track would improve the performance of my mainframe indexing software. Well, let me tell you: Knowledge is a tough nut to crack.
Now you can appreciate my consternation when the two words are juxtaposed and used by search vendors to sell indexing. Dr. Chivers did not have a clue about what I was doing and why. I tried to avoid getting involved in discussions that referenced existentialism, hermeneutics, and related subjects. Hey, I liked the indexing thing and the grant money. To this day, I avoid talking about knowledge.
Selected Findings
Back to the study. Coveo reports:
We recently polled R&D teams about how they use and share innovation across offices and departments, and the challenges they face in doing so. Because R&D is a primary creator and consumer of knowledge, these organizations should be a model for how to utilize and share it. However, as we’ve seen in the demand for our intelligent indexing technology, and as revealed in the study, we found that R&D teams are more apt to duplicate work, lose knowledge and operate in soloed, “tribal” environments where information isn’t shared and experts can’t be found. This creates a huge opportunity for those who get it right—to out-innovate and out-perform their competition.
The question I raised to myself was, “How were the responses from Twitter verified as coming from qualified respondents?” And, “How many engineers with professional licenses versus individuals who like Yahoo’s former president just arbitrarily awarded themselves a particular certification were in the study?” Also, “What statistical tests were applied to the results to validate the the data met textbook-recommended margins of error?”
I may have the answers to these questions in the source documents. I have written about “number shaping” at some of the firms with which I have worked, and I have addressed the issue more directly in my opt in, personal news service Honk. (Honk, a free weekly newsletter, is a no-holds-barred look at one hot topic in search and content processing. Those with a propensity to high blood pressure should not subscribe.)
Making Data Easy with Training Wheels? The Nielsen Dust Up
July 31, 2012
In the Honk newsletter, I have been plugging away at some of the flights of fancy that surround big data, next generation analytics, and all things predictive. I am nervous about “training wheels” on complex mathematical processes. Like the fill-in-the-blanks functions in Excel, a person without a foundation in math can fiddle around until the software spits out a number which “looks good.” In one of my jobs, my boss was a master at “the flow.” The idea was that numbers can be shaped to support a particular point. I recall his comment to me in 1974, “Most of our clients are not smart enough to work through the math. We have to generate outputs which flow.” The idea is one that troubled me. I moved on to greener and less slippery pastures and I kept that notion of “flow” squarely in mind. Numbers should not cause the person looking at a chart or a table to say, “Wow, that number looks weird.” Hence, flow allows the reasoning process to be guided.
I just read a story which I hope is not accurate. I want to document my coming across the item, however. I think it will be an interesting touchstone and search and content processing companies race to be come players in big data and analytics. The story appeared in the Hollywood Reporter, a publication about which I know little. The headline caught my attention because it resonates with advertising and advertising automatically evokes the Google logo for me. “Nielsen Sued for Billions over Allegedly Manipulated TV Ratings” carries a hard hitting subtitle too: “In a huge new lawsuit, the business of TV ratings is fingered for rampant corruption by India’s largest TV news network.” I know even less about India than the Hollywood Reporter.
Fancy math underlies the products and services of many analytics firms which offer products and services to licensees that make interacting with data a matter of pointing and clicking. A happy quack for the equation to http://goo.gl/lBlXV
Here’s the passage I noted:
In a 194-page lawsuit filed in New York court late last week, NDTV accuses Nielsen of violating the Foreign Corrupt Practices Act by manipulating viewership data in favor of channels that are willing to provide bribes to its officials. According to NDTV, rampant manipulation of viewership data has been going on for eight years, and when presented with evidence earlier this year, top executives at Nielsen pledged to make changes. But the Indian news giant says these promises have been false ones.
Like most litigation, the story will unfold slowly and perhaps not at all. The i2 Group Palantir litigation is a relatively recent example. Based on my experience with the boss who wanted numbers to flow, I can see how the possibility of tweaking could be useful to some companies. However, with the dismal state of math skills, how can I now of the problem was a result of human intent, human error, or a training wheels type system driven over rocky terrain. I can’t and I bet that most people thinking about this situation cannot either.
What is interesting to me, however, are these notions:
- How many other fancy math systems are open to similar allegations from their licensees?
- Will this type of legal action cause some of the vendors pitching fancy math and predictive systems to modify their marketing materials to include more caveats and real world anchors instead of bold assertions?
- How will the legal system deal with fancy math litigation? I don’t know many attorneys. The handful with which I have some experience have been quick to point out that math, engineering, and science were not their strengths. Logic and reasoning were their strong suits.
With many search and content processing companies embracing fancy math, sentiment analysis, smart indexing and other math-based functions, will a search vendor find itself in the hot seat? I hope not but the market wants to buy fancy math. Understanding the fancy math may drive demand for individuals who can figure out if the systems and methods do what the licensee believes they do.
Oh, I like the word “billions.” Big money adds to the drama of analytics risk management in my opinion.
Stephen E Arnold, July 31, 2012
Microsoft and Yammer: Extending SharePoint Functionality
July 31, 2012
Yammer is what an enterprise social network tool; organizations implement it to spur collaboration between users. On the Yammer homepage we found a new application which permits Microsoft SharePoint Integration. After reading the specs, we found on the Microsoft blog about “Yammer-The Next Step for Social Networking In Schools?”
According to the post, Microsoft recently purchased Yammer. The post explains Yammer’s basic functions, the dashboard mirrors Facebook’s design with hints of Twitter. The post digs into how Yammer would be used in schools, basically the same way it would for any company: staff would use to communicate between departments, share content, etc. It can also be a boon for students too:
“We know that group work is a great way to encourage students to engage with their peers, but this isn’t easy when they all use different social networks, clouds and systems. By joining Yammer, students can create secure groups via which they can communicate their ideas, ask questions and share files, as well as allowing for their competitive side to come out through ‘Leaderboards’, which show data about who has received the most likes, replies and much.”
Students can perform group work, receive studying help, share content, and even praise each other within Yammer. While it can be a tool of food for students, it can also make cheating and plagiarism easier if not monitored. Yammer should install an app that will be able to detect plagiarism.
The surge of interest in social content is growing in government agencies, commercial organizations, and educational institutions. However, indexing and making this content
findable can be a challenging task. The tools an organization uses require tight integration with
a search system. Mindbreeze provides capabilities to make social content easily findable within a SharePoint environment. A Yammer style can enhance productivity. Mindbreeze offers a range of social and collaborative features and has the engineering expertise to resolve almost any search and retrieval issue. Check out the Mindbreeze social collaboration Web page for more information.
Whitney Grace, July 31, 2012
Sponsored by Mindbreeze
Latent Semantic Technology Tops Business Strategy List
July 26, 2012
The current 2012 year is going by quickly, but there is still time to implement business strategies that could gain your company a bigger presence on the Internet. Venture Beat reported on the “Top 10 Most Important SEO and Social Marketing Tactics of 2012.” Generally these top ten lists yield information we already know: distribute content via social channels, list your social media connection buttons prominently on the page, enable sharing content, join Pinterest, etc. Some of the ideas are new: author guest blog posts, keep your own blog content interesting and new, but the number one suggestion that caught our attention was:
“Get an onsite SEO audit: an onsite SEO audit is the foundation of your SEO campaign. Getting one will help you answer questions like: Are your title and meta tags optimized? How’s your keyword density? Have you correlated certain pages with certain keywords? Is that evident in the copy? Have you done your LSI (latent semantic indexing) research and incorporated it into the copy? An onsite SEO audit is relatively cheap, and it’s a one-time payment that you shouldn’t need to address more than once a year.”
An SEO audit done by a professional company will work wonders, heck, if you do your research you can do provide the service for yourself. One important aspect of the audit is latent semantic indexing, a powerful component of text and document analysis.
Whitney Grace, July 26, 2012
Sponsored by Polyspot
The TREC 2011 Results and Predictive Whatevers
July 20, 2012
Law.com reports “Technology-Assisted Review Boosted in TREC 2011 Results” how technology-assisted review boasts that it may be capable of ousting predictive coding’s title. TREC Legal Track is an annual government sponsored project (2012 was canceled) to examine document review methods. From the 2011 TREC, participants voted in favor of technology-assisted review, but it may have a way to go:
“As such, ‘There is still plenty of room for improvement in the efficiency and effectiveness of technology-assisted review efforts, and, in particular, the accuracy of intra-review recall estimation tools, so as to support a reasonable decision that ‘enough is enough’ and to declare the review complete. Commensurate with improvements in review efficiency and effectiveness is the need for improved external evaluation methodologies,’ the report states.”
The 2011 TREC asked participants to test three document review requests, but different from past years the rules were more specific in requirements by having participants rank documents as well as which were the most responsive. The extra requirement meant that researchers were able to test hypothetical situations, but there were some downsides:
“TREC 2011 had its share of controversy. ‘Some participants may have conducted an all-out effort to achieve the best possible results, while others may have conducted experiments to illuminate selected aspects of document review technology. … Efficacy must be interpreted in light of effort,’ the report authors wrote. They noted that six teams devoted 10 or fewer hours for document review during individual rounds, two took 20 hours, one used 48 hours, and one, Recommind, invested 150 hours in one round and 500 in another.”
We noticed this passage in the write up as well:
“`It is inappropriate –- and forbidden by the TREC participation agreement –- to claim that the results presented here show that one participant’s system or approach is generally better than another’s. It is also inappropriate to compare the results of TREC 2011 with the results of past TREC Legal Track exercises, as the test conditions as well as the particular techniques and tools employed by the participating teams are not directly comparable. One TREC 2011 Legal Track participant was barred from future participation in TREC for advertising such invalid comparisons,’ the report states.”
TREC is sensitive to participants who use the data for commercial purposes. We wonder which vendor allegedly stepped over the end line. We also wonder if TREC is breaking out of the slump which traditional indexing seems have relaxed into. Is “predictive” the future of search? We are not sure about the TREC results. We do have an opinion, however. Predictive works in certain situations. For others, there are other, more reliable tools. We also believe that there is a role for humans, particularly when the risks of an algorithm going crazy exist. A goof in placing an ad on a Web page is one thing. An error predicting more significant events? Well, we are more cautious. Marketers are afoot. We prefer the more pragmatic approach of outfits like Ikanow and we avoid the high fliers whom we will not name.
Stephen E Arnold, July 20, 2012
Sponsored by Polyspot
Useful Graphics Explaining SharePoint 2013
July 19, 2012
SharePoint developers are eagerly waiting for SharePoint 2013. A blogger at the Microsoft Blogs wrote “SharePoint 2013-Initial Take On Changes To Search” and he has been viewing a lot of slideshows on the new version. His favorites are at SharePoint 2012: Presentation: IT Pro Training and all are easy to download. He takes a look a the Module 7: SharePoint Search 2013 that takes an in depth view into enterprise search, including architectural changes to physical and logical topologies and configuration details on crawling, content, and query.
Fast Search functionality is behind much of the SharePoint 2013 enterprise search capability:
“ From the SharePoint 2013 slides, it’s pretty clear that the rumors have played out and core components of SharePoint Search (particularly the Indexing pipeline) effectively got replaced by the Fast Search pipeline… although it will maintain the ‘SharePoint Search’ moniker (Disclaimer: I’m not a marketing guy and have no idea what the licenses will be, so this is just my observation).”
There is a lot of content to digest in from the presentations, but the article pulls out the very detailed and informative diagrams to understand how Fast Search has and will change the search architecture for SharePoint. With more than 30,000 consultant days of Fast implementation experience at Search Technologies, we will be gearing up early to support SharePoint 2013 Search Rollouts.
Iain Fletcher, July 19, 2012
Sponsored by Search Technologies
Predictive Coding Sparks Some Opposition
July 11, 2012
Editor’s Note: The original write up included observations which we learned were “out of bounds.” We want to make a formal apology to those mentioned in the article.
We wrote about a story in The New York Law Journal. You can find the source document “Judge Rejects Recusal Over Support for eDiscovery Method” at this link. We heard from one of our advertisers that Recommind www.recommind.com, a company known for its work in predictive coding, objected to the summary and the opinions expressed in the source article. We have, based on the statements communicated to us by our advertiser, removed the 256 word summary, quote from the source article, and the opinions of the person who wrote the story for Beyond Search. We deeply regret writing an abstract which offended Recommind.
We are sorry that our approach to creating useful pointers to articles of interest offended Recommind, the other individual referenced in the special letter to our advertiser, and to any of our readers who found our article problematic. We have a procedure in place, and we try to allow writers scope and try to point to useful source articles. We want to help inform people about important articles. The opinion the writers express are designed to add value to the abstract. As we note in the About section to this Web log, we are performing a specific type of abstracting and indexing service. We wrote in January 2008 and updated the statement in November 2011:
The data and information provided on this site are for informational purposes only. I, Stephen E. Arnold, make no representations as to accuracy, completeness, timeliness, objectivity, suitability, or validity of any information in a write up or on this site. Our content is pegged to source materials which are reachable via a public network. If you read something in Beyond Search, it is not news or “real” journalism. My views and opinions change. Frequently. Expect to find variances when you compare certain essays with my other written work. I started the Beyond Search Web log for myself, promote my studies, capture information that won’t be in my for-fee work, and have a time stamped record of what I was thinking, why, and when. I am human; therefore, I make errors. If you read something and accept it without verifying my information and interpretation, that’s your decision. Got a problem with my approach? Do not read this Web log. The information is provided on an as-is basis and, at this time, without a fee. Just in case: If I say something dumb in the future, it’s better to be able to point out that the error is mine and a mistake should not be a surprise.
We are sorry, apologize, and will make an effort to do an even better job going forward.
Stephen E Arnold, July 16, 2012, 1 30 pm Eastern
[Source article with offending and objectionable comments, statements, and opinions removed by request.]
We have heard the predictive coding and eDiscovery are the legal community’s future, but change is always met by resistance. The New York Law Journal recounts one of the first challenges for predictive coding: “Judge Rejects Recusal Over Support for eDiscovery Method.” The article describes the case Moore v. Publicis Groupe, 11 Civ. 1279 and how Magistrate Judge Andrew Peck refused to recuse himself from the case. We noted this passage.
” ‘Here, my comments at eDiscovery conferences related to the general use of predictive coding in appropriate cases, and I did not express any opinion regarding the specific issues in the case. Consequently, neither my comments nor the fact that Losey was on some panels with me, nor the fact that MSL’s vendor Recommind sponsored different panels at LegalTech, separately or collectively, are a basis for recusal.’ “
Whitney Grace, July 11, 2012 and revised on July 16, 2012
Sponsored by Content Analyst
AccessData and dtSearch Team Up for Connectors
July 9, 2012
Market Watch recently published the news release “AccessData and dtSearch Announce New AccessData Offering of Connector Libraries for Developers Using the dtSearch Engine.” The article discusses a new text indexing solution that repackages eDiscovery product lines with robust and flexible data connectors.
According to the article, these connectors are currently available for Microsoft Exchange – any user mailbox; Symantec Enterprise Vault (Exchange) – archived Exchange user mailboxes; Oracle URM – all documents. There are also plans to include other content repositories in the near future.
When discussing the new data connectors, Devin Krugly, VP of Marketing and Business Development at AccessData explains:
“These data connectors are purpose-built to integrate with dtSearch technology and allow for full text indexing of a repository’s content. These indexes can then be used to support a wide variety of business solutions including records management, information governance, internal investigations, audit and e-discovery. The harmony of the AccessData connector packages along with the broad data and fielded search options of dtSearch make this OEM offering a malleable addition for solutions of all sizes.”
Based on the expertise that both companies independently bring to the table, I am excited to see what this product brings to the enterprise search industry.
Jasmine Ashton, July 9, 2012
Sponsored by PolySpot
An Ode to Databases
July 4, 2012
The Damien Katz blog recently published a piece on the importance of databases, not unlike a love letter, called “Why Database Technology Matters.”
After rambling on for several paragraphs about how he finds databases so fascinating, Katz starts to narrow into his thesis which is that databases are one of the most significant advancements of humanity. They are as important as telecommunications and the Internet, as well as libraries (the first non-digital database).
After stating this point, Katz then goes on to discuss the databases created by IBM, Google, and Oracle. Katz writes:
“When IBM was at the absolute height of its power, they were the richest, most powerful company on the planet. They primarily sold mainframes for a lot of money, and at the core of those mainframes were big database engines, providing a big competitive advantage their customers gladly paid for.
Google has created a database indexing of the internet. They are force because they found ways to find meaning in the massive amounts of information already available. They are a very visible example of changing the way humanity thinks.”
This piece makes some interesting points regarding the impact that databases have had on our society. I admire Katz’s passion for the subject.
Jasmine Ashton, July 4, 2012
Sponsored by IKANOW