SOLR Relevancy Tuning from Search Technologies
February 11, 2013
Search Technologies introduced “Solr Lucene Relevancy Tuning.” Search Technologies will supply services to improve the relevancy of results within an existing Solr/Lucene implementation. If the service works as advertised, this could be a boon to many organizations awash with extraneous data. The announcement explains:
This engagement will provide powerful relevancy ranking improvements in an existing Solr installation. This includes setting up a basic system for relevancy evaluation, based on a set of sample queries, so that improvements can be quantitatively measured. Additions to the default relevancy formula in Solr Lucene can dramatically improve search results, solving many of the most thorny relevancy problems including:
- Reducing the impact of peripheral content (sidebars, ads, tangential discussions, etc.)
- Automatically handling word phrases in a flexible manner, reducing the need to use complex query constructions to obtain good search results.”
The Search Technologies’ solution changes the default Solr/Lucene functionality, which can overemphasize document size and term frequency. Search Technologies’ new Parameterized Document Similarity Function provides more control over these formulas through configurable parameters. The company’s Gradient Proximity Boost operator eliminates the need to tweak Solr/Lucene’s default “hard window,” the term-proximity parameters which can trigger a document boost. The method does this by measuring the density and completeness of terms across each document, gradually boosting documents in which terms cluster.
The post identifies the expected engagement tasks and deliverables associated with this software. The only pre-requisite listed is the presence of a working Solr /Lucene system with already-indexed documents. The firm promises ongoing maintenance and support services, including an optional round-the-clock support package.
Founded in 2005, Search Technologies bills themselves as the largest (independent) IT services company dedicated to search-engine implementation, consulting, and managed services. Staffed with veterans of the search field, the company prides itself on innovation. Search Technologies is headquartered in Herndon, Virginia, and maintains two other U.S. offices as well as locations in Berkshire, U.K., and San Jose, Costa Rica.
Ken Toth, February 11, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
PR in the Digital Arenas
January 27, 2013
I don’t know zip about public relations. First, I don’t do much “public” work. The connotation of “relations” remains mildly distasteful to me. I suppose that is a consequence of a high school English teacher who did not permit certain words to be used in class. If a student were to encounter a word on the banned list, he or she had to skip it when reading aloud. The notion of “public relations” gives me the willies.
You can check out the best in PR and real journalism in the scary “Microsoft: Google Blames Us for All Its Problems.” I thought I was jaded with corporate slickness. One is never too old to learn how the big guys handle communications.
I had a client ask me about a company which could post messages to LinkedIn and other social media. I motioned that the work was getting difficult. For example, Instagram wants a person who posts a picture to register with a government issued ID card. Now that is interesting because I use a passport for identification, and I am not too keen on having that information in the hands of a 20 something engineer working from a drafty apartment in a country to which the data processing has been outsourced. Also, LinkedIn has a number of groups which are managed by those who start the groups. LinkedIn wants anyone who found the group interesting to participate or the “member” is kicked out of the group. Some groups are lax about advertising. Other groups are not. LinkedIn has turned into a job hunting and marketing service, so its utility to me has declined. I find the “expert” commentary sent to me by LinkedIn employees annoying tool. Facebook is a wild and crazy place. I am not sure how the new Facebook search will work when a person posting can be linked to “interesting” topics and “friends.” The Google Plus thing is mandatory with each post linked to a “real” person. Maybe Google will just issue official ID cards and skip the government angle. Google’s mission to North Korea was fascinating, and I hope no one draws a connection between the Google visit and the increasingly hostile rhetoric from that country toward the United States.
So what about public relations.
I did a quick check online and found that a consulting and publishing company called O’Dwyer Company, Inc. publishes a list of the PR firms ranked by revenue. After all, what could be more important than revenue in today’s economic climate. (Do I hear a tiny voice saying, “Quality and integrity”? No, not here in Harrod’s Creek.
The list exists in a couple of different forms. The dates covered by the list are not clear to me. But the PR league table I reviewed contained 118 firms. Of these 118, the total revenue reported by O’Dwyer was $1,776,859,523, slightly more than the revenues for the enterprise search market which I wrote about here. The top 10 firms generated $1,120,706,215 or 63 percent of the total revenue in the O’Dwyer report. What’s interesting is that this concentration of money is similar to the concentration of revenues in enterprise search prior to the consolidation craze which peaked in 2012. Once a search vendor is absorbed into a giant new owner like Microsoft or Oracle, the revenues from search related deals disappears into the accounting miasma. Become too open about enterprise search revenues and an Autonomy type of situation may unfold.
What I found interesting was that of the top ten firms, two were flat with no significant increase in revenue and one new entrant was able to pump out $21 million quickly. Whoa, Nelly.
Another point I found interesting is that I recognized the “name” of these firms of the 118:
- Edelman, not sure why
- Waggener Ekstrom, the Microsoft PR outfit
- Ruder Finn, not sure why.
Several observations:
- PR seems to be a low profile business. I am confident that the big dogs know how to market, but I am quite certain that most of the firms do not build a “brand” nor do they play a role in my world as “thought leaders.” I presume the reason is that the PR firms are so focused on their clients that any visibility for the PR firm would be a big no no.
- The revenues for PR are almost identical to those reported for enterprise search by Forrester. Does this mean that PR is a better business from a revenue point of view that search or content processing. Presumably the search vendors hire PR firms so the cash available for search marketing helps pump up the PR revenues. Interesting, particularly at a time when it is difficult to track sales to PR. (After all, if PR worked, wouldn’t the firms showing flat and declining revenue use their own tools to get those sales going?)
- PR, like enterprise search, generates one of those nifty long tale graphs which are so popular in today’s learned discussions about “concentration,” “oligopolies,” and “market forces.”
I told the client to take the O’Dwyer list and pick a firm close to home. The challenge is that the biggest firms are in the big cities; for example, Manhattan boasts 31 firms on the list, more if I include New Jersey and Connecticut. A quick check of Louisville, Kentucky’s PR density revealed 18 firms. More were listed if I tossed in marketing communications, social media, and similarly nebulous terms. PR advisors are as plentiful as consultants it seems. The swelling ranks of the unemployed creates a fertile ground for advisors, wizards, mavens, and poobahs in search, business consulting, and public relations.
My big finding is that the vast majority of public relations firms are likely to be struggling to generate revenue. What’s new in today’s economy? Is PR a discipline? Don’t know. Don’t care. I do know I tell those who write me PR spam that I am not a journalist. I get pretty frisky when people ignore my about page and assume I am, at age 69, a real journalist. Heaven forbid that I should be confused with a real journalist, a PR person, or an effective marketer. I am none of those things. Never will be.
Stephen E Arnold, January 26, 2013
Forrester Fills the Gap in Search Market Size Estimates
January 25, 2013
I used to enjoy the search market size estimates of IDC (the time it takes to find info group), Gartner (the magic quad folks), Forrester (yep, the “wave” people), and Ovum (we do it all experts), among others.
I read “Growth of Big Data in Businesses Intensifies Global Demand for Enterprise Search Solutions, Finds Frost & Sullivan” and found several items of interest in the brief news story which arrived via Germany. Is Germany a leader in enterprise search? I heard that 99 percent of Germany’s search means Google. The numerous open source players are not setting the non-German world on fire, but I could be wrong. Check out GoPubMed, for example, of an interesting system which has a modest profile.
Now to the size of the search market.
The first thing I noticed was the nod to Big Data, which is certainly the hook on which many dreams for Big Money hang. With enterprise search vendors looking for a way to gain traction in a market which has been caught in awkward positions when licensing and deploying “search,” new words and new Velcro patches are needed. I won’t mention the Hewlett Packard Autonomy matter nor the Fast Search & Transfer matter nor the millions pumped into traditional search vendors with little chance of paying back the investments. No. No. No.
I want to quote this statement from :
The growth of Big Data across verticals presents the enterprise search solutions market with further opportunities. Since newer data types are not confined to a relational database within an organization, solutions that can search information outside the scope of these relational frameworks are widely accepted. Demand for personalized search tools that operate in a pool of unlimited data from internal servers, the Internet, or third-party sources is also growing.
Ah, but how does one crawfish away from exaggeration? Easy. I noted:
However, the disparity between customer expectations and actual search outcomes could dissuade future investments. Customers expect a single query to retrieve the right results immediately. Therefore, search providers must offer timely and relevant results, taking into account the continuous addition of new data to repositories.
But “How big is the market? my inner child yelps. The answer:
Hybrid Cloud with Cloning Capability May Not Bode Well for Cloud Platform Developers
December 17, 2012
The introduction of hybrid technology comes as no surprise, but one has to wonder how current developers will feel about being cloned in the future. TechCrunch’s article “CloudVelocity Launches With $5M from Mayfield to Bring the Hybrid Cloud to the Enterprise” discusses the introduction of a hybrid cloud and its growing potential, along with its cloud cloning ability.
This new technology could save companies a bundle off initial investments, but smart platform designers may take precautions against cloning in the future. One has to wonder what preparations have already been made, if any. Investors want to be certain the risk of this approach is worth the effort.
“One Hybrid Cloud platform, aims to extend the enterprise data center to the public cloud, by enabling multi-tier applications to run without modification in the cloud and access services that reside in the enterprise data center. In a nutshell, the startup allows enterprises to get the benefits of private clouds in the public cloud. Users can discover, blueprint, clone, and migrate applications between data centers and public clouds. Currently, CloudVelocity supports full server, networking, security and storage integration with AWS but plans to integrate other public clouds.”
The excitement around startups and cloud solutions is great but corporations are reluctant to take chances with sensitive data. Those enterprises seeking stability in the growing hybrid cloud universe may find some assurance in relying on a mature, capable enterprise provider. Intrafind offers consultative solutions and reliable cloud solutions with secure access.
Jennifer Shockley, December 17, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
Rise Beyond the Cloud: Enterprise Search Consultation Powers Decisions
November 23, 2012
Savvy companies will not be getting their head out of the clouds anytime soon because the cloud is the place to be for businesses seeking success. ZDNet’s article “A Snapshot of Enterprise Cloud Adopters at Workday Rising” explains why the sky is not the limit when it comes to the enterprise cloud as it is continuing to evolve.
In order to keep up with the growing demands of the enterprise, Cloud providers have to stay on top of the latest developments:
“Having opted for the speed, agility and on-demand responsiveness of a cloud platform, customers rely on their chosen vendor maintaining the same pace going forward. One of the most remarkable things about watching a cloud vendor like Workday evolve is the extent of innovation that happens in the underlying infrastructure. These are not static structures. New components are constantly being introduced that advance performance, scalability and functionality — not just the three-times-a-year functional updates but architectural advances too.”
With the ongoing evolution of the enterprise, utilizing the expert advice of an established enterprise search consultant can keep a company’s head above the cloud, so to speak. Cloud search adopters could benefit from the services of enterprise search providers who offer a full range of assessment and solution selection services. Intrafind has offered solid guidance to customers seeking the right solution that improves efficiency and offers a solid return on investment in enterprise search for well over 10 years.
Jennifer Shockley, November 23, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
Quote to Note: Real Consultant Illuminates HP Autonomy
November 22, 2012
Heck, let’s put the 451 Group quote in granite.
I am getting ready to head from England, the land my ancestors fled in 1596, to return to Harrod’s Creek, Kentucky, the culinary center for down home Thanksgiving cooking. (We do squirrel stew. How about you? Now I am sitting in the Dublin airport relishing comments allegedly made by a “real” expert, a 451 Group consultant no less. My source is “HP-Autonomy Fraud Allegations Fallout: The Winners and Losers.” Please, read the original article. It is priceless.
Here is what has me seeing those cartoon stars and exclamation points popular in the funny papers:
The fraud scandal “probably will” affect Autonomy’s fortunes “and I think that’s a shame…Autonomy always had very good technology, but they weren’t able to leverage it very effectively. In HP’s hands, it could be leveraged much better. On the other hand, Autonomy was “acting as a holding company to a large extent, anyway,” following many acquisitions. Only its core IDOL product is quite closely linked to the Autonomy name, “There are some good pieces there and still some strong brands.” (Attributed to a real consultant doing business as Alan Pelz-Sharp.)
Yowza. Consider these observations I jotted down.
- The word “probably”. Better safe than confident in one’s facts.
- I absolutely love “shame”. Yep, investment bankers are really sensitive to shame, particularly a year after a deal closes and the fees have been paid.
- The assertion “they [Autonomy] were not able to leverage it.” Stunning. I thought Autonomy had been in business for 13 or 14 years and had generated north of $700 million, bought a number of companies, and got its logo on a football team and an F1 car. To my albeit limited knowledge, no other search vendor has come close to Autonomy’s top line revenue. $700 million is a healthy figure and somewhere along the line taxes had to be paid, financial reviews conducted, and invoices sent.
- Autonomy was a holding company. Okay, this is not the analogy I would choose but let’s assume Autonomy was a holding company. Perhaps the 451 Group’s expert knows line of business revenues and pre tax profits by each of Autonomy’s operating units. Where are the breakdowns?
- Only IDOL is linked to Autonomy. My obviously misguided notion was that Autonomy bought a company and then slapped the IDOL brand on the company’s products and services. Then integration tagged along but Autonomy management worked hard to make IDOL the brand for many of its services as Autonomy managed the heck out of staff costs and embarked on upselling. Guess I was ill informed.
- “There are some good pieces” begs some detail. Perhaps a good piece is Virage or the virtual reality Aurasma technology which I covered in one of my Information Today columns?
- Again that categorical “only” troubles me but I will soldier on.
- Yes, I presume fraud will have some impact.
- I find the Latinate predicates stimulating as well.
Some color is in order.
451 Group (the “number” references either a science fiction novel or the flash point of paper, which is going out of fashion among some digerati) is a consulting and professional services firm with expertise in “cloudscape,” “multitenant data centers, and a half dozen other specialties. Disclaimer. I used to work for Booz, Allen & Hamilton and now I write “reports” for IDC, another outfit in the same line of work at 451. This means that one has to determine which consulting firm is on the beam. I will let the reader decide, but my bet is on folks who have done time at one of the pantheon firms, not the firms lower in the rental brains’ pecking order.
I have commented on Autonomy is a number of published reports. These range from the now hoary Enterprise Search Report which first appeared in 2003 or 2004 to my 2011 study “The New Landscape of Enterprise Search.” Free download here. I had to sit through various briefings, do some independent verification and validation work, and interact with some of the folks who had a little first hand experience with things Autonomy. I have to check my files, but I think I did a tiny job for Autonomy, but it is tough for me to remember after 40 years of blundering around in the online, search, and content processing sectors. Unlike the 451 Group’s experts, I am not qualified in the cloudscape thing or the multi whozit. I am so so in search and okay in a couple of other disciplines. I eat squirrels too.
My official position is that folks who license enterprise search systems like Autonomy or Endeca usually wind up with some interesting challenges and some hefty bills. When one taps Autonomy-like system or even a free, open source search solution, the likelihood of financial excitement six months into a search project is quite high. When signing on for an enterprise search system, arrogance and ignorance on the part of the customer can equal the spirit of a gaggle of Googlers. One difference: The Googlers are usually okay with resources, money, code and math. Procurement teams, in my experience, assume their intellectual firepower would decimate other mortals. Wrong. But optimism can be a positive attribute. Just not in big time search solutions.
Enter the marketers, lawyers, and MBAs.
Now a covey of these professionals can draw fascinating conclusions about enterprise search. My problem with the write up and the quote is that the facts of the matter are stark. HP paid billions for a search vendor. HP is in a bit of a financial muddle. HP wants to blame everyone except itself. The losers omitted from the write up are the ones which matter: HP employees, HP management, and the HP Board of Directors.
Maybe the accounting was erroneous. Maybe the auditors stayed up late and drifted through the various audits. Maybe the lawyers, MBAs, and auditors were more focused on their billable hours and commissions. Maybe Autonomy explained how search and content processing deals worked and assumed that the HP team “understood” the upsides and downsides of search.
Somewhere along the line “real” consultants and the legions of people grabbing a piece of the billion dollar deal may be revealing their own intellectual shortcomings. Search is tough. The price HP paid for Autonomy is what it is. A year after buying a property, HP wants its money back. That’s the charming world of business today.
With “real” consultants and big companies looking for a silver bullet to complete their imaginary magic wands, bad deals are the rule. What makes Autonomy interesting is the amount paid by a company which is fraught with management, financial, and technical challenges. I used to keep a list of missteps. But after fumbling the Alta Vista search system opportunity, I lost interest. HP and the “real” consultants want to explain away a mistake without pointing out who triggered the problem. Arrogance, superficial analysis, and lack of search expertise caught fire when big money was in play. Combustible mixture? Yes, indeed.
Stephen E Arnold, November 22, 2012
Forrester 2012, Gartner 2010, and Autonomy
November 21, 2012
I don’t have much interaction with Autonomy and I have even less with Forrester or Gartner, both azure chip consultants stuffed with high IQ big thinkers about technology. How can someone residing in Harrod’s Creek hope to compare to sleek, real consultants who work in cities with electricity and running water.
However, on a recent trip to a dumpy Internet cafe near Victoria Station, I read “Hewlett’s Loss: A Folly Unfolds, by the Numbers.” In that article I noted this quote from an azure chip consultant working at the tony Forrester Research outfit. Here’s the passage that made my feathers twitch:
Autonomy, too, was facing challenges after years of fast growth but poor customer relations, according to Leslie Owens, an analyst with Forrester Research. “They didn’t invest in R&D; they didn’t have regular software releases; they weren’t transparent with a road map of where they were going; they didn’t seek customer feedback,” she said. “Customers complained, but the promise of managing all their information and making better decisions was so attractive. They bought more.” Soon after the H.P. acquisition, Ms. Owens said, Autonomy announced a new version of its core product. “We asked for a demo,” she said, and “we’re still waiting.”
Okay. I remember seeing a Boston Consulting Group dog, question mark, star, and cow type chart in 2009. Allegedly produced by another high end think tank, Gartner Group. I did not recall Autonomy getting low marks. I did some poking around and I would like to direct you, gentle reader, to this Web address: http://www.contentmanager.eu.com/graphics/gartner-wcm2010.jpg.
I am fearful of azure chip retribution, so you have to navigate to the page and look at the 2010 BCG style chart by Gartner Group experts.
What is interesting is that Gartner pegs Autonomy in the leaders quadrant for Web content management. I don’t know what that means. I do understand what it means to be a “leader”, singled out for excellence on whatever yardstick was used to size up 17 vendors of a particular type of enterprise software.
What is interesting is that two expert consulting firms have such conflicting opinions about Autonomy less than 18 to 20 months apart. Forrester “knows” that Autonomy had some issues. Gartner seems to find the company superior to such rivals as IBM and Microsoft.
Did Autonomy crash and burn between these two azure chip viewpoints? Are Forrester’s analysts more sveltish and brighter than Gartner’s high protein crowd?
Assume that each of these consulting outfits have comparable intellectual horsepower. Assume that each firm’s experts gathered information from open source and private sources. What causes two apparently superficial assessments of Autonomy.
My question: “If two blue chip consultants see Autonomy differently, won’t the truth and beauty of Autonomy will be in the eye of the beholder?”
In a legal dispute, subjective, maybe emotion, will play a larger role than dull old objective data. Little wonder so many advisors interpreted Autonomy differently. Enterprise software as an interpretation problem in 21st century business poetry. Lawyers are happy. HP and its shareholders, not so much.
Stephen E Arnold, November 21, 2012
ElasticSearch Analysis Published by IDC
November 15, 2012
Short honk: The ArnoldIT team worked with IDC to produce “ElasticSearch: An Open Source Search Option for Big Data.” The write up discusses the origins of the company. Compass was the precursor of Elastic Search. My understanding is that Compass was built on Lucence. The new incarnation is built on Solr. There is a discussion of ElasticSearch’s enhancements to the Solr system. The most significant part of the report is the explanation of the advantages and disadvantages of the ElasticSearch approach. The analysis is not written from the developers’ point of view. The focus is on the business value of ElasticSearch in the highly volatile, increasingly crowded market for search systems based on open source technology. Already published in the multi-part research series are analyses of Attivio, LucidWorks, and PolySpot. Unlike the cheerleading on free blogs and developer forums, the IDC analyses cost $3,500 per report. IDC customers have access to the analyses, but should check with their IDC account manager to determine if access is permitted under their subscription plan.
Stephen E Arnold, November 15, 2012
Sponsored by ArnoldIT. Watch for our new professional social media consulting services coming in 2013
Tweaking the Cuban Two Part World
August 26, 2012
I agree in general with “Which USA Do You Work In? The premise is that it is good to be smart and digital. Those who are not smart and pretty much users of ATM machines or maybe robbers of those who use ATM machines. Mr. Cuban, a digital and smarts mogul, writes:
The problem for those who work in brick and mortar is that as the intelligence is sucked out of the job. The intelligence required to do the
job is reduced. Yes, you still have to be good at what you do. But you can be great at customer service or great in a factory line with out a college education. The competition for jobs that don’t require degrees has pushed down the wages paid for brick and mortar jobs as well. When there are no specific skills beyond basic people and communication skills required the job pool competing for any openings expands considerably. Forcing down wages. Leaving more unemployed unemployed. The other unfortunate part of working brick and mortar is that as intelligence is moved out of of physical locations it also reduces the number of jobs available. Have you ever seen a cashier at an Apple Store ? Unemployment is sky high in the brick and mortar world.
In my goose pond, the split is described as “knowledge value.” And “split” is not exactly right. Knowledge value suggests that information can be monetized. If the facts or skills one possesses match a need, then one may be able to charge to deliver knowledge value. Skip the socio-political implications of this idea and focus on work, money, and influence.
I like to visualize one of those sliding controls in a user interface. Move the virtual know to the left and the knowledge value drops to zero. Slide the virtual knob to the right and you get into McKinsey and Bain territory. If you know about the compensation for blue chip consultants, you see the monetary value of the high knowledge value setting.
What’s this mean to the issues Mr. Cuban addresses?
First, those who have low knowledge value jobs are stuck unless the individuals pump up their knowledge value. Here’s how it works. You know how to fix an MBA’s laptop so it will print a document. You can do this for free or you can charge big bucks. If the person with the know how is into the knowledge value game, the MBA may pony up some cash to get the document. Now think about a nasty legal situation. Do you want a low knowledge value or a high knowledge value attorney helping you out?
Second, some knowledge work can deliver a big payday. I suggest people think about getting jobs in terms of knowledge value. When one leverages knowledge value in an optimal way, money awaits the person who can find a person or company requiring knowledge value. How does one get knowledge? How does one communicate value? High knowledge value has more magnetism than low knowledge value. One can determine one’s knowledge value when others find you.
Do schools teach this knowledge value method? See The Knowledge-Value Revolution, Or, a History of the Future.
Booz Allen: Protesting a Wee Bit Too Much
July 25, 2012
Short honk: Check out “Why Booz Allen Is Significantly Undervalued.” Interesting. The main point, as I understand it, is that Booz Allen is a great deal for investors. Okay, if it is such a great deal why does it take 1,300 words and a dozen charts to get the point across. There was one omission in the “protest too much write up”. Booz Allen is dependent on government contracts. When governments cut spending, outfits like Booz Allen face some revenue challenges. I assume this aspect of the firm’s business is irrelevant in light of alleged backlog, explanations, and lots of tables and bar charts. Booz Allen was once a blue chip outfit. As I have noted, the firm has a history of being private, going public, going back to private, splitting up, and going public. Is the firm’s management fancy dancing to cope with the realities of a blue chip firm transforming to azure?
Stephen E Arnold, July 25, 2012
-
- Subscribe to Beyond Search
Feature archive
News archive
-
