• Home
• About
• Writers
• Landscape
• Wizards Index
• Fraud
• Wanna Be Like Larry?

Quote to Note: Dick Brass on MSFT Innovation

I met Dick Brass many years ago. He left Oracle and joining Microsoft to contribute to a confidential initiative. Mr. Brass worked on the ill-fated Microsoft tablet, which Steve Jobs has reinvented as a revolutionary device. I am not a tablet guy, but one thing is certain. Mr. Jobs knows how to work public relations. Mr. Brass published an article in the New York Times, and it captured the attention of Microsoft and millions of readers who enjoyed Mr. Brass’s criticism of his former employer. I have no opinion about Microsoft, its administrative methods, or its ability to innovate. I did find a quote to note in the write up:

Microsoft is no longer considered the cool or cutting edge place to work. There has been a steady exist of its best and brightest. (“Microsoft’s Creative Destruction”, the New York Times, February 4, 2010, Page 25, column 3, National Edition)

Telling because if smart people don’t work at a company, that company is likely to make less informed decisions than an organization with smarter people. This applies in the consulting world. There are blue chip outfits like McKinsey, Bain, and BCG). Then there are lesser outfits which I am sure you can name because these companies “advertise”, have sales people who “sell” listings, and invent crazy phrases to to create buzz and sales. I am tempted to differentiate Microsoft with a reference to Apple or Google, but I will not. Oh, why did I not post this item before today. The hard copy of my New York Times was not delivered until today. Speed is important in today’s information world.

The quote nails it.

Stephen E Arnold, February 7, 2010

No one paid me to write this, not a single blue chip consulting firm, not a single savvy company. I will report this lack of compensation to the experts at the IRS, which is gearing up for the big day in April.

Lazarus, Azure Chip Consultants, and Search

A person called me today to tell me that a consulting firm is not accepting my statement “Search is dead”. Then I received a spam email that said, “Search is back.” I thought, “Yo, Lazarus. There be lots of dead search vendors out there. Example: Convera.

Who reports that search has risen? An azure chip consultant! Here’s what raced through my addled goose brain as I pondered the call and the “search is back” T shirt slogan:

In 2006, I was sitting on a pile of research about the search market sector. The data I collected included:

• Interviews with various procurement officers, search system managers, vendors, and financial analysts
• My own profiles of about 36 vendors of enterprise search systems plus the automated content files I generate using the Overflight system. A small scale version is available as a demo on ArnoldIT.com
• Information I had from my work as a systems engineering and technical advisor to several governments and their search system procurement teams
• My own experience licensing, testing, and evaluating search systems for clients. (I started doing this work after we created in 1993 The Point (Top 5% of the Internet) and sold it to Lycos, a unit of CMGI. I figured I should look into what Lycos was doing so I could speak with authority about its differences from BRS/Search, InQuire, Dialog (RECON), and IBM STAIRS III. I had familiarity with most of these systems through various projects in my pre Point (Top 5% of the Internet life).
• My Google research funded by the now-defunct BearStearns outfit and a couple of other well heeled organizations.

What was clear in 2006 was the following:

First, most of the search system vendors shared quite a bit of similarity. Despite the marketing baloney, the key differentiators among the flagship systems in 2006 were minor. Examples range from their basic architecture to their use of stemming to the methods of updating indexes. There were innovators, and I pointed out these companies in my talks and various writings, including the three editions of the Enterprise Search Report I wrote before I fell ill in February 2007 and quit doing that big encyclopedia type publication. These similarities made it very clear to me that innovation for enterprise search was shifting from the plain old key word indexing of structured records available since the advent of RECON and STAIRS to a more freeform approach with generally lousy relevance.

Get information access wrong, and some folks may find a new career. Source: http://www.seeing-stars.com/Images/ScenesFromMovies/AmericanBeautyMrSmiley%28BIG%29.JPG

Second, the more innovative vendors were making an effort in 2006 to take a document and provide some sort of context for it. Without a human indexer to assign a classification code to a document that is about marketing but does not contain the word “marketing”, this was rocket science. But when I examined these systems, there were two basic approaches which are still around today. The first was to use statistical methods to put documents together and make inferences and the other was a variation on human indexing but without humans doing most of the work. The idea was that a word list would contain synonyms. There were promising demonstrations of software methods that could “read” a document, but there were piggy and of use where money was no object.

Third, the Google approach which used social methods—that is, a human clicking on a link—were evident but not migrating to the enterprise world. Google was new but to make their 2006 method hum, lots of clicks were needed. In the enterprise, most documents never get clicked, so the 2006 Google method was truly lousy. Google has made improvements, mostly by implementing the older search methods, not by pushing the envelope as it has been doing with its Web search and dataspace efforts.

Fourth, most of the search vendors were trying like Dickens to get out of a “one size fits all” approach to enterprise search. Companies making sales were focusing on a specific niche or problem and selling a package of search and content searching that solved one problem. The failure of the boil the ocean approach was evident because user satisfaction data from my research funded by a government agency and other clients revealed that about two thirds of the users of an enterprise search system were dissatisfied or very dissatisfied with that search system. The solution, then, was to focus. My exemplary case was the use of the Endeca technology to allow Fidelity UK sales professionals to increase their productivity with content pushed to them using the Endeca system. The idea was that a broker could click on a link and the search results were displayed. No searching required. ClearForest got in the game by analyzing the dealer warranty repair comments. Endeca and ClearForest were harbingers of focus. ClearForest is owned by Thomson Reuters and in the open source software game too.

When I wrote the article in Online Magazine for Barbara Quint, one of my favorite editors, I explained these points in more detail. But it was clear that the financial pressures on Convera, for example, and the difficulty some of the more promising vendors like Entopia were having made the thin edge of survival glint in my desk lamp’s light. Autonomy by 2006 had shifted from search and organic growth to inorganic growth fueled by acquisitions that were adjacent to search.

Read more

Can Microsoft and Its Petascale Financial Services Mining Project Succeed

The goslings and I were chattering and quacking in Harrod’s Creek. One of our cousins was killed and eaten for an American holiday. What a way for our beloved friend and colleague to go: deep fried in an oil drum behind the River Creek Inn.

As we were recalling the best moments in Theodore the Turkey’s life, we discussed the likelihood of Microsoft’s petascale content mining project hitting a home run. The ideas, as we addled geese understand it, is that Microsoft wants to process lots of content and generate high value insights for the money crazed MBAs in the world’s leading financial institutions.

The project tackles a number of tough technical problems; for example, getting around the inherent latency in petascale systems, dealing with the traditional input output balkiness of Windows plumbing, and crunching enough data with sufficient accuracy to make the exercise worth the time of the financial client. You may find my earlier post germane.

Other outfits are in this game as well. Some are focused on the hardware / firmware / software side like Exegy. Others provide toolkits like Kapow Technologies. Some Beltway Bandits operate low profile content filtering systems for governmental and commercial clients. And there is the old nemesis, Googzilla, happily chewing through one trillion documents every few days. Finally, some of the financial institutions themselves have pumped support into outfits like Connotate. Even the struggling Time Warner owns some nifty technology in the Relegence unit. So, what’s new?

Three thoughts as I prepare to comment about the push into perceived real time processing at the International Online
Show:

1. The cost of slashing latency with any type of content is going to be one expensive proposition. Not even some governments have the cash to muscle up a serve with terabytes of RAM. Yep, terabytes.
2. Figuring out what process left another process gasping for air requires some programmers who can plow through code looking for an errant space, a undefined variable, or a bit of an Assembler hack that push when it should have popped
3. Latency outside the span of control of the system can render some outputs just plain wrong. Delay is bad; bad outputs are even worse.

If you have not been tracking Microsoft’s big initiatives, you may want to spend some more time grinding through the ACM and other scholarly papers such as “Towards Loosely Coupled Coupled Programming on Petascale Systems.  and poking around on the Microsoft Web site. To find useful stuff, I use the Google Microsoft index. If you aren’t familiar with it, check it out here.

I wonder if this stuff will be part of SharePoint 2011 and available as a Microsoft Fast ESP plug in?

Stephen Arnold, December 1, 2009

Yes, oh, yes. Let me disclose to the National Institute of Science and Technology that I was not paid to write this humorous essay. Consider it a name day present. If I am late, that’s latency. If I am early, that’s predictive output.

Microsoft and the Cloud Burger: Have It Your Way

I am in lovely and organized Washington, DC, courtesy of MarKLogic. The MarkLogic events pull hundreds of people, so I go where the action is. Some of the search experts are at a search centric show, but search is a bit yesterday in my opinion. There’s a different content processing future and I want to be prowling that busy boulevard, not sitting alone on a bench in the autumn of a market sector.

The MarkLogic folks wanted me to poke my nose into its user meeting. That was a good experience. And now I am cooling my heels for a Beltway Bandit client. I have my watch and my wallet. With peace of mind, I thought I would catch up on my newsreader goodies.

I read with some surprise “Windows Server’s Plan to Move Customers Back Off the Cloud” in beta news. As I understand the news story, Microsoft wants its customers to use the cloud, the Azure service. Then when fancy strikes, the customer can license on premises software and populate big, hot, expensive to maintain servers in the licensee’s own data center. I find the “have it your own way” appealing. I was under the impression that the future was the cloud. If I understand this write up, the cloud is not really the future. The “future” is the approach to computing that has been here since I took my first computer programming class in 1963 or so.

I found this passage in the article interesting:

If you write your code for Windows Server AppFabric, it should run on Windows Azure,” said Ottaway, referring to the new mix-and-match composite applications system for the IIS platform. “What we are delivering in 2010 is a CTP [community technology preview] of AppFabric, called Windows Azure AppFabric, where you should be able to take the exact same code that you wrote for Windows Server AppFabric, and with zero or minimal refactoring, be able to put it up on Windows Azure and run it.” AppFabric for now appears to include a methodology for customers to rapidly deploy applications and services based on common components. But for many of these components, there will be analogs between the on-Earth and off-Earth versions, if you will, such that all or part of these apps may be translated between locales as necessary.

Note the “shoulds”. Also, there’s a “may be”. Great. What does this “have it your own way” mean for enterprise search?

First, I don’t think that the Fast ESP system is going to be as adept as either Blossom, Exalead, or Google at indexing and serving results from the cloud for enterprise customers. The leader in this segment is not Google. I would give the nod to Blossom and Exalead. There’s no “should” with these systems. Both deliver.

Second, the latency for a hybrid application when processing content is going to be an interesting challenge for those brave enough to tackle the job. I recall some issues with other vendors’ hybrid systems. In fact, these performance problems were among the reasons that these vendors are not exactly thriving today. Sorry, I cannot mention names. Use your imagination or sift through the articles I have written about long gone vendors.

Third, Microsoft is working from established code bases and added layers—wrappers, in my opinion—to these chunks of code that exist. That’s an issue for me because weird stuff can happen. Yesterday one Internet service provider told me that his shop was sticking with SQL Server 2000. “We have it under control”, he said. With new layers of code, I am not convinced that those building a cloud and on premises solution using SharePoint 2010 and the “new” Fast ESP search system are going to have stress free days.

In short, more Microsoft marketing messages sound like IBM’s marketing messages. Come to think of it hamburger chains have a similar problem. I think this play is jargon for finding ways to maximize revenues, not efficiencies for customers. When I go to a fast food chain, no matter what I order, the stuff tastes the same and delivers the same health benefits. And there’s a “solution accelerator.” I will have pickles with that. Just my opinion.

Stephen Arnold, November 19, 2009

I hereby disclose to the Internal Revenue Service and the Food and Drug Administration that this missive was written whilst waiting for a client to summon me to talk about topics unrelated to this post. This means that the write up is a gift. Report it as such on your tax report and watch your diet.

Analyst Underestimates Impact of Microsoft and Google in Enterprise Search

I have written a report for the Gilbane Group and think highly of the firm’s analysis. However, a recent news item forwarded to me by a reader of this Web log triggered some conversation in Harrod’s Creek today. The article was “Competition among Search Vendors,” and it was dated in early August 2009. The article included this assertion:

This additional news item indicates that Microsoft is still trying to get their search strategy straightened out with another new acquisition, Applied Discovery Selects Microsoft FAST for Advanced E-Discovery Document Search. E-discovery is a hot market in legal, life sciences and financial verticals but firms like ISYS, Recommind, Temis, and ZyLab are already doing well in that arena. It will take a lot of effort to displace those leaders, even if Microsoft is the contender. Enterprises are looking for point solutions to business problems, not just large vendors with a boatload of poorly differentiated products.

In the last three months, I have noticed an increase in the marketing activity from quite a few enterprise search and content processing vendors. One one hand, there are SharePoint surfers. These are vendors who don’t want to be an enterprise solution. The vendors are content to license their technology to some of the 100 million SharePoint licensees. Forget search. SharePoint is a market that can be harvested.

On the other hand, there are vendors in the process of changing their stripes, and not just once. Some companies have moved from intelligence applications to marketing to sentiment to call center applications. I have to tell you. I don’t know if some of these companies are even in the search business any more.

Looming over the entire sector are the shadows of Google and Microsoft. Each has a different strategy for sucking revenue blood from the other. Smaller vendors are going to need agility to avoid getting hurt when these two semi-clueless giants rampage across the enterprise to do battle to one another.

The notion that niches will support the more than 300 companies in the search and content processing market may be optimistic. I wanted to use a screen shot from the TV show Fantasy Island but I won’t. A number of search vendors are gasping for oxygen now. I am keen to keep a sunny outlook on life. But when it comes to search and content processing realism is useful. Just ask some of the musical chair executives making the rounds among the search and content processing companies.

What will Google and Microsoft do to get shelf space in the enterprise? What do big companies with deep pockets typically do? Smaller companies can’t stay in this winner take all game.

Stephen Arnold, October 2, 2009

Exclusive Interview with SurfRay President

SurfRay has come roaring back into the search and content processing sector. SurfRay, like many other companies, had to tighten its belt and straighten its tie in the wake of the global financial turmoil. With the release of new versions of Ontolica (a search system that snaps into SharePoint) and MondoSearch (a platform independent Web site search solution), SurfRay is making sales again. ArnoldIT.com spoke with Søren Pallesen about the firm’s new products. In a wide ranging interview, Mr. Pallesen, a former Gartner Group consultant, said:

SurfRay’s mission is to deliver tightly packaged search software solutions for businesses to provide effective search for both internal and external users. With Packaged we mean easy to try, install and use. Our vision is to be our customer’s first choice for packaged enterprise search solutions and to become the world’s third largest search solution provider in the world measured on number of paying business customers by 2012. The last six months have been an exciting time for SurfRay. I took over as CEO; we significantly increased investment in product development and an ambitious expansion of the organization. This has paid off. SurfRay is profitable, and we have released new versions of our core products. Ontolica is now in version 4.0, including a new suite of reporting and analytics, and MondoSearch 5.4 is in beta for a Q4 release. As a profitable company we are in the fortunate position to be able to fund our own growth and we are expanding in North America among other by hiring more sales people as well as formation of a Search Expert Center in Vancouver, Canada that will serve customers across the Americas. We are also expanding in Europe most recently with formation of SurfRay UK and Ireland, allowing us to expand sales and support with local people on the ground in this important European market.

When asked about the difference between MondoSearch and Ontolica, Mr. Pallesen told me:

Customers that buy our products typically fall into a number of usage scenarios. Simply put Ontolica solves search problems inside the firewall and MondoSearch outside the firewall. Firstly customers with SharePoint implementations look for enhanced search functionality, and turn to our Ontolica for SharePoint product. Secondly, businesses that do not use SharePoint but have the need for an internal search solution on an intranet, file servers, across email, applications and other sources buy Ontolica Express and use it in combination with Microsoft Search Server Express for simple single server installation or Micro Search Server for multiple load balanced server installations. Thirdly, customers with the need for robust and highly configurable web site search buy MondoSearch. Especially popular with businesses that want to implement up- and cross selling on their search results page.

You can read the full text of the interview in the Search Wizards Speak series on ArnoldIT.com. For more information about SurfRay, visit the company’s revamped Web site at http://www.surfray.com.

Stephen Arnold, September 29, 2009

Explaining the Difference between Fast ESP and MOSS 2007 Again

When a company offers multiple software products to perform a similar function, I get confused. For example, I have a difficult time explaining to my 88 year old father the differences among Notepad, WordPad, Microsoft Works’ word processing, Microsoft Word word processing, and the Microsoft Live Writer he watched me use to create this Web log post. I think it is an approach like the one the genius at Ragu spaghetti sauce used to boost sales of that condiment. When my wife sends me to the store to get a jar of Ragu spaghetti sauce, I have to invest many minutes figuring out what the heck is the one I need. Am I the only male who cannot differentiate between Sweet Tomato Basic and Margherita? I think Microsoft has taken a different angle of attack because when I acquired a Toshiba netbook, the machine had installed Notepad, WordPad, and Microsoft Works. I added a version of Office and also the Live Writer blog tool. Some of these were “free” and others products came with my MSDN subscription.

Now the same problem has surfaced with basic search. I read “FAST ESP versus MOSS 2007 / Microsoft Search Server” with interest. Frankly I could not recall if I had read this material before, but quit a bit seemed repetitive. I suppose when trying to explain the differences among word processors, the listener hears a lot of redundant information as well.

The write up begins:

It took me some time but i figured out some differences between Microsoft Search Server / MOSS 2007 and Microsoft FAST ESP. These differences are not coming from Microsoft or the FAST company. But it came to my notice that Microsoft and FAST will announce a complete and correct list with these differences between the two products at the conference in Las Vegas next week.These differences will help me and you to make the right decisions at our customers for implementing search and are based on business requirements.

Ah, what’s different is that this is a preview of the “real” list of differences. Given the fact that the search systems available for SharePoint choke and gasp when the magic number of 50 million documents is reached, I hope that the Fast ESP system can handle the volume of information objects that many organizations have on their systems at this time.

The list in the Bloggix post numbers 14. Three interested me:

1. Scalability
2. Faceted navigation
3. Advanced federation.

Several observations:

First, scalability is an issue with most search systems. Some companies have made significant technical breakthroughs to make adding gizmos painless and reasonably economical. Other companies have made the process expensive, time consuming, and impossible for the average IT manager to perform. I heard about EMC’s purchase of Kazeon. I thought I heard that someone familiar with the matter pointed to problems with the Fast ESP architecture as one challenge for EMC. In order to address the issue, EMC bought Kazeon. I hope the words about “scalability” are backed up with the plumbing required to deliver. Scaling search is a tough problem, and throwing hardware at hot spots is, at best, a very costly dab of Neosporin.

Second, faceted navigation exists within existing MOSS implementations. I think I included screenshots of faceted navigation in the last edition of the Enterprise Search Report I wrote in 2006 and 2007. There was a blue interface and a green interface. Both of these made it possible to slice and dice results by clicking on an “expert” identified by counting the number of documents a person wrote with a certain word in them. There were other facets available as well, although most we more sophisticated that the “expert” function. I hope that the “new” Fast ESP implements a more useful approach for users of Fast ESP. Of course, identifying, tagging, and linking facets across processed content requires appropriate computing resources. That brings us back to scaling, doesn’t it? Sorry.

Third, federation is a buzz word that means many different things because vendors define the term in quite distinctive ways. For example, Vivisimo federates, and it is  or was at one time a metasearch system. The query went to different indexing services, brought back the results, deduplicated them, put the results in folders on the fly, and generated a results list. Another type of federation surfaces in the descriptions of business intelligence systems offered by SAS. The system blends structured and unstructured data within the SAP “environment”. Others are floating around as well, including the repository solutions from TeraText which federates disparate content into one XML repository. What I find interesting is that Microsoft is not delivering “federation” which is undefined. Microsoft is, according to the Bloggix post, on the trail of “advanced federation”. What the heck does that mean. The explanation is:

FAST ESP supports advanced federation including sending queries to various web search APIs, mixing results, and shallow navigation. MOSS only supports federation without mixing of results from different sources and navigation components, but showing them separately.

Okay, Vivisimo and SAP style for Fast ESP; basic tagging for MOSS. Hmm.

To close, I think that the Fast ESP product is going to add a dose of complexity to the SharePoint environment. Despite Google’s clumsy marketing, the Google Search Appliance continues to gain traction in many organizations. Google’s solution is not cheap. People want it. I think Fast ESP is going to find itself in a tough battle for three reasons:

1. Google is a hot brand, even within SharePoint shops
2. Microsoft certified search solutions are better than Fast ESP based on my testing of search systems over the past decade
3. The cost savings pitch is only going to go so far. CFOs eventually will see the bills for staff time, consulting services, upgrades, and search related scaling. In a lousy financial environment, money will be a weak point.

I look forward to the official announcement about Fast ESP, the $1.2 billion Microsoft spent for this company is now going to have to deliver. I find it unfortunate that the police investigation of alleged impropriety at Fast Search & Transfer has not been resolved. If a product is so good as Fast ESP was advertised to be, what went wrong with the company, its technology, and its customer relations prior to the Microsoft buy out? I guess I have to wait for more information on these matters. When you have a lot of different products with overlapping and similar services, the message I get is more like the Ragu marketing model, not the solving of customer problems in a clear, straightforward way. Sigh. Marketing, not technology, fuels enterprise search these days I fear.

Stephen Arnold, September 7, 2009

Microsoft Fast for Portals

Author’s Note: The images in this Web log post are the property of Microsoft Corp. I am capturing my opinion based on a client’s request to provide feedback about “going with Fast for SharePoint” versus a third party solution from a Microsoft Certified Partner. If you want happy thoughts about Microsoft, Fast ESP, and search in SharePoint environments, look elsewhere. If you want my opinions, read on. Your mileage may vary. If you have questions about how the addled goose approaches these write ups, check out the editorial policy here.

Introduction

Portals are back. The idea is that a browser provides a “door” to information and applications is hot again. I think. You can view a video called “FAST: Building Search Driven Portals with Microsoft Office SharePoint Server 2007 and Microsoft Silverlight” to get the full story. I went back through my SharePoint search links. I focused on a presentation given in 2008 by Two Microsoft Fast engineers–Jan Helge Sageflåt and Stein Danielsen.

After watching the presentation for a second time, I formed several impressions of what seems to be the general thrust of the Microsoft Fast ESP search system. I have heard reports that Microsoft is doing a full court press to get Microsoft-centric organizations to use Fast ESP as the industrial strength search system.

Let me make several observations about the presentation by the Microsoft Fast engineers and then conclude with a suggestion that caution and prudence may be fine dinner companions before one feasts on Fast ESP. Portals are not a substitute for making it easy for employees to locate the item of information needed to answer a run-of-the-mill business information need.

Observations about the 2008 Demo

First, the presentation focuses on building interfaces and making connections to content in SharePoint. Most organizations want to connect to the content scattered on servers, file systems, and enterprise application software data stores. That is job one or it was until the financial meltdown. Now organizations want to acquire, merge, search, and tap into social content. Much of that information has a short shelf life. The 2008 presentation did not provide me with evidence that the Microsoft Fast ESP system could:

• Acquire large flows of non-SharePoint content
• Process that information without significant latency
• Identify the plumbing needed to handle flows of real time content from RSS feeds and the new / updated content from a SharePoint system.

Read more

Microsoft Embraces Scale

The year was 2002. A cash-rich, confused outfit paid me to write a report about Google’s database technology. In 2002, Google was a Web search company with some good buzz among the alleged wizards of Web search. Google did not have much to say when its executives gave talks. I recall an exchange between me and Larry Page at the Boston Search Engine Meeting in 1999. The topic? Truncation. Now that has real sizzle among the average Web surfer. I referenced an outfit called InQuire, which supported forward truncation. Mr. Page asserted that Google did not have to fool around with truncation. The arguments bored even those who were search experts at the Boston meeting.

I realized then that Google had some very specific methods, and those methods were not influenced by the received wisdom of search as practiced at Inktomi or Lycos, to name two big players in 2000. So I began my research looking for differences between what Google engineers were revealing in their research papers. I compiled a list of differences. I won’t reference my Google studies, because in today’s economic climate, few people are buying $400 studies of Google or much else for that matter.

I flipped through some of the archives I have on one of my back up devices. I did a search for the word “scale”, and I found that it was used frequently by Google engineers and also by Google managers. Scale was a big deal to Google from the days of BackRub, according to my notes. BackRub did not scale. Google, scion of BackRub, was engineered to scale.

The reason, evident to Messrs. Brin and Page in 1998, was that the problem with existing Web search systems was that the operators ran out of money for exotic hardware needed to keep pace with the two rapidly generating cells of search: traffic and new / changed content. The stroke of genius, as I have documented in my Google studies, was that Google tackled the engineering bottlenecks. Other search companies such as Lycos lived with the input output issues, the bottlenecks of hitting the disc for search results, and updating indexes by brute force methods. Not the Google.

Messrs. Brin and Page hired smart men and women whose job was “find a solution”. So engineers from Alta Vista, Bell Labs, Sun Microsystems, and other places where bright folks get jobs worked to solve these inherent problems. Without solutions, there was zero chance that Google could avoid the fate of the Excites, OpenText Web index, and dozens of other companies without a way to grow without consuming the equivalent of the gross domestic product for hardware, disc space, bandwidth, chillers, and network devices.

Google’s brilliance (yes, brilliance) was to resolve in a cost effective way the technical problems that were deal breakers for other search vendors. AltaVista was a pretty good search system but it was too costly to operate. When the Alpha computers were online, you could melt iron ore, so the air condition bill was a killer.

Keep in mind that Google has been working on resolving bottlenecks and plumbing problems for more than 11 years.

I read “Microsoft’s Point Man on Search—Satya Nadella—Speaks: It’s a Game of Scale” and I shook my head in disbelief. Google operates at scale, but scale is a consequence of Google’s solutions to getting results without choking a system with unnecessary disc reads. Scale is a consequence of using dirt cheap hardware that is mostly controlled by smart software interacting with the operating system and the demands users and processes make on the system. Scale is a consequence of figuring out how to get heat out of a rack of servers and replacing conventional uninterruptable power supplies with on motherboard batteries from Walgreen’s to reduce electrical demand, heat and cost. Scale comes from creating certain propriety bits of hardware AND software to squeeze efficiencies out of problems caused by physics of computer operation.

If you navigate to Google and poke around you will discover “Publications by Googlers”. I suggest that anyone interested in Google browse this list of publications. I have tried to read every Google paper, but as I age, I find I cannot keep up. The Googlers have increased their output of research into plumbing and other search arcana by a factor of 10 since I first began following Google’s technical innovations. Here’s one example to give you some context for my comments about Mr. Nadella’s comments, reported by All Things Digital; to wit: “Thwarting Virtual Bottlenecks in Multi-Bitrate Streaming Servers” by Bin Liu and Raju Rangaswami (academics_) and Zora Dimitrijevic (Googler). Yep, there it is in plain English—an innovation plus hard data that shows that Google’s system anticipates bottlenecks. Software makes decisions to avoid these “virtual bottlenecks.” Nice, right? The bottleneck imposed by the way computers operate and the laws of physics are identified BEFORE they take place. The Google system then changes its methods in order to eliminate the bottleneck. Think about that the next time you wait for Oracle to respond to a query across a terabyte set of data tables or you wait as SharePoint labors to generate a new index update. Google’s innovation is predictive analysis and automated intervention. This explains why it is sometimes difficult to explain why a particular Web page declined in a Google set of relevance ranked results. The system, not humans, is adapting.

I understand the frustration that many Google pundits, haters, and apologists express to me. But if you take the time to read Google’s public statements about what it is doing and how it engineers its systems, the Google is quite forthcoming. The problem, as I see it, has two parts. First, Googlers write for those who understand the world as Google does. Notice the language of the “Thwarting” paper. Have you thought about Multi-bitrate streaming servers in a YouTube.com type of environment. YouTube.com has lots of users, and streams a lot of content. The problems are that Google’s notion of clarity is show in the statement below:

Second, very few people in the search business deal with the user loads that Google experiences. Looking up the location of one video and copying it from one computer to another is trivial. Delivering videos to a couple of million people at the same time is a different class of problem. So, why read the “Thwarting” paper? The situation described does not exist for most search companies or streaming media companies. The condition at Google is, by definition, an anomaly. Anomalies are not what make most information technology companies hearts go pitter patter more quickly. Google has to solve these problems or it is not Google. A company that is not Google does not have these Google problems. Therefore, Google solves problems that are irrelevant to 99 percent of the companies in the content processing game.

Back to Mr. Nadella. This comment sums up what I call the Microsoft Yahoo search challenge:

Nadella does acknowledge in the video interview here that Microsoft has has not been able to catch up with Google and talks about how that might now be possible.

I love the “might”. The thoughts the went through my mind when I worked through this multi media article from All Things Digital were:

1. Microsoft had access to similar thinking about scale in 1999. Microsoft hired former AltaVista engineers, but the Microsoft approach to data centers is a bit like the US Navy’s approach to aircraft carriers. More new stuff has been put on a design that has remained unchanged for a long time. I have written about Microsoft’s “as is” architecture in the Web log with snapshots of the approach at three points in time
2. Google has been unchallenged in search for 11 years. Google has an “as is” infrastructure capable of supporting more than 2,200 queries per second as well as handling the other modest tasks such as YouTube.com, advertising, maps, and enterprise applications. In 2002, Google had not figured out how to handle high load reads and writes because Google focused on eliminating disc reads and gating writes. Google solved that problem years ago.
3. Microsoft has to integrate the Yahoo craziness into the Microsoft “as is”, aircraft carrier approach to data centers. The affection for Microsoft server products is strong, but adapting to Yahoo search innovations will require some expensive, time consuming, and costly engineering.

In short, I am delighted that Mr. Nadella has embraced scale. Google is becoming more like a tortoise, but I think there was a fable about the race between the tortoise and the hare. Google’s reflexes are slowing. The company has a truck load of legal problems. New competitors like Collecta.com are running circles around Googzilla. Nevertheless, Microsoft has to figure out the Google problem before the “going Google” campaign bleeds revenue and profits from Microsoft’s pivotal business segments.

My hunch is that Microsoft will run out of cash before dealing the GOOG a disabling blow.

Stephen Arnold, August 4, 2009

Rethinking the Microsoft Corp Search Cost Burden

I am waiting to give a talk. I have been flipping through hard copy newspapers and clicking around to see what is happening as I cool my heels. Three items caught may attention. Both the New York Times and the Wall Street Journal reported that the Yahoo deal is good for Yahoo. Maybe? What I think is that Microsoft assumed the cost burden of Yahoo’s search operation. Since my analysis of Yahoo’s search costs in 2006, I have gently reminded folks that Yahoo had a growing cost problem and its various management teams could not do much about these costs. So Yahoo focused on other matters and few people  brought the focus back on infrastructure, staff, and duplicative search systems.

Now Microsoft has assumed this burden.

I scanned John Gruber’s “Microsoft’s Long, Slow Decline”, and I noted this comment:

Microsoft remains a very profitable company. However, they have never before reported year-over-year declines like this, nor fallen so short of projected earnings. Something is awry.

Dead on. What is missing is thinking about the challenge Microsoft has in search. My thoughts are:

First, Microsoft has to make headway with its $1.2 billion investment in enterprise search. I think the uptake of SharePoint will produce some slam dunk sales. But competitors in the enterprise search sector know that SharePoint is big and fuzzy, and many Microsoft centric companies have here and now problems. I think there is a real possibility for Microsoft to cut the price of Fast ESP or just give it away if the client buys enough CALs for other Microsoft products. What I wonder is, “How will Microsoft deal with the punishing engineering costs a complex content processing and search system imposes during its first six to 18 months in a licensee’s organization. Microsoft partners may know SharePoint but I don’t think many know Fast ESP. Then there is the R&D cost to keep pace with competitors in search, content processing, and the broader field of findability. Toss in business intelligence and you have a heck of a cost anchor.

Second, Bing.com may get access to Yahoo’s Ford F 150 filled with software. But integrating Yahoo technology with Microsoft technology is going to be expensive. There are other costs as well; for example, Microsoft bought Powerset and some legacy technology from Xerox Parc. Layer on the need for backward compatibility and you have another series of cost black holes.

Finally, there are the many different search technologies that Microsoft has deployed and must presumably rationalize. Fast ESP has a better SQL query method than SQL Server. Will customers get both SQL and Fast ESP or will there be more product variants. Regardless of the path forward, there are increased costs.

Now back to Mr. Gruber’s point: a long, slow decline requires innovation and marketing of the highest order. I think the cost burden imposed by search will be difficult for Microsoft to control. Furthermore, I hypothesize:

• Google will become more aggressive in the enterprise sector. Inflicting several dozen wounds may be enough to addle Microsoft and erode its profitability in one or two core product lines
• Google’s share of the Web search market may erode but not overnight’. The Googlers won’t stand still and the deal with Yahoo strikes me as chasing the Google of 2006, not the Google of 2009.
• Of the more than 200 competitors in enterprise search and content processing, I am confident that a half dozen or more will find ways to suck cash from Microsoft’s key accounts because increasingly people want solutions, not erector sets.

In short, Microsoft’s cost burdens come at a difficult time for the company. Microsoft and Yahoo managers have their work cut out for them.

Stephen Arnold, July 31, 2009

« Previous Page — Next Page »

• 

• Search the site

•  

  Subscribe to Beyond Search

  • 
  • 
   
  • 
   
  
  Feature archive
  News archive
   
  
  
  Stephen E. Arnold monitors search, content processing, text mining and related topics from his high-tech nerve center in rural Kentucky. He tries to winnow the goose feathers from the giblets. He works with colleagues worldwide to make this Web log useful to those who want to go "beyond search". Contact him at sa [at] arnoldit.com. His Web site with additional information about search is arnoldit.com.

• Categories

  • 3D-Printing
  • Acquisition
  • Advertising
  • Aggregation
  • AI
  • Alexa
  • algorithms
  • Amazon
  • Amazonia
  • Analytics
  • Appliance
  • Applications
  • Audio
  • Augmented Reality
  • Big data
  • Bing
  • Bitcoin
  • Bitext
  • Book review
  • Business intelligence
  • Business process
  • Business strategy
  • Censorship
  • Cloud computing
  • Company Profile
  • Conferences
  • Connectors
  • Consulting
  • Consumer
  • Content processing
  • Copyright
  • Corporate Concerns
  • Cost
  • Crawl
  • Crowdfunding
  • cryptocurrency
  • Customer support
  • Cyber OSINT
  • cybercrime
  • cybersecurity
  • Dark Web
  • DarkCyber
  • Data
  • Data mining
  • Database
  • Deepfakes
  • Digital Assistant
  • Digital Library
  • E2EE
  • ECommerce
  • EDiscovery
  • Editorial opinion
  • Education
  • Emoticons
  • Enterprise
  • Enterprise search
  • Entity extraction
  • Ethics
  • Facebook
  • Faceted search
  • Factualities
  • Feature
  • Federated search
  • Financial
  • Google
  • Governance
  • Government
  • Hackers
  • healthcare
  • IBM Watson
  • Image search
  • Indexing
  • Infrastructure
  • Innovation
  • Integration
  • intelware
  • Interface
  • Internet
  • Interview
  • Investment
  • law enforcement
  • Legal matters
  • Library automation
  • Management
  • Marketing
  • Mathematics
  • Metadata
  • Microsoft
  • Mobile
  • Natural language processing
  • News
  • NGIA
  • Online (general)
  • Open Access
  • Open source
  • OSINT
  • Osint Radar
  • Overflight
  • Palantir
  • Patents
  • Personnel
  • Podcast
  • Policeware
  • Portals
  • Predictive coding
  • Privacy
  • Profile
  • Publishing
  • Quotation
  • Real time search
  • Reference tool
  • Rich media
  • Robot Writer
  • Search
  • Search enabled applications
  • search engine
  • Search quality
  • Security
  • Semantic
  • Sentiment analysis
  • SEO
  • SharePoint
  • Short Honks
  • Smart Technology
  • Social
  • Social Media
  • software
  • Statistics
  • Taxonomy
  • Technology
  • Text analytics
  • Text processing
  • Tools
  • Tor
  • Training
  • Translation
  • Twitter
  • Uncategorized
  • Unstructured Data
  • User experience
  • User Interface
  • Vertical search
  • Video
  • visualization
  • Voice search
  • Voice technology
  • Web 3
  • Web Services
  • Webinar
  • Windows
  • Work flow
  • XML
  • Yahoo

• Archives

  Archives Select Month August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024 December 2023 November 2023 October 2023 September 2023 August 2023 July 2023 June 2023 May 2023 April 2023 March 2023 February 2023 January 2023 December 2022 November 2022 October 2022 September 2022 August 2022 July 2022 June 2022 May 2022 April 2022 March 2022 February 2022 January 2022 December 2021 November 2021 October 2021 September 2021 August 2021 July 2021 June 2021 May 2021 April 2021 March 2021 February 2021 January 2021 December 2020 November 2020 October 2020 September 2020 August 2020 July 2020 June 2020 May 2020 April 2020 March 2020 February 2020 January 2020 December 2019 November 2019 October 2019 September 2019 August 2019 July 2019 June 2019 May 2019 April 2019 March 2019 February 2019 January 2019 December 2018 November 2018 October 2018 September 2018 August 2018 July 2018 June 2018 May 2018 April 2018 March 2018 February 2018 January 2018 December 2017 November 2017 October 2017 September 2017 August 2017 July 2017 June 2017 May 2017 April 2017 March 2017 February 2017 January 2017 December 2016 November 2016 October 2016 September 2016 August 2016 July 2016 June 2016 May 2016 April 2016 March 2016 February 2016 January 2016 December 2015 November 2015 October 2015 September 2015 August 2015 July 2015 June 2015 May 2015 April 2015 March 2015 February 2015 January 2015 December 2014 November 2014 October 2014 September 2014 August 2014 July 2014 June 2014 May 2014 April 2014 March 2014 February 2014 January 2014 December 2013 November 2013 October 2013 September 2013 August 2013 July 2013 June 2013 May 2013 April 2013 March 2013 February 2013 January 2013 December 2012 November 2012 October 2012 September 2012 August 2012 July 2012 June 2012 May 2012 April 2012 March 2012 February 2012 January 2012 December 2011 November 2011 October 2011 September 2011 August 2011 July 2011 June 2011 May 2011 April 2011 March 2011 February 2011 January 2011 December 2010 November 2010 October 2010 September 2010 August 2010 July 2010 June 2010 May 2010 April 2010 March 2010 February 2010 January 2010 December 2009 November 2009 October 2009 September 2009 August 2009 July 2009 June 2009 May 2009 April 2009 March 2009 February 2009 January 2009 December 2008 November 2008 October 2008 September 2008 August 2008 July 2008 June 2008 May 2008 April 2008 March 2008 February 2008 January 2008 November 5

• Recent Posts

  • An Ed Critique That Pans the Sundar & Prabhakar Comedy Act
  • A Familiar Cycle: The Frustration of Almost Solving the Search Problem
  • Deep Fake Service?
  • Pragmatic AI: Individualized Monitoring
  • Canadians Unhappy about Tax on Streaming Video

• Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org