SchemaLogic and Its MetaPoint
November 3, 2009
At the SharePoint conference, SchemaLogic announced its MetaPoint software. According to the company:
Multipoint integrates with Microsoft Office to tag and classify documents automatically when they are created and suggests to the user where documents should be stored on the Microsoft Office SharePoint Server. Multipoint helps employees find and share information more effectively while improving corporate information governance and regulatory compliance.
I have been notified about a number of products and systems that add functionality to SharePoint. I am having some fun trying to figure out which features and functions are the distinguishing ones. SchemaLogic offers its metadata repository to organizations eager to bring consistency to metadata across different enterprise software systems. The Metapoint service offers a similar functionality for SharePoint 10.
I am not able to endorse any particular SharePoint 10 metadata management system at this time. This is on our to do list. In the meantime, procurement teams will have the opportunity to install, test, and evaluate these systems. Exciting and time consuming. SharePoint 10 itself is a ton of fun and integrating a third party metadata system will be an outstanding learning experience in my opinion.
Stephen Arnold, November 3, 2009
Notice to the Department of Commerce: no one paid me to write this article explaining its fun factor.
Pingar for Personalized Search
November 2, 2009
A happy quack to the reader who alerted me to Pingar, a SharePoint centric content processing vendor with offices in New Zealand. The company, founded in 2006, announced its enterprise search solution for SharePoint in mid October 2009. According to the company:
“The intelligent enterprise search tool that can be embedded into the upcoming release of Microsoft SharePoint Server 2010 will take the browsing out of browsing,” says Pingar’s co-founder and Managing Director Peter Wren-Hilton.
I stumbled on the notion of taking “the browsing out of browsing”. I use a browser to browse. If don’t want to browse, I use another method. Nevertheless, the system, according to the company:
[the] solution goes inside data documents, finds the content the user is seeking and then places it into a dynamically generated PDF or XPS document, rather than just presenting a list of links like the traditional search model. Pingar’s solution also sorts the search into categories to minimize reading times.
Pingar.com
In a Pingar report, a hit includes a back link to the original source document.
According to the Tauranga Eastern Link Newsletter:
Pingar has developed dynamic software to create an ‘intelligent’ search engine, which enables users to type in a specific question and get an exact answer. Pingar’s new offices will be in Hong Kong’s prestigious Science & Innovation Park, close to one of its key partners as well as a range of potential customers in China and Asia. Sharon-May McCrostie, New Zealand’s Trade Commissioner to Hong Kong, endorses Pingar’s move into the Chinese market. “Pingar has a revolutionary, clever way of search that will transform so many industry sectors, including publishing and monetizing online content. It has been very exciting for us to see a New Zealand company taking on the world and making real inroads…
Fast to Integrate with SharePoint
November 2, 2009
Overflight’s SharePoint search container has been an empty can for weeks. Today I saw a link to the Microsoft Enterprise Search Blog’s “Fast Meets SharePoint – What’s Coming in Search for SharePoint 2010”. Microsoft has owned Fast Search & Technology since April 2008. Since that time, there’s been a Web part and lots of speculation. The “legacy” Fast ESP customers seem to have the impression that Microsoft will support those systems for years. I heard that the commitment was for a decade. The mystery has been the union of Fast ESP with SharePoint. In my opinion, both software systems are bundles of subsystems, and the complexities of each separate product are familiar to me and my goslings here in Harrod’s Creek, Kentucky.
The October 28, 2009, write up contains a number of interesting points. Let me run down those that struck me as significant to my work. Your mileage may vary because the points in the Microsoft blog post are a feature run down, not substantive information about the “new” Fast ESP for SharePoint.
First, the SharePoint conference was sold out. This reminds me that training people to use Microsoft products is a big business. Obviously this is an important point because it focuses on what matters—conference attendance—not search it seems to me.
Second, there was a flashback to a conference held in February 2009. That tells me that there’s not much new in the way of SharePoint / Fast ESP news.
Third, there is a reminder to me that SharePoint is a work in progress. I think someone told me that it is the next operating system from Microsoft. The Fast component will be called Fast Search Server 2010 for SharePoint. Now the story gets interesting. Here’s what’s coming:
- A content processing pipeline. Most enterprise search systems use intake, content processing, query processing, and administrative controls. Frankly I don’t know how this “pipeline” will differ from other search systems crafted in the late 1990s, when the core of Fast Search was built.
- Metadata extraction. The idea of identifying concepts to help a user find documents or other content objects “about” a topic is not new. The hitch in the existing systems is that metadata extraction imposes a performance hit on a system. Some metadata system do “deep extraction” which takes significant computational time. Some vendors use the notion of metadata and facets interchangeably. I will be interested to see if Fast ESP will bring the product into line with the systems available from such companies as Coveo, Exalead, and Ontolica, among others. (Some of the terms used to describe the approach remind me of Endeca’s current approach to extending its system.)
- Structured data search. I find the notion of merging data from database tables or third party applications which use an RDBMS to house data and unstructured text interesting. Attivio, Clarabridge, and other firms are working in this niche now. There are significant challenges related to data transformation. In fact, data transformation can chew up significant resources * before * the search system can begin intake and processing.
- Visual search. This is the Bing 3D interface. I am not sure how that will fly because it does not seem to be a component of text processing which remains the thorn in the side of the SharePoint user.
- Advanced linguistics. My recollection is that Fast Search & Transfer had a shop in Germany that worked on certain linguistic functions. Some linguistic manipulations require set up and testing. Out of the box linguistic functions are available from some vendors like Basis Tech, but there is a lot of work that must be done to get these systems working so that the outputs match the needs of some users.
- Best bets. This is a variation of what I think of as Google’s “I’m feeling lucky”. These best bets work when there are sufficient data to make the recommendations useful. If the Fast ESP system uses a simple metric such as the number of documents “about” a topic written by a SharePoint user, I don’t think the best bet will be particularly helpful. More sophisticated methods require big data to generate useful results. Sophisticated methods operating of too small a data flow return bad bets in my experience.
- Development platform. The original Fast ESP system was coded in a range of languages. Each time an issue was discovered in the installation of Fast on my watch, new scripts had to be written and inserted into the Fast ESP system. My recollection is that Fast was not a homogeneous system. My guess is that the development system will be Microsoft’s own programming tools which include its scripting language and the VisualStudio line of products. It’s easy to talk about a development platform, but the reality of complex systems like Fast ESP may require considerable creativity to achieve certain objectives.
- Customization. User profiles have been around a long time. The SharePoint version of Fast will make use of available flags as well as information about a user or the groups to which a user belongs in order to present a Google “ig” type of interface. (“IG” means “individualized Google”, which is a newer version of the MyYahoo feature.)
The last point in the write up is the one that took my breath away. Earlier today I wrote about Autonomy’s assertion about “infinite scalability”. Now I read “extreme scale and performance” for the SharePoint / Fast search system. Keep in mind that “extreme” implies some non standard behavior. With Microsoft’s up and out approach to scaling, this phrase “extreme scale and performance” mean that lots and lots of hardware are going to be needed to handle large content flows. In fact only a handful of systems we have tested this year can deal with petascale content flows. I want to mention the little known Perfect Search as one example of a company that has nailed big data with a very modest hardware footprint. I will have to test the new Fast system before I can accept this “extreme” assertion.
Three thoughts crossed my mind as I worked through this SharePoint / Fast blog post from Microsoft:
First, why wasn’t the Fast ESP for SharePoint rolled out at the SharePoint conference? A delay suggests that something was not in sync. I wondered, “Is this not an interesting business strategy for a search vendor to implement?”
Second, these Microsoft blog posts recycle the same old information without adding any substantive new data. Where’s the block diagram? Where’s the sample / default interfaces? Where’s the list of methods?
Third, I keep thinking about the interactions among two complex systems. Who is going to have the time, money, patience, and management support to get these beasties to cuddle in a sleeping bag? With an alleged 100 million SharePoint licenses, those Microsoft Certified Professionals will be eager to give the work a try.
Billing for consulting services ahead for some search experts!
Stephen Arnold, November 2, 2009
Microsoft routed a question to me last week, but no money was forthcoming. After this article, I am a gone goose.
SharePoint and Its Origin
October 25, 2009
One of the articles I set aside to review when I was in Beltway Land last week was “Meet the Father of Microsoft Share Point: Jeff Teper”. I have an interest in SharePoint. The US government finds SharePoint a Swiss Army knife of possibilities. Any information that helps me understand where this product’s roots are anchored is of interest to me. The first thing I noticed about the article was the eyes. You can click here and draw your own conclusion. I see in these eyes a certain intensity. Remarkable. The second impact on me hit me when I read:
Bringing the idea for SharePoint to Gates and Ballmer resulted in two different takes from two different high-level Microsoft managers, Teper reminisced. He said Gates asked a lot of questions about the long-term architecture (SQL Server, .Net, etc.) behind what evolved into SharePoint. Gates also asked a lot of usability questions, Teper said. Ballmer, on the other hand, used his classic “I don’t get this” line of questioning to bring SharePoint’s charter into focus. “Ballmer said we need to make it simple, simple, simple,” Teper said. “He wanted to keep the message very simple.” So how did all this talk about simplicity yield a product that even Teper himself acknowledges is quite ambitious and complex? (He called it this week the “ultimate Swiss Army Knife.”)
SharePoint was to be simple. SharePoint consists of six servers and is, in my opinion, more complex that most enterprise applications. I think the story of SharePoint’s origins provides some insight into how large companies permit products to evolve. The focus is not upon better; the focus is upon more. The idea that problems can be ameliorated by adding additional features and functions is the DNA of SharePoint.
The subhead “A Great Success Born from a Great Failure” struck me as ironic. The ZDNet article stated:
With SharePoint 2003, Microsoft replaced the Exchange data store with a SQL one. Microsoft also purchased NCompass Labs during this period, and integrated its web-content-management technology with SharePoint. In 2007, Microsoft morphed SharePoint yet again, this time developing and realigning it to be more of an intranet and Internet focused tool. Microsoft launched the SharePoint Server 2007 release shortly before it made yet another related acquisition: enterprise search vendor FAST Search & Transfer. The upgrade process between the 2003 and 2007 versions was anything but smooth, the Softies acknowledged this week.
I am not sure what the meaning of “success” is. Perhaps it is the revenue, estimated at more than $1 billion out of Microsoft $65 billion in revenue. Okay. Perhaps it is the large number of alleged SharePoint licenses that are in the use. There are, I have heard, 100 million licenses. How many are in use? How many are freebies, bundled with other Microsoft products? Perhaps it is the legions of certified Microsoft professionals who earn a living making SharePoint work? I have heard that SharePoint consulting is a very solid business for some companies.
After reading the article, I think I know more about the zig zag path of SharePoint from its inception to the present day. With CFOs worrying about costs, I wonder if the costs of customizing and scaling SharePoint are a consequence of its DNA or of the nurturing Microsoft has given the product.
And search? What about search? A work in progress because of the complexities perhaps?
Stephen Arnold, October 25, 2009
No, no, another essay for no dough.
SharePoint 2010 Keynotes
October 23, 2009
Short honk: If you hate Vegas, you did not go to the SharePoint shin dig. If you need to know what Microsoft said at the SharePoint shin dig, then you have to watch the videos. I would prefer to have a PowerPoint deck and a transcript, but I am a very old addled goose. The world of MSFT is ruled by those who go down the video trail. You can while away a few hours by navigating to Arpan Shah’s Web log and reading “SharePoint Conference Keynotes.” Enjoy. Learn. Think about the consulting revenue you can generate getting this collection of six servers and their many features, functions, and services. Even my goslings are excited.
Stephen Arnold, October 23, 2009
Not even a crust of bread for this write up.
SharePoint 2010: The Twitter Version
October 23, 2009
Short honk: I found “Random Tweets on SharePoint 2010” quite useful. Imagine. Several days of conference glamour boiled down to a string of 140 character messages. When someone asks me “What is the value of Twitter?”, I am going to point out that a handful of tweets can save days of conference fun.
Stephen Arnold, October 23, 2009
Reflections on SharePoint and Search
October 19, 2009
I had an interesting conversation with a librarian at the international library conference in Hammersmith on December 15, 2009. This professional from a central European country asked my views about SharePoint. As I understood her comments, she was in a SharePoint centric environment and found that making small changes to improve information access was difficult, if not impossible.
One reason was her organization’s bureaucratic set up. Her unit was in the same building at the information technology group, but IT reported to one senior manager and her library to another. Another reason was the reluctance of the IT team to make too many changes to SharePoint and its indifference to her legacy library systems. Her challenge was even more difficult because there were multiple legacy systems in the information center. One of these system offered search, and she wanted to have SharePoint make use of the content in that legacy system’s repository. She was not sure which vendors’ search system was in the legacy system, but she thought it was from the “old” Fast Search & Transfer outfit.
The motto of IT and another unit’s management.
Okay, I thought. Fast Search & Transfer, the company Microsoft bought in 2008 and had spent the last 18 months converting into the world class search system for SharePoint. Fast ESP would break through the 50 million document ceiling in SharePoint search and add user experience plus a nifty range of functions.
To make a long story short, she wanted to know, “Will Microsoft and SharePoint support the “old” versions of Fast ESP?” I told her that I recalled reading that Microsoft would stand behind the Fast ESP technology for a decade. She replied, “Really?” Frankly, I do not know if that guarantee extends to OEM instances of the “old” Fast ESP. My hunch is that the problem will fall upon the vendor licensing the Fast ESP engine. But I have learned and suffered from the fact that it is very easy to write marketing collateral and somewhat more difficult to support a system—any system—for a decade. I know mainframe systems that have been around for 30 years, possibly more. But the Linux centric search systems built to index the Web content are a different kettle of Norwegian herring.
My understanding of the trajectory of Fast ESP is that the company took the core of its high speed Web indexing system and reworked it to handle enterprise content. The “old” Fast ESP abandoned the Web search market when the company’s top brass realized that Google’s technology was powering past the Fast technology. Enterprise search became the focus of the company. Over the years, the core of Fast’s Web search system has been “wrapped” and “tweaked” to handle the rigors and quite different requirements of enterprise search. The problem with the approach as I pointed out in the three editions of the Enterprise Search Report I wrote between 2003 and 2006 were:
- Lots of moving parts. My research revealed that a change in a Fast script “here” could produce an unexpected result elsewhere in the system “there”. Not even my wizard son could deal with these discomforting technical killer bees. Chasing down these understandable behaviors took time. With time converted to money, I concluded that lots of moving parts was not a net positive in my enterprise search engagements. Once a system is working, the attitude of the librarian’s IT department is my reaction. Just leave something working alone.
- Performance. Web search required in the late 1990s a certain type of spidering. Today, indexing has become more tricky. Updates are often larger than in the past, so the content processing subsystem has to do more work. Once the index has been updated, other processes can take longer because indexes are bigger or broken into chunks. Speeding up a system is not a simple matter of throwing hardware at the problem. In fact, adding more hardware may not improve performance because the bottleneck may be a consequence of poor engineering decisions made a long time ago or because the hardware was added on but to the wrong subsystem; for example, the production server, not the indexing subsystem.
- Ignorance. Old systems were built by individuals who may no longer be at the company. Today, the loss of the engineers with implicit knowledge of a subsystem make it very difficult—maybe even impossible– to resolve certain functions such as inefficient use of scratch space for index updates. I recall one job we did at a major telco several years ago. The legacy search system was what it was. My team froze it in place and worked with data the legacy system wrote to a file on a more modern server on a fixed schedule. Not perfect, but it was economical and it worked. No one at the company knew how the legacy search system worked. The client did not want to pay my team to figure out the mystery. And I did not want to make assumptions about how long gone engineers cooked their code. Companies that buy some vendors’ search systems may discover that there is a knowledge problem. Engineers often document code in ways that another engineer cannot figure out. So the new owner has to work through the code line by line to find out what is going on. Executives who buy companies make some interesting but often uninformed assumptions; for example, the software we just bought works, is documented, and assembled in a textbook manner. Reality is less tidy than the fantasies of a spreadsheet jockey.
SharePoint: The Enterprise Platform
October 12, 2009
I read “SharePoint 2010: The Enterprise Platform” with an open mind. Microsoft is “all over” the US Federal government. Many of the information technology savvy folks with whom I speak point out the advantages of the SharePoint solution. Programming is getting easier. Users are comfortable with the basic features and functions of the system. Competitors’ products are often more expensive to license. SharePoint is easily shaped into what an information professional needs to solve a particular problem. Microsoft makes available a large number of software “MRE”s; that is, ready to eat, no extra effort required to get certain capabilities or functionality.
Jeremy Thake’s article provides some useful background for SharePoint 2010. This release of SharePoint adds a number of new capabilities to an already richly endowed system. He did make a comment that I found interesting:
In my opinion and a lot of others SharePoint is “a jack of all trades and a master of none”, much like most of the other vendors who played the same card. SharePoint is extremely strong in the collaboration area from an End User perspective, but is weak for example in Records Management, Business Intelligence and Digital Asset Management.The days of purchasing a product for a specific area have clearly gone which is a shame because you pick one of the Enterprise Platforms and suffer in the weaker areas.
He concludes his write up with a reference to MOSS 2007 “horror stories” and makes clear that he loves SharePoint “anyway”.
My thought is that overburdened information technology professionals may find the charms of SharePoint fading when complexity and costs begin to rise. These two issues may be the stepping stones for Google, despite its flaws and weaknesses, to make significant gains at a time when Microsoft is hoping that SharePoint 2010 blunts the appeal of Google’s enterprise offerings.
Google is no match for Microsoft in terms of marketing. But Google does a much better job with the technology for a hybrid platform in my opinion. Can Google deal with the buzz saw of SharePoint 2010? Interesting face off to watch in the last weeks of 2009.
Stephen Arnold, October 12, 2009 No dough
SharePoint and User Adoption
October 11, 2009
I have been working through the SharePoint search related items that Overflight generates. One article “Ideas to Increase End Use Adoption” did not interest me when I first read it. I went back this morning and reviewed the article in SharePoint Buzz because it connected with a remark I heard in a meeting in Arlington, Virginia, last week. The article is straightforward. Some SharePoint features don’t get a quick uptake by users. In order to boost use of a SharePoint system, the author presents a number of ideas. These range from in person training to creating FAQs and other textual information to help users understand the features and functions of a SharePoint system. The article identifies multimedia content as a useful idea. A community-based support service is another good idea.
Now the question, “Why did I return to what is a common sense article about a software system?”
The answer is, “Users resist systems that create more hassles than solved problems.”
The SharePoint blog post underscored three points:
- User ignored systems are a problem, not problem solvers. Maybe training will help resolve this problem, but if users don’t use a system, there’s a deeper issue to resolve. It may be interface. It may be performance. It may be the functions are unrelated to the work task. I don’t know but I know there is a problem.
- Vendors are trying to resolve marketing issues by pushing users in certain directions. When I looked at the list of ways to boost adoption and usage of SharePoint functions, I thought about how some of my grade school teachers approached subjects.
- Microsoft’s new emphasis on UX or the user experience may be a lower cost way to solve deeper issues of a system’s design. The system itself may not deliver a solution, so the easier route is to put some lipstick on the beast and call it a day.
In my opinion, the discussion of user acceptance of certain SharePoint-based applications may point to a deeper and more troubling set of issues within the architecture of SharePoint itself. A developer may like what he or she has built. But users who ignore the service are making clear that something is off base. I am not sure training or an interface can do much if the problem resides within the deeper core of the SharePoint suite.
Stephen Arnold, October 11, 2009
A SharePoint Dev Wiki: No NDA Required
September 28, 2009
A happy quack to the reader who sent me a link to a new SharePoint developer wiki. You can locate the service at SharePoint Dev Wiki. The person responsible for the wiki is Jeremy Thake, a Microsoft whiz. He wrote:
I want to try and make this a collective site to map all the information on WSS 4.0 and SP2010 from around the web in blogs, forum, twitter and web sites around the Internet. This should allow people to start here and either take a shallow dip into the information or deep dive into certain features and find the external resources of relevance.
When I visited, there were a number of current posts. None was about search, but that will change when the SharePoint 10 beastie leaves the lair. Looks promising. No NDA required too.
Stephen Arnold, September 28, 2009
-
- Subscribe to Beyond Search
Feature archive
News archive
-
