Al Jazeera English, an international 24 hour English-Language news and current affairs TV channel headquartered in Doha, Qatar, has now reached 250 million homes — 5 million of those being in the U.S.

The Los Angeles Times reported on this startling milestone in the article “Al Jazeera English Now Reaches 250 Million Households.”

We learned:

Five years after its launch, there are 130 countries that carry Al Jazeera English, but in the U.S., the channel has limited availability; it can be found on cable systems in Washington, D.C.; New York; Burlington, Vt.; Toledo, Ohio; and, recently, Chicago and in Los Angeles on KCET. And while the U.S. makes up a fraction of the quarter-billion households, it is a major source of AJE’s Web traffic, totaling 40 percent, according to the network.

The fact that Al Jazeera English has such a large web following in the United States despite its limited availability, leads me to think that a significant shift has taken place.

Jasmine Ashton, January 24, 2012

Sponsored by Pandia.com

On October 12, Google Plus rolled out two new features; both allow users to create custom news streams based around topics being shared and build upon the search functionality of the network. The first feature, a real-time search, finds results from Google+ posts that are related to the search term a user enters. As new posts are created centering around the search topic, the user is notified and a real-time stream of posts is begun. ZDNet’s article, “Google+ Real-Time Search: The Social News “Ticker” tells us more about the changes:

… Google engineer Vic Gundotra – who posted the news from his Google Plus feed – notes that it’s a great way to keep up with real-time news events, such as a speech, a court trial or a sporting event. Basically, it’s a real-time news ticker for niche topics. The second feature – hashtag support – essentially turns any hashtag in a post into a searchable term that can be used as another way to create feeds and real-time streams.

This is a catchy notion. I’m interested to see if Google+ will begin integrating all social networking posts into their search results. What they’re doing right now isn’t groundbreaking; Twitter already offers the exact same feature. However, it would be groundbreaking to be able to follow trending topics on all the major social networking sites as they correlate to breaking news.

But Google did real time before. What’s “real time”? Whatever Google wants it to be I suppose from a marketing viewpoint.

Stephen E Arnold, October 28, 2011

Sponsored by Pandia.com

The first example is the challenge of filtering information required by orgnaizatio0ns produced within the organization and by the organizations staff, contractors, and advisors. We learned in the course of our investigation that the promises of processing updates to Web pages, price lists, contracts, sales and marketing collateral, and other routine information are largely unmet. One of the problems is that the disparate content types have different update and change cycles. The most widely used content management system based on our research results is SharePoint, and SharePoint is not able to deliver a comprehensive listing of content without significant latency. Fixes are available but these are engineering tasks which consume resources. Cloud solutions do not fare much better, once again due to latency. The bottom line is that for information produced within an organization employees are mostly unable to locate information without a manual double check. Latency is the problem. We did identify one system which delivered documented latency across disparate content types of 10 to 15 minutes. The solution is available from Exalead, but the other vendors’ systems were not able to match this problem of putting fresh, timely information produced within an organization in front of system users. Shocked? We were.

Reducing latency in search and content processing systems is a major challenge. Vendors often lack the resources required to solve a “hard problem” so “easy problems” are positioned as the key to improving information access. Is latency a popular topic? A few vendors do address the issue; for example, Digital Reasoning and Exalead.

Second, when organizations tap into content produced by third parties, the latency problem becomes more severe. There is the issue of the inefficiency and scaling of frequent index updates. But the larger problem is that once an organization “goes outside” for information, additional variables are introduced. In order to process the broad range of content available from publicly accessible Web sites or the specialized file types used by certain third party content producers, connectors become a factor. Most search vendors obtain connectors from third parties. These work pretty much as advertised for common file types such as Lotus Notes. However, when one of the targeted Web sites such as a commercial news services or a third-party research firm makes a change, the content acquisition system cannot acquire content until the connectors are “fixed”. No problem as long as the company needing the information is prepared to wait. In my experience, broken connectors mean another variable. Again, no problem unless critical information needed to close a deal is overlooked.

Third, many organizations are seeking to make sense of flows of social content. In our research, we learned that a majority of those with whom we spoke expressed interest in Facebook, Twitter, and other social content. However, few firms were using social content, electing to tap into a subset of information via a third party provider like DataSift and processing the content with a third party solution optimized for that content stream. This is a workable solution, but it underscores the problem of making needed information available to employees or analysts who need timely data for a business decision.

Net net.

It is easy to talk about low latency and real time content processing, but it remains expensive and mostly an unmet goal. Many vendors focus on jumping over less formidable hurdles.

We found the write up “Lucid Imagination Brings Open Source Search to Big Data” quite interesting. I have not associated open source search and big data in the manner set forth in the Fierce write up. Here’s a Lucid statement which caught my attention:

One of the biggest issues for companies adopting open source software is that they often lack the polished installation packages of the more commercial software packages. LucidWorks 2.0 provides new setup and management tools via an administrative console that in its words, “streamlines configuration, deployment and operations” for IT when setting up, deploying and managing the application.

This is indeed a point. Installation of some open source software impossible without appropriate technical expertise. However, installation is manageable by those with enough knowledge to decide to use open source software in the first place. Managers may not be able to do it, but managers can hire people who can. My experience is that those who are interested in downloading Lucene/Solr from Apache.org presumably have some technical chops.

More important, in my opinion, is that open source search and content processing must be more capable with regard to the three examples I identified above. A query run on incomplete or stale information is likely to undermine a decision at some point. All to often in my research, I encounter search results which often—perhaps usually—do not reflect the most current information in digital form.

Search is going to climb Big Data Mountain. Aren’t specialized tools and systems needed?

I am okay with improved installation and deployment, but vendors will have to work overtime to beat the Google Search Appliance or Blossom Software’s hosted solution for speedy, painless, search-today roll outs. In short, the problem with “open source search” in the enterprise is narrowed to installation. Several firms are robust, yet different approaches to open source search; for instance:

• Digital Reasoning. The company has proprietary software, but it has teamed with other firms so that it can deliver a comprehensive solution to massive flows of content in a near zero latency implementation. Yep, near zero latency. Installation is part of a search solution, but it is the starting point. The end point is making sure the outputs are usable and reflect the freshest possible data in a form appropriate to the user of the system.
• FLAX. This open source solution delivers a range of features. Installation is reliable and dozens of organizations have found the FLAX approach flexible, scalable, and cost effective. Its open source nature eliminates the “license handcuffs” that limit degrees of freedom imposed by some commercial search system vendors.
• SearchBlox. This firm’s approach seems quite similar to the one described in the Fierce article. Amazon highlights the company’s use of Amazon Web services in a case study. We are not sure how much revenue SearchBlox is generating, but the company says, it is “a leading provider of enterprise search solutions based on Apache Lucene. Over 300 customers in 30 countries use SearchBlox to power their website, intranet and custom search. SearchBlox Software, Inc. was founded in 2003 with the aim to develop commercial search products based on Apache Lucene.”

One negative in the open source search world was the fact that Tesuji, an open source search vendor in Hungary, is repositioning its operations. Despite the strong interest in open source search, the Tesuji approach did not gain sufficient traction.

Our view is that open source search is a viable option for many organizations with the appetite for a solution that sidesteps license fees and permits by-the-drink technical support. However, most of today’s search solutions have far more significant challenges to address. One may argue with the price tags paid for Fast Search & Transfer, Exalead, and Autonomy, but there is a reason. IBM has embraced open source search and wrapped it with its particular array of add ons and special services.  The problem with open source search as well as other search solutions is that once deployed the systems exhibit the latency problem, the connector glitch problem, and the single point of access problem.

My view is that a single attribute like easy installation echoes Oracle’s “secure enterprise search” marketing approach. A single factor is presented as a way to capture market interest. We think that the issues associated with content acquisition, system latency, index updates, interface, and a single point of access to needed information are ultimately more important than installation hassles or even the open source idea itself. I will explore open source search in my next Online Magazine column. The connector issue is of particular interest to me, almost as important as the latency challenge. After all, who wants to run a query and get information which has not been refreshed to make the most current information findable?

Stephen E Arnold, September 30, 2011

Sponsored by Pandia.com

Pairing Endeca Latitude®, an Agile BI platform, with the breadth of social data like Facebook, Twitter, and WordPress as well as other popular social solutions, enables organizations to react to the “big data fire hose” alongside internal data, for marketing analytics, customer intelligence, CRM and competitive intelligence. Endeca and DataSift will demonstrate their joint offering at O’Reilly’s Strata Conference on September 22-23 in New York.

DataSift’s granular and modular sifting abilities combine with Endeca Latitude’s intuitive interface to produce a product that is both powerful and cost-effective. The yet unnamed offering will help companies mine the business value out of the gushing well of new social data.

Our view is that “latency” exists across the six major types of “real time” solutions. What does “real time” mean? Well, it means different things depending upon the application. Some solutions are mind bogglingly expensive. Think Thomson Reuters’ feeds of financial data on certain investments. Others are pretty leisurely; for example, what is trending in the world of Lady Gaga. Interesting tie up. No solid definition of latency yet. We are watching and waiting. You know. Latency.

Emily Rae Aldridge, September 23, 2011

If organizations cannot search by individual as to who needs information, they will invariably push content onto a whole group of people. AFV-News reported “U.S. Army Deploys AtHoc IWSAlerts Emergency Mass Notification System.”

Businesses, schools, universities, and military groups all employ the usage of emergency alerts, providing mass notifications to everyone in their system. Fort Jackson brags that their AtHoc alerts span 25,000 personnel and dependents.

AtHoc IWS Alerts offer control from a unified Web-based console, which allows Fort Jackson to send alerts to cell phones, landlines, smart phones, SMS text and email. It’s not just Fort Jackson—AtHoc services more than 1.5 million Department of Defense, more than any other provider.

We learned about AtHoc’s capabilities and infrastructure from the AFV-News article:

[The] system integrates with the post’s existing Internet Protocol network services, which means reduced infrastructure and maintenance costs. Personnel accountability is accomplished through the bi-directional capability, allowing responses to notifications in real-time. Network alert delivery and response can be tracked, ensuring that targeted recipients have received and responded to alerts.

While alerts for dangerous situations and testing can save lives and are obviously a necessity, mass alert systems also unfortunately end up in too many unnecessary inboxes.

Megan Feil, August 21, 2011

Sponsored by Pandia.com

Most unstructured content is either trapped in silos across the organization or entirely unmanaged “content in the wild.” A majority of that unstructured content can be deemed unnecessary – over-retained, irrelevant, or duplicate – and should be either decommissioned or deleted.

As we understand it, one licenses the  Classification Module and/or Content Analytics software to prevent the previously stated problem and to provide content classification.

Sounds great like the ads for IBM mainframes and the promotional information about

But a disturbing question to the ArnoldIT goslings who wear blue IBM logos: What if this stuff costs too much and does not deliver on the fly classification for real time processing of tweets and Google Plus public content?

Maybe an IBM box of parts with an expensive IBM engineering team is not exactly what some outfits require? Perhaps IBM should look around and maybe snap up one of the hot players in the space. IBM has been announcing partnerships with a number of interesting companies. We track  Digital Reasoning and and think its technology looks very promising? IBM is in a good position to have an impact in the data analysis space, but it needs tools that go beyond its in house code and Cognos and SPSS methods in our opinion.

Jasmine Ashton, August 19, 2011

Sponsored by Pandia.com, publishers of The New Landscape of Enterprise Search

The service brings in all your subscribed content from major social networks, then offers a number of different ways to sort what it finds. My favorite is the filter called “Potentially Missed – links from people who don’t share a lot of links.

Know About It explains on its Web site they collect all the links passing through your social streams and perform a “bunch of analysis on each one to determine which are most likely to be of interest to you.”

Sounds helpful. The idea of sorting all your inbound information in a variety of ways is appealing. You can also look at the service’s recommendations based on your expressed interest or get a personalized email digest.

Mr. Kirkpatrick has not yet tested the service but likes the idea. What isn’t mentioned? Privacy. So what is the ‘bunch of analysis’ and where do all those links end up? Advertisers? If the start-up is successful, time will tell. But with the social web moving at a never-ending pace and growing, social media users wanting to sort their feeds likely won’t mind too much. We think these types of tools are likely to grow in importance as free real time search becomes a difficult service to monetize.

Philip West, August 2, 2011

Sponsored by Pandia.com, publishers of The New Landscape of Enterprise Search

Unlike TweetDeck, Unity is not free. However, the cost may be worth it. The article points to two components that put this app far ahead:

• “It monitors much more than Facebook or Twitter, in particular it crawls user forums, which is trickier and in practice is often much more important for many brands;
• It works in over 30 languages. Synthesio has teams of translators around the world and around the clock that monitor conversations in many languages and make it all accessible to marketers in one dashboard.”

For your money, you get information about how to customize your dashboard. Regular analytic reports are available for an added cost. Such monitoring of the real time environment may soon be essential for companies to stay competitive, “or well”, bring the future home today..

Cynthia Murrell, May 22, 2011

With the federal government’s increased focus on cloud computing, (Digital Reasoning’s) flagship product Synthesys® provides a unique Entity Oriented Analytics solution that enables government agencies to tap into the power of big data. The Advisory Board represents a team with unique insight into the requirements of Big Data, text analytics and intelligence solutions for government agencies.

The board members are: Gen. William T. Hobbins, who retired as Commander, U.S. Air Forces in Europe; Bob Flores, founder and president of Applicology Inc., who spent 31 years in the US intelligence community; Anita K. Jones, who managed the Department of Defense’s science and technology program; Capt. Nick Buck, who spent 15 years in National Security Space, including 10 years in the National Reconnaissance Office; and Mike Miller, currently president of M4 Associates and previously VP of Juniper Networks’ Public Sector Division where he was responsible for all business with Juniper’s Public Sector customers in the US. This kind of talent should be valuable guiding Digital Reasoning’s federal sector strategy.

We have tracked this Franklin, Tennessee, company since its inception. To get some insight into the firm’s approach, you may want to read these two interviews ArnoldIT.com, the owner of this news service, conducted with Tim Estes, the founder of Digital Reasoning. The February 2010 interview explores the core technology of the firm and how it differs from other vendors’ methods. The December 2010 interview probes the new version of the firm’s flagship technology.

Stephen E Arnold, May 16, 2011

Freebie

Mediasift and Twitter have agreed to a partnership that has the potential to change how marketers and companies understand conversations about their products as well as how they choose to market them to target audiences. By utilizing the advanced DataSift software they are able to break down “tweets” into a language that is easily understandable and searchable and is still quite cost effective with it’s “pay per use” subscription. The article said:

As a company we have been very fortunate to have access to the Twitter Firehose for quite some time. This has enabled us over the past two years to refine our thinking, leading to the incarnation of DataSift.

DataSift compiles multiple social media feeds and additional data sets to create a common abstract layer which provides meaningful insight into much of the chaotic and unstructured data from the outlets. It took nearly 18 months to complete the DataSift platform but it has already seen a huge outpouring of company and marketing support with more than a billion requests per month.

Important stuff for the real time crowd.

Leslie Radcliff, April 15, 2011

Freebie