Information Access and Analytics are Vital in Subjective World of Big Data
February 7, 2013
Are decisions in the business world ultimately based on rational and objective facts? Or are they based on intuition? University of Virginia, Darden School Professor Robert Carraway weighs in on the subject in his recent guest post on Forbes. The article “Meeting the Big Data Challenge: Don’t Be Objective” expands on these questions.
Carraway reports that the Corporate Executive Board of U.Va.’s Darden School of Business created a new tool called Insight IQ. This tool is used to assess the tendency of managers to rely on intuition versus analysis. It turns out that 19% of over 5,000 managers in major global corporations are “Visceral decision makers” that rely almost exclusively on intuition.
The author argues that big data and more logical, objective analytical tools and frameworks places even more weight on the role of intuition.
This is one reason why I think so many CEO’s say they are far more likely to rely on intuition than analysis in making decisions. Almost by definition, the issues they deal with are complex and multi-faceted, impossible to completely model or holistically analyze. My experience is that every CEO I know DOES use analysis, but what is far more visible to them is how they have to rely on intuition to fill in the gaps, and this is what they self-report.
If intuition is inevitable in the process of making decisions then it is even more important to rely on technological solutions to offer the analysis which intuition becomes informed by. We suggest taking a look at PolySpot solutions for enterprise information access.
Megan Feil, February 7, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search.
More Real Time BI Promises
February 5, 2013
Have we not heard this story before? Datamation reports that, “Real Time Is BI Right Time BI,” an article about real time intelligence being incorporated in businesses. While the technology does exist, companies are reluctant to adopt real time because of the cost and the complexity associated with it. Technology has changed, as have consumers’ requirements. Real time solutions have become a desired if not necessary BII solution, as found in a series of surveys (though it is not mentioned who conducted it).
We can assume this company issues the surveys:
“Tony Cosentino, VP & Research Director of Ventana Research, noted that the survey found even stronger interest in implementing real time BI/analytics in the next two years via complex event processing.”
But the main question that keeps popping up is: what exactly is real time intelligence?
“’There is still a lot of confusion over what right time and real time really are,’ noted Cosentino. ‘When we talk about real-time transaction processing, with data streaming into mobile devices right from transaction systems, rules-based analytics trigger actions. This complex event process is like when a car crashes and the air bags deploy. These rule-based systems are different than the complex algorithms that may be at work behind the scenes.’ In addition to the immediate reaction to an event, Cosentino and others warn IT that everybody expects everything right away. “If someone waits more than two seconds for something to appear on their screen, they will jump to another screen,’ Cosentino added.”
Throw in people’s constant ned for immediate access to everything and analytics and the IT departments have a lot of work cut out for them. Will real time finally deployed? Maybe now more than ever, but we will have to see…again.
Whitney Grace, February 05, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
Business Decisions No Longer Guess Work with PolySpot Solutions
January 31, 2013
In a naturally chaotic world, having a penchant for theorizing answers about the fundamental mysteries of the world and less serious daily issues has always aided people in successfully navigating the world around them and their companies or organizations. A recent article from Wired theorizes that big data may bring about the end of theory as we know it in a thought provoking article called, Big Data, Language and the Death of the Theorist.
People like Kalev Leetaru, who found clues pointing to Osama Bin Laden’s hiding location through publicly available data (after Laden’s death), predict that forecasting future events is possible and currently enabled.
The article discusses the intelligence aspect of big data analytics that is even applicable to language:
For scientists and mathematicians, working with supercomputers makes sense — their information is numerical. It already exists in a language that machines can read. The interesting thing here for historians and sociologists and literature critics, and everyone else who works with language and the vagaries of the human condition, is that we’ve reached a point where supercomputers are fast enough to crunch that data just as easily as anything else.
Both the inherent power of language associated with semantic enrichment in business intelligence solutions such as those from PolySpot and their library of connectors that enable enriched information access across the enterprise will aid organizations to make better decisions in real time. In the areas where the paradigm of the world changes, the same goes for business.
Megan Feil, January 31, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search.
Thoughts about Commercial Databases: 2013
January 29, 2013
After the dress rehearsal for my weaponized information webinar, a couple of librarians and I were talking about the commercial database business. I narrowed the focus to the commercial outfits selling primary and secondary information to libraries and other professionals; namely, to the legal and health care sectors.
In a nutshell, the digital future does not look too bright for companies such as:
- Ebsco Electronic Publishing (everything but the kitchen sink coverage)
- Elsevier (scientific and technical with Fast Search in its background)
- ProQuest (everything but the kitchen sink coverage plus Dialog)
- Thomson Reuters (multiple disciplines, including financial real time info)
- Wolters Kluwer (mostly legal and medical and a truckload of individual brands)
I just reread “Why Acquisitions Fail: The Five Main Factors by Pearson Education. This outfit has a long and storied past. The irony of Pearson Education explaining the problems of making an acquisition work is interesting but not germane to the main points in the write up. the fact that this item was available to me without charge via the Internet is amusing to me as well. Here’s what the Pearson analyst suggests about the causes of failure:
Survey after survey has proclaimed that most acquisitions fail. Denzil Rankine’s Executive Briefing on Why Acquisitions Fail (FT Prentice Hall) examines why. There are five key factors, which we will examine below:
- Flawed business logic
- Flawed understanding of the new business
- Flawed deal management
- Flawed integration management
- Flawed corporate development
No argument from me. The business model for these firms has been built on selling “must have” information to markets who need the information to do their job. The reason for the stress on this group of companies is that the traditional customers are strapped for cash or have lower cost alternatives.
If one of these outfits buys a company, the likelihood that the acquisition will be a home run revenue success is low. These five companies are bottom-line oriented, so the acquisitions will have to perform. The idea of massive investment to realize the promise of the purchase is not in the game plan.
So big traditional commercial database companies have to find a way to work around the Pearson Education hurdles. Let me consider some of the options available to the Ebscos, Elseviers, ProQuests, Thomsons, and Wolters Kluwers of the world. (Yes, there are oligopolies in a number of other countries, not just the US and Western Europe.)
The Hail, Mary Deal
This is the option which makes investment bankers’ and deal brokers’ hearts go pitty patter. We know how that approach works.
Buy One Another
The idea is that no other outfit wants to buy commercial database companies. Ergo: These outfits buy one another in some combination. Good for the investment bankers but long term, the customers may not be able to cope with ever increasing prices. Librarians, lawyers, and accountants are not exactly in a GEICO made of money mode.
The Microsoft Dell Variant
The idea is that a third party like Google buys one or more commercial database companies and monetizes the content with ads. (I would lobby for this if I were attached to a giant money machine like the Google.)
Fire Sale
I think that Thomson Reuters’ effort to get out of the health fraud business makes clear that the price offered kills the deals. Nevertheless, some of the commercial database publishers may be forced to chop off fingers and toes to keep the core alive. Highly probable path opine I.
Raise Prices and Innovate from Within
This option keeps the Board of Directors engaged. The reality is that such innovation goes nowhere. Ah, I am looking forward to annoyed vice presidents asserting, “I am innovative. We do innovate.” Okay, okay.
Net net?
Big changes are coming for commercial database producers, access to curated content, and the quality of the commercial information. Lawyers are looking to cut costs. No good for Lexis and West. Librarians are under severe financial pressure. Accountants? Accountants don’t want to spend their own money.
Looks like the future is moving in directions different from what these traditional, commercial database producers are going. I suppose after a couple of decades of evolution, the arrival of the End of Times is tough to accept.
Disagree? Agree? Surprise me. Keep in mind that I don’t have a stake in these companies and find myself baffled by the management challenges each has created for itself.
Stephen E Arnold, January 29, 2013
Sponsored by Dumante.com
Information Confusion: Search Gone South
January 26, 2013
I read “We Are Supposed to Be Truth Tellers.” I think the publication is owned by a large media firm. The point of the write up is that “real news” has a higher aspiration and may deal with facts with a smidgen of opinion.
I am not a journalist. I am a semi retired guy who lives in rural Kentucky. I am not a big fan of downloading and watching television programs. The idea that I would want to record multiple shows, skip commercials, and then feel smarter and more informed as a direct result of those activities baffles me.
Here’s what I understand:
A large company clamped down on a subsidiary’s giving a recording oriented outfit a prize for coming up with a product that allows the couch potato to skip commercials. The fallout from this corporate decision caused a journalist to quit and triggered some internal grousing.
The article addresses these issues, which I admit, are foreign to me. Here’s one of the passages which caught my attention:
CNET reporters need to either be resigning or be reporting this story, or both. On CNET. If someone higher up removes their content then they should republish it on their personal blogs. If they are then fired for that they should sue the company. And either way, other tech sites, including this one, would be more than happy to make them job offers.
I agree I suppose. But what baffles me are these questions:
- In today’s uncertain financial climate, does anyone expect senior management to do more than take steps to minimize risk, reduce costs, and try to keep their jobs? I don’t. The notion that senior management of a media company embraces the feel good methods of Whole Earth or the Dali Lama is out of whack with reality in my opinion.
- In the era of “weaponized information,” pay to play search traffic, and sponsored content from organizations like good old ArnoldIT—what is accurate. What is the reality? What is given spin? I find that when I run a query for “gourmet craft spirit” I get some darned interesting results. Try it. Who are these “gourmet craft spirit” people? Interesting stuff, but what’s news, what’s fact, and what’s marketing? If I cannot tell, how about the average Web surfer who lets online systems predict what the user needs before the user enters a query?
- At a time when those using online to find pizza and paradise, can users discern when a system is sending false content? More importantly, can today’s Fancy Dan intelligence systems from Palantir-likeand i2 Group-like discern “fake” information from “real” information? My experience is that with sufficient resources, these advanced systems can output results which are shaped by crafty humans. Not exactly what the licensees want or know about.
Net net: I am confused about the “facts” of any content object available today and skeptical of smart systems’ outputs. These can be, gentle reader, manipulated. I see articles in the Wall Street Journal which report on wire tapping. Interesting but did not the owner of the newspaper find itself tangled in a wire tapping legal matter? I read about industry trends from consulting firms who highlight the companies who pay to be given the high intensity beam and the rah rah assessments. Is this Big Data baloney sponsored content, a marketing trend, or just the next big thing to generate cash in a time of desperation. I see conference programs which feature firms who pay for platinum sponsorships and then get the keynote, a couple of panels, and a product talk. Heck, after one talk, I get the message about sentiment analysis. Do I need to hear from this sponsor four or five more times. Ah, “real” information? So what’s real?
In today’s digital world, there are many opportunities for humans to exercise self interest. The dust up over the CBS intervention is not surprising to me. The high profile resignation of a real journalist is a heck of a way to get visibility for “ethical” behavior. The subsequent buzz on the Internet, including this blog post, are part of the information game today.
Thank goodness I am sold and in a geographic location without running water, but I have an Internet connection. Such is progress. The ethics stuff, the assumptions of “real” journalists, and the notion of objective, fair information don’t cause much of stir around the wood burning stove at the local grocery.
“Weaponized information” has arrived in some observers’ consciousness. That is a step forward. That insight is coming after the train left the station. Blog posts may not be effective in getting the train to stop, back up, and let the late arrivals board.
Stephen E Arnold, January 26, 2013
Stunning Visuals Show How Datasets Connect
January 25, 2013
Data analysis can be tricky business, especially when you have been staring at a computer screen and all the information blurs together. What if there was a way to make the data more visually stimulating, not to mention could take out the guesswork in correlations? Gigaom may have found the answer, “Has Ayasdi Turned Machine Learning Into A Magic Bullet?” Ayasdi is a startup company that has created software for visually mapping hidden connections in massive datasets. The company just opened its doors with $10.25 million in funding, but what is really impressive is their software offering:
“At its core, Ayasdi’s product, a cloud-based service called the Insight Discovery Platform, is a mix of distributed computing, machine learning and user experience technologies. It processes data, discovers the correlations between data points, and then displays the results in a stunning visualization that’s essentially a map of the dataset and the connections between every point within it. In fact, Ayasdi is based on research into the field of topological data analysis, which Co-founder and President Gunnar Carlsson describes a quest to present data as intuitively as possible based solely on the similarity of (or distance between, in a topological sense) the data points.”
The way the software works is similar to social networking. Social networking software maps connections between users and their content, but the algorithms do not understand what the connections mean. Ayasdi makes it easier for its users to attach meaning to the correlations. The article also points out that Ayasdi’s software is hardly a new concept, but for some working in BI it takes out a lot of the discovery work. The software may be really smart, but humans are still needed to interpret the data.
Whitney Grace, January 25, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
Faster Decisions Made Using Information Delivery from PolySpot
January 23, 2013
The amount of data a company has in its possession means nothing unless the company has the tools and team to extract information from the sheer numbers and data bits. CIO recently revealed an article called, “How To Use Big Data to Make Faster and Better Business Decisions.”
Important business related questions requiring action can be answered and assessed in days rather than months. With such potential on the table for big data, NewVantage Partners conducted a study to determine how organizations are using big data. It turns out that 85% of their respondents reported big data initiatives as currently underway.
The article states:
Respondents gave a number of reasons for their investments in Big Data, from reducing risk to creating higher-quality products and services. But two reasons were clear leaders: achieving better, fact-based decision-making and improving the customer experience. Of course, these are leading reasons for investments in traditional business intelligence (BI) analytics, too.
Emphasis rests on the amount of time it takes for companies to answer the questions they need answered. Luckily, there are technologies such as PolySpot that enable information access across the enterprise. Communication and analysis can thus take place in near real-time.
Megan Feil, January 23, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
The Question Drives the Search
January 22, 2013
Over at Chiliad, an article called “Search Vs. Correlation Vs. Causality-What Do Your Goals Require?” discusses how different types of questions change search results. Business intelligence and search are different aspects of the same end result and together they can generate more useful results. Correlations provide analytics, thus turning up unexpected and often useful relationships. The value is not in observations, but rather connections between data, which then influences decision making. The “why” factor is also a big part, because it explains how the data will be used and what the end result will be.
It involves more legwork than anything else:
“Iterative Discovery—understanding “why”—requires a different approach. Not only does digging in deliver more information, it suggests new inquiry and allows you to dig deeper. It helps you understand—across all your sources—what matters most. Although Chiliad named this approach Iterative Discovery, we didn’t invent it. Great researchers and analysts did. We simply observed them—and created a tool tuned to figuring out…’What does it mean?’”
If the why question cannot be answered than search, business intelligence, and everything else is useless. Users conduct these actions to find an answer and if an answer is not provided the action are worthless.
Whitney Grace, January 22, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
Social Search: Don Quixote Is Alive and Well
January 18, 2013
Here I float in Harrod’s Creek, Kentucky, an addled goose. I am interested in other geese in rural Kentucky. I log into Facebook, using a faux human alias (easier than one would imagine) and run a natural language query (human language, of course). I peck with my beak on my iPad using an app, “Geese hook up 40027.” What do I get? Nothing, Zip, zilch, nada.
Intrigued I query, “modern American drama.” What do I get? Nothing, Zip, zilch, nada.
I give up. Social search just does not work under my quite “normal” conditions.
First, I am a goose spoofing the world as a human. Not too many folks like this on Facebook, so my interests and my social graph is useless.
Second, the key words in my natural language query do not match the Facebook patterns, crafted by former Googlers and 20 somethings to deliver hook up heaven and links to the semi infamous Actor’s Theater or the Kentucky Center.
Social search is not search. Social search is group centric. Social search is an outstanding system for monitoring and surveillance. For information retrieval, social search is a subset of information retrieval. How do semantic methods improve the validity of the information retrieved? I am not exactly sure. Perhaps the vendors will explain and provide documented examples?
Third, without context, my natural language queries shoot through the holes in the Swiss Cheese of the Facebook database.
After I read “The Future of Social Search,” I assumed that information was available at the peck of my beak. How misguided was I? Well, one more “next big thing” in search demonstrated that baloney production is surging in a ailing economy. Optimism is good. Crazy predictions about search are not so good. Look at the sad state of enterprise search, Web search, and email search. Nothing works exactly as I hope. The dust up between Hewlett Packard and Autonomy suggests that “meaning based computing” is a point of contention.
If social search does not work for an addled goose, for whom does it work? According to the wild and crazy write up:
Are social networks (or information networks) the new search engine? Or, as Steve Jobs would argue, is the mobile app the new search engine? Or, is the question-and-answer formula of Quora the real search 2.0? The answer is most likely all of the above, because search is being redefined by all of these factors. Because search is changing, so too is the still maturing notion of social search, and we should certainly think about it as something much grander than socially-enhanced search results.
Yep, Search 2.0.
But the bit of plastic floating in my pond is semantic search. Here’s what the Search 2.0 social crowd asserts:
Let’s embrace the notion that social search should be effortless on the part of the user and exist within a familiar experience — mobile, social or search. What this foretells is a future in which semantic analysis, machine learning, natural language processing and artificial intelligence will digest our every web action and organically spit out a social search experience. This social search future is already unfolding before our very eyes. Foursquare now taps its massive check in database to churn out recommendations personalized by relationships and activities. My6sense prioritizes tweets, RSS feeds and Facebook updates, and it’s working to personalize the web through semantic analysis. Even Flipboard offers a fresh form of social search and helps the user find content through their social relationships. Of course, there’s the obvious implementations of Facebook Instant Personalization: Rotten Tomatoes, Clicker and Yelp offer Facebook-personalized experiences, essentially using your social graph to return better “search” results.
Semantics. Better search results. How does that work on Facebook images and Twitter messages?
My view is that when one looks for information, there are some old fashioned yardsticks; for example, precision, recall, editorial policy, corpus provenance, etc.
When a clueless person asks about pop culture, I am not sure that traditional reference sources will provide an answer. But as information access is trivialized, the need for knowledge about the accuracy and comprehensiveness of content, the metrics of precision and recall, and the editorial policy or degree of manipulation baked into the system decreases.
See Advantech.com for details of a surveillance system.
Search has not become better. Search has become subject to self referential mechanisms. That’s why my goose queries disappoint. If I were looking for pizza or Lady Gaga information, I would have hit pay dirt with a social search system. When I look for information based on an idiosyncratic social fingerprint or when I look for hard information to answer difficult questions related to client work, social search is not going to deliver the input which keeps this goose happy.
What is interesting is that so many are embracing a surveillance based system as the next big thing in search. I am glad I am old. I am delighted my old fashioned approach to obtaining information is working just fine without the special advantages a social graph delivers.
Will today’s social search users understand the old fashioned methods of obtaining information? In my opinion, nope. Does it matter? Not to me. I hope some of these social searchers do more than run a Facebook query to study for their electrical engineering certification or to pass board certification for brain surgery.
Stephen E Arnold, January 18, 2013
Dr. Jerry Lucas: Exclusive Interview with TeleStrategies ISS Founder
January 14, 2013
Dr. Jerry Lucas, founder of TeleStrategies, is an expert in digital information and founder of the ISS World series of conferences. “ISS” is shorthand for “intelligence support systems.” The scope of Mr. Lucas’ interests range from the technical innards of modern communications systems to the exploding sectors for real time content processing. Analytics, fancy math, and online underpin Mr. Lucas’ expertise and form the backbone of the company’s training and conference activities.
What makes Dr. Lucas’ viewpoint of particular value is his deep experience in “lawful interception, criminal investigations, and intelligence gathering.” The perspective of an individual with Dr. Lucas’ professional career offers an important and refreshing alternative to the baloney promulgated by many of the consulting firms explaining online systems.
Dr. Lucas offered a more “internationalized” view of the Big Data trend which is exercising many US marketers’ and sales professionals’ activities. He said:
“Big Data” is an eye catching buzzword that works in the US. But as you go east across the globe, “Big Data” as a buzzword doesn’t get traction in the Middle East, Africa and Asia Pacific Regions if you remove Russia and China. One interesting note is that Russian and Chinese government agencies only buy from vendors based in their countries. The US Intelligence Community (IC) has big data problems because of the obvious massive amount of data gathered that’s now being measured in zettabytes. The data gathered and stored by the US Intelligence Community is growing beyond what typical database software products can handle as well as the tools to capture, store, manage and analyze the data. For the US, Western Europe, Russia and China, “Big Data” is a real problem and not a hyped up buzzword.
Western vendors have been caught in the boundaries between different countries’ requirements. Dr. Lucas observed:
A number of western vendors made a decision because of the negative press attention to abandon the global intelligence gathering market. In the US Congress Representative Chris Smith (R, NJ) sponsored a bill that went nowhere to ban the export of intelligence gathering products period. In France a Bull Group subsidiary, Amesys legally sold intelligence gathering systems to Lybia but received a lot of bad press during Arab Spring. Since Amesys represented only a few percent of Bull Group’s annual revenues, they just sold the division. Amesys is now a UAE company, Advanced Middle East Systems (Ames). My take away here is governments particularly in the Middle East, Africa and Asia have concerns about the long term regional presence of western intelligence gathering vendors who desire to keep a low public profile. For example, choosing not to exhibit at ISS World Programs. The next step by these vendors could be abandoning the regional marketplace and product support.
The desire for federated information access is, based on the vendors’ marketing efforts, is high. Dr. Lucas made this comment about the existence of information silos:
Consider the US where you have 16 federal organizations collecting intelligence data plus the oversight of the Office of Director of National Intelligence (ODNI). In addition there are nearly 30,000 local and state police organizations collecting intelligence data as well. Data sharing has been a well identified problem since 9/11. Congress established the ODNI in 2004 and funded the Department of Homeland Security to set up State and Local Data Fusion Centers. To date Congress has not been impressed. DNI James Clapper has come under intelligence gathering fire over Benghazi and the DHS has been criticized in an October Senate report that the $1 Billion spent by DHS on 70 state and local data fusion centers has been an alleged waste of money. The information silo or the information stovepipe problem will not go away quickly in the US for many reasons. Data cannot be shared because one agency doesn’t have the proper security clearances, job security which means “as long as I control access the data I have a job,” and privacy issues, among others.
The full text of the exclusive interview with Dr. Lucas is at http://www.arnoldit.com/search-wizards-speak/telestrategies-2.html. The full text of the 2011 interview with Dr. Lucas is at this link. Stephen E Arnold interviewed Dr. Lucas on January 10, 2013. The full text of the interview is available on the ArnoldIT.com subsite “Search Wizards Speak.”
Worth reading.
Donald Anderson, January 14, 2013
-
- Subscribe to Beyond Search
Feature archive
News archive
-
