CyberOSINT banner


Enterprise Search: Security Remains a Challenge

Download an open source enterprise search system or license a proprietary system. Once the system has been installed, the content crawled, the index built, the interfaces set up, and the system optimized the job is complete, right?

Not quite. Retrofitting a keyword search system to meet today’s security requirements is a complex, time consuming, and expensive task. That’s why “experts” who write about search facets, search as a Big Data system, and search as a business intelligence solution ignore security or reassure their customers that it is no big deal. Security is a big deal, and it is becoming a bigger deal with each passing day.

There are a number of security issues to address. The easiest of these is figuring out how to piggyback on access controls provided by a system like Microsoft SharePoint. Other organizations use different enterprise software. As I said, using access controls already in place and diligently monitored by a skilled security administrator is the easy part.

A number of sticky wickets remain; for example:

  • Some units of the organization may do work for law enforcement or intelligence entities. There may be different requirements. Some are explicit and promulgated by government agencies. Others may be implicit, acknowledged as standard operating procedure by those with the appropriate clearance and the need to know.
  • Specific administrative content must be sequestered. Examples range from information assembled for employee health or compliance requirements for pharma products or controlled substances.
  • Legal units may require that content be contained in a managed system and administrative controls put in place to ensure that no changes are introduced into a content set, access is provided to those with specific credential, or kept “off the radar” as the in house legal team tries to figure out how to respond to a discovery activity.
  • Some research units may be “black”; that is, no one in the company, including most information technology and security professionals are supposed to know where an activity is taking place, what the information of interest to the research team is, and specialized security steps be enforced. These can include dongles, air gaps, and unknown locations and staff.


An enterprise search system without NGIA security functions is like a 1960s Chevrolet project car. Buy it ready to rebuild for $4,500 and invest $100,000 or more to make it conform to 2015’s standards.  Source:

How do enterprise search systems deal with these access issues? Are not most modern systems positioned to index “all” content? Is the procedures for each of these four examples part of the enterprise search systems’ administrative tool kit?

Based on the research I conducted for CyberOSINT: Next Generation Information Access and my other studies of enterprise search, the answer is, “No.”

Read more »


Interview with Dave Hawking Offers Insight into Bing, FunnelBack and Enterprise Search

The article titled To Bing and Beyond on IDM provides an interview with Dave Hawking, an award-winner in the field of information retrieval and currently a Partner Architect for Bing. In the somewhat lengthy interview, Hawking answers questions on his own history, his work at Bing, natural language search, Watson, and Enterprise Search, among other things. At one point he describes how he arrived in the field of information retrieval after studying computer science at the Australian National University, where he the first search engine he encountered was the library’s card catalogue. He says,

“I worked in a number of computer infrastructure support roles at ANU and by 1991 I was in charge of a couple of supercomputers…In order to do a good job of managing a large-scale parallel machine I thought I needed to write a parallel program so I built a kind of parallel grep… I wrote some papers about parallelising text retrieval on supercomputers but I pretty soon decided that text retrieval was more interesting.”

When asked about the challenges of Enterprise Search, Hawking went into detail about the complications that arise due to the “diversity of repositories” as well as issues with access controls. Hawking’s work in search technology can’t be overstated, from his contributions to the Text Retrieval Conferences, CSIRO, FunnelBack in addition to his academic achievements.

Chelsea Kerwin, December 09, 2014

Sponsored by, developer of Augmentext

Latest News

Google: Circling the Semi Virtual Wagons

Google has an interesting track record with nation states, supra national agencies, and regulators. Google has a new Euro boss, Matt Brittin. According to “Here’s... Read more »

February 26, 2015 | | Comment

IBM and the Functioning Assumption

I read “Could IBM Really Function with Tens of thousands Fewer Staff?” I think this is an interesting headline. It contains an assumption that IBM is indeed... Read more »

February 26, 2015 | | Comment

Amidst the Google News, a Titbit about YouTube

The Wall Street Journal and then Web information services reported that YouTube is not making Google much money. Maybe none? I liked this comment in “YouTube Still... Read more »

February 26, 2015 | | Comment

Interface Changes for Google Mobile Search

Google is the top search US search engine for many reasons and it can maintain this title because the company is constantly searching (pun unintended) to improve... Read more »

February 26, 2015 | | Comment

Smartlogic and SmartLogic: Brand Clash

I am a simple person, gliding slowly into the assisted living facility. I know I cannot keep up with the management wizards in the search and content processing... Read more »

February 26, 2015 | | Comment

New Version of On Premises SharePoint Due Out This Year

Many users were nervous that Microsoft was phasing out on-site SharePoint installations. Fortunately for those concerned, that is not the case. Microsoft has announced... Read more »

February 26, 2015 | | Comment

Google Search: Live Chat, But Relevance?

i read “Google Tests Live Chat With Businesses From Search Results.” According to the write up: Google is testing out a service that incorporates live chat with... Read more »

February 25, 2015 | | Comment

Hewlett Packard: More Misfires

I am no financial whiz kid. I read “Hewlett Packard Shares Take a Beating on Poor Sales.” Even I was able to figure out that “HP’s corporate business struggled... Read more »

February 25, 2015 | | Comment

Smart Software: A Semi History

Wondering how smart software came to be, how it works, what it can do? Navigate to “The Believers.” There’s information about a number of smart software initiatives,... Read more »

February 25, 2015 | | Comment

Tibco and Predictive Analytics

Business intelligence and infrastructure firm Tibco has been busy making deals lately. A press release at Digital Journal tells us that “Tibco and Lavastorm Analytics... Read more »

February 25, 2015 | | 1 Comment