Penn State Research Team Uses Big Data to Explore Crime Rates
February 2, 2017
The article on E&T titled Social Media and Taxi Data Improve Crime Pattern Picture delves into a fascinating study that uses big data involving taxi routes and social media location labels from sites like Foursquare to discover a correlation between taxis, locations of interest, and crime. The study was executed by Penn State researchers who are looking for a more useful way to estimate crime rates rather than the traditional approach targeting demographics and geographic data only. The article explains,
The researchers say that the analysis of crime statistics that encompass population, poverty, disadvantage index and ethnic diversity can provide more accurate estimates of crime rates … the team’s approach likens taxi routes to internet hyperlinks, connecting different communities with each other… One surprising discovery is that the data suggests areas with nightclubs tend to experience lower crime rates – at least in Chicago. The explanation may be that it reflects people’s choices to be there.
This research will be especially useful to city planners interested in how certain spaces are being used, and whether people want to go to those spaces. But the researcher Jessie Li, an assistant professor of information sciences, explained that while the correlation is clear, the underlying cause is not yet known.
Chelsea Kerwin, February 2, 2017
Gallup Survey Shows American Faith in Almost All Institutions Waning Except Military
December 17, 2016
The article on The Washington Examiner titled Fishwrap: Confidence in Newspapers, TV News Hits Bottom relays a Gallup survey that shows Americans trust in media slipping to a new low. The focus of the survey was general loss of confidence in various institutions such as churches, banks, big business, and Congress. The article translates the findings,
Overall, said Gallup’s analysis, “Confidence in banks — which took a hit amid the bursting housing bubble in 2007 and 2008, and dropped further after the ensuing financial crisis — fell the most, plunging from 49% in 2006 to 27% now. Confidence in organized religion, which has felt the effects of the scandals enveloping the Catholic Church, dropped from 52% to 41%, one point below last year’s previous low of 42%.
TV news, print news, and Congress saw the greatest falls of about 10%, perhaps unsurprisingly given the lackluster diplomatic skills of our current Congress and the outright failure of the media to properly vet and address the presumptive Republican nominee. Interestingly, the scandals involving General Petraeus as well as those surrounding the military’s failure to manage sexual assault of its female members over the last few years have done nothing to diminish that institution in the eyes of the public. It stands solid and unshakeable at 73%.
Chelsea Kerwin, December 17, 2016
Android Has No Competition in Mobile OS Market
November 23, 2016
Google’s Android OS currently powers 88% of the smartphones in the world, leaving minuscule 12.1 percent to Apple’s iOS and the remaining 0.3 percent for Windows Mobile, BlackBerry OS and Tizen.
IBTimes in an article titled Android Rules! 9 out of Every 10 Phones Run Google’s OS says:
Google’s Android OS dominated the world by powering 88 percent of the world’s smartphone market in the third quarter of 2016. This means 9 out of every 10 mobile phones in the world are using Android, while the rest rely on iOS or other mobile OS such as BlackBerry OS, Tizen and Windows Phone.
The growth occurred despite the fact that smartphone shipments are falling. China and Africa which were big markets have been performing poorly since last three-quarters. Android’s gain thus can be attributed to the fact that Android is an OpenSource system that can be used by any device manufacturer.
Despite being the clear leader, the mobile OS is full of bugs and other inherent problems, as the article points out:
Android platform is getting overcrowded with hundreds of manufacturers, few Android device vendors make profits, and Google’s new Pixel range is attacking its own hardware partners that made Android popular in the first place.
At present, Samsung, Huawei, Oppo and Vivo are the leading Android phone makers. However, Google recently unveiled Pixel, its flagship phone for the premium category. Does it mean that Google has its eyes set on the premium handset category market? Only time can tell.
Vishal Ingole, November 23, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Big Data: Stunners from Researchers
November 18, 2016
I read “Big Data Shows People’s Collective Behavior Follows Strong Periodic Patterns.” May I suggest you sit down, take a deep breath, and contemplate a field of spring flowers before you read these findings. I am not kidding. Hot stuff, gentle reader.
According to the write up,
New research has revealed that by using big data to analyze massive data sets of modern and historical news, social media and Wikipedia page views, periodic patterns in the collective behavior of the population can be observed that could otherwise go unnoticed.
Here are the findings. I take no responsibility for the impact of these Big Data verified outputs. You are on your own. You now have your trigger warning about the findings from online news, newspapers, tweets, and Wikipedia usage. The findings are:
- “People’s leisure and work were regulated by the weather with words like picnic or excursion consistently peaking every summer in the UK and the US.”
- Diet, fruits, foods, and flowers were influenced by the seasons.
- Measles surface in the spring
- Gooseberries appear in June. (Well, maybe not in Harrod’s Creek.)
- Football and Oktoberfest become popular in the fall. (Yep, October for Oktoberfest, right?)
- People get depressed in the winter.
Now you have it. Big Data delivers.
Stephen E Arnold, November 18, 2016
Neural-Net AI Service Echobox Manages Newspaper Presences on Social Media
November 18, 2016
An article at Bloomberg Technology, titled “It Took Robots for This French Newspaper to Conquer Twitter,” introduces Echobox, a startup that uses a neural-network approach to managing clients’ social media presences. The newspaper mentioned in the title is the esteemed Liberation, but Echobox also counts among its clients the French Le Monde, Argentinia’s La Nacion, and The Straits Times out of Singapore, among many others. Apparently, the site charges by the page view, though more pricing details are not provided. Writer Jeremy Kahn reports that Echobox:
… Determines the most opportune time to post a particular story to drive readership, can recommend what headline or tweet to send out, and can select the best photograph to illustrate the post. Using the software to post an average of 27 articles per day, Grainger [Liberation’s CTO] said that Liberation had seen a 37 percent increase in the number of people it reached on Facebook and a 42 percent boost in its reach on Twitter. ‘We have way more articles being seen by 100,000 people or more than before,’ Grangier said. He also said it made life easier for his digital editors, allowing them to spend more time curating the stories they wanted to publish to social media and less on the logistics of actually posting that content.
So, it seems like the service is working. Echobox’s CTO Marc Fletcher described his company’s goal—to create a system that could look at content from an editor’s point of view. The company tailors their approach to each customer, of course. There are competitors in the social-media-management space, like SocialFlow and Buffer, but Kahn says Echobox goes further. He writes:
Echobox professes to offer a fuller range of automation than those services, with its software able to alter a posting schedule to adjust to breaking news, posting content related to that event, and delaying publication of less relevant stories. Echobox uses a neural network, a type of machine learning that is designed to mimic the way parts of the human brain works. This system first learns the audience composition and reading habits for each publication and then makes predictions about the best way to optimize a particular story for social media. Over time, the predictions should get more accurate as it ‘learns’ the nuances of the brand’s audience.
This gives us one more example of how AI capabilities are being put to practical use. Founded in 2013, Echobox is based in London and maintains an office in New York City. The company also happens to be hiring as I write this.
Cynthia Murrell, November 18, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Google Gives Third Day Keynote at Pubcon
November 1, 2016
Technology conferences are the thing to do when you want to launch a product, advertise a new business, network, or get a general consensus about the tech industry. There are multiple conferences revolving around different aspects in the tech industry held each month. In October 2016, Pubcon took place in Las Vegas, Nevada and they had a very good turn out. The thing that makes a convention, though, is the guests. Pubcon did not disappoint as on the third day, Google’s search expert Gary Illyes delivered the morning keynote. (Apparently, Illyes also hold the title Chief of Sunshine and Happiness at Google). Outbrain summed up the highlights of Pubcon 2016’s third day in “Pubcon 2016 Las Vegas: Day 3.”
Illyes spoke about search infrastructure, suggesting that people switch to HTTPS. His biggest push for HTTPS was that it protected users from “annoying scenarios” and it is good for UX. Google is also pushing for more mobile friendly Web sites. It will remove “mobile friendly” from search results and AMP can be used to make a user-friendly site. There is even bigger news about page ranking in the Google algorithm:
Our systems weren’t designed to get two versions of the same content, so Google determines your ranking by the Desktop version only. Google is now switching to a mobile version first index. Gary explained that there are still a lot of issues with this change as they are losing a lot of signals (good ones) from desktop pages that are don’t exist on mobile. Google created a separate mobile index, which will be its primary index. Desktop will be a secondary index that is less up to date.
As for ranking and spam, Illyes explained that Google is using human evaluators to understand modified search better, Rankbrain was not mentioned much, he wants to release the Panda algorithm, and Penguin will demote bad links in search results. Google will also release “Google O for voice search.
It looks like Google is trying to clean up search results and adapt to the growing mobile market, old news and new at the same time.
Whitney Grace, November 1, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Machine Learning Changes the Way We Learn from Data
October 26, 2016
The technology blog post from Danial Miessler titled Machine Learning is the New Statistics strives to convey a sense of how crucial Machine Learning has become in terms of how we gather information about the world around us. Rather than dismissing Machine Learning as a buzzword, the author heralds Machine Learning as an advancement in our ability to engage with the world around us. The article states,
So Machine Learning is not merely a new trick, a trend, or even a milestone. It’s not like the next gadget, instant messaging, or smartphones, or even the move to mobile. It’s nothing less than a foundational upgrade to our ability to learn about the world, which applies to nearly everything else we care about. Statistics greatly magnified our ability to do that, and Machine Learning will take us even further.
The article breaks down the steps of our ability to analyze our own reality, moving from randomly explaining events, to explanations based on the past, to explanations based on comparisons with numerous trends and metadata. The article positions Machine Learning as the next step, involving an explanation that compares events but simultaneously progresses the comparison by coming up with new models. The difference is of course that Machine Learning offers the ability of continuous model improvement. If you are interested, the blog also offers a Machine Learning Primer.
Chelsea Kerwin, October 26, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Knowledge Management: Dazed and Confused?
October 16, 2016
I read an interview posted by TallyFox. If you are not familiar with the company, TallyFox provides a collaboration and content management system. The idea is that a company’s real and off site workers can share information. The company states on its LinkedIn page:
TallyFox’s intelligence platform, makes knowledge sharing fun and dynamic. With our proprietary algorithm SmartMatchPro, access to expertise is facilitated, collective knowledge becomes accessible, and you can benefit from it right now, anywhere in the world.
The TallyFox interview with Dr. Nancy Dixon (Common Knowledge, a non profit and a book) is interesting. I noted these factoids and assertions:
- almost 50% of workers are virtual, or “distributed”
- people who are communicating only virtually tend to lose the sense of purpose of what the organization is about.
- A challenge is “to motivate our experts to share tacit knowledge to make the knowledge from inside of a project available to the team of another project.”
- “Collective Sensemaking is a piece of the process which will show us how to take advantage of the virtual and still stay connected in a human way. We are doing it by crowdsourcing, by Innovation Jams, by Working Out Loud, and all of those ways are bringing back the Human Side into the Virtual.”
- “People don’t offer their knowledge because they don’t know what the other person needs…”
Sounds good.It strikes me that Facebook’s Workplace may be encroaching on the collaboration segment. Does Facebook embrace knowledge management?
Stepping back: Knowledge management leaves me dazed and confused about what, how, where, and why? Perhaps knowledge management should become knowledge “Kumbaya” with people online and posting to Facebook while sitting around a Mac with a fireplace screensaver.
Stephen E Arnold, October 16, 2016
Improvements in News and Military Technology Coming
September 23, 2016
I read two stories. These stories seem unrelated. The first is “Defense Department Reaffirms Its Commitment to Venture Investing.” The second is “Facebook and Twitter Join Coalition to Improve Social Media Newsgathering.”
Let’s look at the short item about the US Department of Defense reaffirming its interest in funding new technology. In my forthcoming, Dark Web Notebook, I point to a Web page which contains a run down of more than 100 open source software components. The software does information collection and processing functions. But the main point is that the organizations creating the code is one of the more interesting lists of entities performing next generation innovation for the Department of Defense. The write up cited above states:
Not everyone is comfortable with a government entity backing what can be sensitive technologies (not to mention the privacy issues wrought by the NSA’s practices and deployment of new tech tools).
My view is that In-Q-Tel is a more visible entity than some of the Department of Defense activities. DoD, in fact, has been in the innovation far longer than In-Q-Tel. One might suggest that substantive innovation emerges from the DoD programs; for example, the DoD is the progenitor of the Internet. My view is that more disruption may be evident in what the DoD is funding than in what the In-Q-Tel organization is funding. The write up misses an important point in my opinion. DoD looks out the windshield of innovation and In-Q-Tel looks at the world via a rear view mirror. Case in point: funding open source software related to Dark Web actions. In-Q-Tel funding companies which often have been in existence for years prior to receiving an infusion of cash and some help making sales calls in the US government.
The second write up also underscores a need for change. The idea is that old fashioned approaches are not needed. New fangled approaches are the cat’s pajamas. The problem is that the new fangled methods make some interesting errors. To fix this, high profile social media companies are going to invent a fix via a coalition.
A method with practiced for news gathering exists. Traditional newspapers illustrate the method. The process works reasonably well. More accurately, the process worked when resources were available to employ individuals who conducted interviews and performed research.
The traditional method changed with software able to count who clicked on what, people with many digital friends, and systems which collect information and figure out what is important.
Now after some interesting mistakes, Internet giants are eager to improve what I call the millennial news method:
Channel 4 News, the Telegraph, the New York Times, Washington Post, BuzzFeed News, ABC News in Australia and Agence France-Presse are among more than 20 news organizations to have signed up to the partner network, which is being organized through Google-backed First Draft.
Now Facebook (big dog) and Twitter (starving dog) are in the game. The point is that the millennial methods appear to work. Unfortunately fake news and other oddities creep into the smart systems. The new methods also help foster tension between the remaining traditional news outfits and the comparative newcomers or disruptors.
The idea of teaming up to improve smart software is interesting. The goal, of course, is to obtain high value information at the lowest possible cost; that is, with the fewest number of humans as possible.
When I read these two articles, I noted three ideas which struck me as worth thinking about:
- Methods exist which work yet interest gravitates away from what works to a need to find a better, more innovative process
- The perception that traditional methods practiced by the Department of Defense and old school newspapers are less useful than “new” approach may slow down innovation or, even worse, get the focus fuzzy.
- The Silicon Valley fascination with the bright and shiny may produce wasteful, duplicate efforts.
Stephen E Arnold, September 23, 2016
Thunderstone Gets an Upgrade
September 1, 2016
Pokémon Go is the latest mobile gaming craze and all of the players want to have a Pikachu as their main Pokémon. Eventually players will evolve their Pikachu into the more powerful Raichu using candy and stardust, but old school Pokémon gamers know that the true way to evolve a Pikachu is with a Thunderstone. The hardest part of evolving a Pikachu, however, was finding the actual Thunderstone. Compulsive searchers have their own difficulties trying to find their information and other related content in their systems. There is a software search solution coincidentally named Thunderstone and it recently went through an upgrade: “Thunderstone Releases Version 16.”
Thunderstone’s newest release includes updates that improve search quality across the board: intranets, aggregators, and public facing Web sites. There also are more authorization options for better security, including a central authentication service and negotiate Kerberos option. Perhaps the biggest upgrade is the following:
Simplified crawl configuration
- Sitemaps allowing easier crawling of sites where URLs are not easily determined from a crawl.
- XML/XSL site support by applying stylesheets to sites that deliver content via XML and XSL instead of HTML; the searchable text is better identified.
- Proxy Auto-config (PAC) file support which makes it easier to index and crawl enterprises composed of different networks with varying proxy rules: the same config files used by browsers may now be used at crawl time.
The Ajax crawlable URL scheme from Google is supported, allowing Ajax based dynamic sites that support it to be crawled and indexed more effectively.”
Thunderstone now packs a more powerful punch for search quality and returning results. Now if only finding Cubone could be improved as well.
Whitney Grace, September 1, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
-
- Subscribe to Beyond Search
Feature archive
News archive
-
