Real Time Remains Complicated and Costly
July 14, 2016
I love it when search and content processing vendors yammer about their “real time systems.” Years ago I did a report for a client in Europe about the costs of real time. Summarizing the six month research effort is easy: Real time is tough in computer systems. Latency exists.
To get a sense of how tough it is to accelerate certain online actions, navigate to “Legendary Hedge Fund Wants to Use Atomic Clocks to Beat High-Speed Traders.” Despite the wonky “legendary,” I noted this comment:
Patent application no. 14/451,356 [to find this puppy use US2016/0035027] has one goal: to outrun the speed demons of Wall Street. The 16-page document was quietly published by the U.S. Patent and Trademark Office in February. Replete with schematic drawings, the filing describes a novel way for “executing synchronized trades in multiple exchanges.” The invention consists of not only sophisticated algorithms and a host of computer servers, but atomic clocks — precisely calibrated to vibrations of irradiated cesium atoms — to sync orders to within a few billionths of a second.
I also highlighted:
Its invention, developed by the firm’s co-chief executive officers, Robert Mercer and Peter Brown, first sends an order to a central server, which breaks it up into multiple smaller orders. Those are then routed to venues that offer the best prices and most liquidity, much the same as brokers do now. But before that happens, the smaller orders are sent to servers located as close to the exchanges as possible, along with instructions on the precise times they should be executed. The co-located servers sync their transactions so HFT firms won’t have enough time to identify an order on one exchange and then race to another to trade against it. A crucial part of the system is the optical, atomic or GPS clocks that will be used synchronize those orders. Renaissance says in its application that GPS clocks are accurate to within nanoseconds and any time differences between them are “too small to be perceived” by HFT firms.
How much is an atomic clock centric trading system? A well heeled Wall Street firm can afford these systems. The reason is that the infrastructure to pull off this near real time approach is out of reach for many outfits.
How close to real time is a search and content processing system? You can believe the index is up to date, but I would suggest that you are looking at last week’s leftover barbeque chicken in a Tupperware box. Saying “real time” is less difficult and expensive than shaving milliseconds off an online action. Enjoy that chicken?
Stephen E Arnold, July 14, 2016
Remote Companies: No Office, No Physical Presence
June 16, 2016
I came across a database of companies with [a] no office and no physical presence or [b] jobs for people who don’t have to show up at a facility. The idea is that outsourcing and the gig economy make it possible for a firm to do provide products and services without folks hanging around the coffee machine or hunching over a laptop in an open work space. (Commutes are bummers. Years ago, I learned to read the Washington Post whilst creeping along the jammed expressway from Rockville to Bethesda. My really cheap Pinto was perfect for this type of slog.) You can access Remotebase and browse job listings and the names of the companies for free. It seems the service is free. Most of the companies whose links I explored were new to me; for example, the canine inspired Corgibytes and the Google killer DuckDuckGo. Think of it. Work from a public library, a Pinto or equivalent, a coffee shop, or (heaven forbid) an airport lounge.
Stephen E Arnold, June 16, 2016
Quote to Note, Nay, Memorize from Yahoo
February 23, 2016
I read “Yahoo Forms Panel to Explore Strategic Options.” I looked past the notion that a committee can come up with options beyond sell this puppy. I did find a gem of a phrase in the write up. Here’s the keeper:
Mayer said in a statement, while emphasizing that everyone at Yahoo wanted to return the “iconic company to greatness”.
I like the notion of Yahoo as an icon. I think directories from the 1990s should be icons. The concept of greatness is a good one. My hunch is that revenue would be a help for whatever Yahoo is going to return to.
My thought is that the Yahooligans and the Xoogler look more like a business school case study for a course covering Touchstones in Internet Successes and Failures. I am debating whether Yahoo is a success or a failure. I have decided. Failure: Geocities, the Google GoTo Overture legal matter, the Semel reign, the CEO with sketchy credentials, the internal squabbling, et al.
Yep, let’s have a committee meeting. Great idea. Thank goodness I have IBM Watson to pick up the slack created by the silliness at Yahoo.
Stephen E Arnold, February 23, 2016
Need to Ace Your Alphabet Google Interview?
January 25, 2016
I know some “real” journalists and employees of “real” publishing outfits like Pearson want to work at the Alphabet Google thing? Sounds interesting, doesn’t it.
Navigate to “41 of the Trickiest Questions Google Will Ask You in a Job Interview” to get the inside scoop on landing that perfect gig. Just think. T shirts, mouse pads with the Google logo, maybe a day at Google I/O.
I have a copy of the GLAT or Google Labs Aptitude Test. I believe this bit of tomfoolery was retired years ago, and I can say with some confidence that the questions presented by the UK newspaper may not do the trick for unemployed journalists and/or professional publishers riffed by outfits like Pearson.
Here are three questions to land you in your dream cubicle:
Math and probability: A coin was flipped 1000 times and there were 560 heads. Do you think the coin is biased? — Quantitative Analyst, September 2015
Search: How many ways can you think of to find a needle in a haystack? — Business Associate, May 2014
Fortune telling: How do you think the digital advertising world will change in the next 3 years? — Creative Director, January 2016
Those who want the thrill of the Alphabet Google life are now able to begin their preparatory work.
Oh, here’s another question:
Self entertainment for life’s sound track: If you could only choose one song to play every time you walked into a room for the rest of your life, what would it be? — Associate Account Strategist Interview, March 2014
Tip: Don’t choose a track from an Apple service.
Stephen E Arnold, January 25, 2016
Big Data and Enterprise Search: The Caution Lights Are Flashing
December 15, 2015
I read “How You Should Explain Big Data to Your CEO.” The write up included a link which triggered thoughts of how enterprise search dug itself a hole and climbed. Unable to extricate itself from a problem enterprise search vendors created, the entire sector has been marginalized. In some circles, enterprise search is essentially a joke. “Did you hear about the three enterprise search vendors who walked into a bar?” The bartender says, “What is this? Some kind of joke?”
The link pointed me to a Slideshare (owned by the email and job hunting champion LinkedIn). That presentation, “5 Signs Your Big Data Project is Doomed to Fail,” could have been borrowed from one of my talks about enterprise search in 2001. It was not, but the basic message was identical: Big Data has created a situation in which there are some challenges here and now.
The presentation was prepared by Qubole (maybe pronounced cue ball?). Qubole is a click to query outfit. This means that reports from Big Data are easy to generate.
Here are the problems Big Data faces:
- Failed implementations. Qubole asserts that 87 percent of the Big Data implementations are flops
- 73 percent of executive describe the Big Data project as flop
- 45 percent of Big Data projects are completed
These data are similar to the results of “satisfaction” with enterprise search solutions.
Why? Qubole asserts:
- Inaccurate project scope
- Inadequate management support
- No business case
- Lack of talent (in search the talent may be present but overestimates its ability to deal with enterprise search technologies and processes)
- “Challenging tools.” I think this means that in the Big Data world there are lots of complexities.
What can one charged with either search or Big Data tasks do with this information?
My view is, “Ignore it.”
The “can do” spirit carries professionals forward. Hiring a consultant provides some job protection but does little to reverse the failure and disappointment rate.
My view is that the willingness of executives to grab at a magic solution presented by a showman marketer overrides failure date. The arrogance of those involved create a “that won’t happen to us” belief.
Who is to blame? The company for believing in baloney? The marketer for painting a picture and showing a Hollywood style demo? The developers who created the Big Data solution, knowing that chunks were not complete or just did not work before the ship date? The in house engineers who lacked self knowledge to understand their own limitations?
Everyone is in the hole with the enterprise software vendors. The hole is deep. Magic solutions are difficult to pin down. The future of Big Data is likely to parallel to some degree the dismal track record of enterprise search. Fascinating. I can hear the mid tier consultants and the handful of remaining enterprise search vendors asserting that Qubole’s points are not applicable to their specific situation.
Yep, and I believe in the tooth fairy and Santa.
Stephen E Arnold, December 15, 2015
Deleting Data: Are They Really Gone?
November 17, 2015
I read “Gawker Media’s Data Guru Presents the Case for Deleting Data.” The main idea is that hoarding permits a reality TV program. Hoarding data may not be good TV.
The write up points out that data cleaning is not cheap. Storage also costs money.
A Gawker wizard is quoted as saying:
We effectively are setting traps in our data sets for our future selves and our colleagues… Increasingly, I find that eliminating this data from our databases is the best solution. Gawker’s traffic data is maintained for just a few months. In our own logs and databases, we only have traffic data since February. and even that’s of limited use: We’ll toss some of it before the end of the year.
Seems reasonable. However, there may be instances when dumping or just carelessly overwriting log files might not be expedient or legal. For example, in one government agency, the secretary’s “bonus” depends on showing how Internet site usage relates to paperwork reduction. The idea is that when a “customer” of the government uses a Web site and does not show up in person at an office to fill out a request, the “customer” allegedly gets better service and costs, in theory, should drop. Also, some deals require that data be retained. You can use your imagination if you are an ISP in a country recently attacked by terrorists and your usage logs are “disappeared.” SEC and IRS retention guidelines? Worth noting in some cases.
The question is, “Are data really gone once deleted?” The fact of automatic backups, services in the middle routinely copying data, and other ways of creating unobserved backups may mean that deleted data can come back to life.
Pragmatism and legal constraints as well as the “men in the middle” issue can create zombie data, which, unlike the fictional zombies, can bite.
Stephen E Arnold, November 17, 2015
Billing and Meetings Remain Easier Than Usable Digital Systems
November 11, 2015
I have bumped against digital initiatives in government and industry a number of times. The experience and understanding I gained were indispensible. Do you remember the “paperless office”? The person attributed with creating this nifty bit of jargon was, if memory serves me, Harvey Poppel. I worked with the fellow who coined this term. He also built a piano. He became an investment wizard.
Later I met a person deeply involved with reducing paperwork in the US government. The fellow, an authoritative individual, ran an advertising and marketing company in Manhattan. I recall that he was proud of his work on implementing strategies to reduce dead tree paper in the US government. I am not sure what happened to him or his initiative. I know that he went on to name a new basketball arena, selecting a word in use as the name of a popular vitamin pill.
Then a mutual acquaintance explained the efforts of an expert who wrote a book about Federal digitalization. I enjoyed his anecdotes. I was, at the time, working as an advisor to a government unit involved in digital activities, but the outfit ran on paper. Without paper, the Lotus Notes system could not be relied upon to make the emails and information about the project available. The fix? Print the stuff on paper. The idea was to go digital, but the information highway was built on laser printer paper.
I thought about these interactions when I read “A Decade into a Project to Digitize U.S. Immigration Forms, Just 1 is Online.” (If the link is dead, please, contact the dead tree publisher, not me.)
According the article:
Heaving under mountains of paperwork, the government has spent more than $1 billion trying to replace its antiquated approach to managing immigration with a system of digitized records, online applications and a full suite of nearly 100 electronic forms. A decade in, all that officials have to show for the effort is a single form that’s now available for online applications and a single type of fee that immigrants pay electronically. The 94 other forms can be filed only with paper.
I am not surprised. The article uses the word “mismanaged” to describe the process upon which the development wheels would turn.
The write up included a quote to note:
“You’re going on 11 years into this project, they only have one form, and we’re still a paper-based agency,’’ said Kenneth Palinkas, former president of the union that represents employees at the immigration agency. “It’s a huge albatross around our necks.”
What’s interesting is that those involved seem to be trying very hard to implement a process which puts data in a database, displays information online, and reduces the need for paper, the stuff from dead trees.
The article suggests that one vendor (IBM) was involved in the process:
IBM had as many as 500 people at one time working on the project. But the company and agency clashed. Agency officials, for their part, held IBM responsible for much of the subsequent failure, documents show.
The company’s initial approach proved especially controversial. Known as “Waterfall,” this approach involved developing the system in relatively long, cascading phases, resulting in a years-long wait for a final product. Current and former federal officials acknowledged in interviews that this method of carrying out IT projects was considered outdated by 2008.
Several observations are warranted, but these are unlikely to be particularly life affirming:
- The management process is usually not focused on delivering a functioning system. The management process is designed to permit billing and cause meetings. The actual work appears to be cut off from these administrative targets of having something to do and sending invoices for services rendered.
- Like other interesting government projects such as the upgrading of the IRS or the air traffic control system, figuring out what to do and how to do it are sufficiently complex that everyone involved dives into details, political considerations, and briefings. Nothing much comes from these activities, but they constitute “work” so day to day, week to week, month to month, and year to year process becomes its own goal. The new system remains an abstraction.
- No one working on a government project, including government professionals and contractors, has responsibility to deliver a solution. Projects become a collection of fixes, which are often demonstrations of a small scale function. The idea that a comprehensive system will actually deliver a function results in software quite similar to the famous HealthCare.gov service.
I am tempted to mention other US government initiatives. I won’t. Shift to the United Kingdom. That country has been working on its National Health Service systems for many years. How similar have been the initiatives to improve usability, functionality, and various reductions. These have ranged from cost reduction to waiting time reduction. The project is not that different from US government efforts.
What’s the fix?
Let me point out that digitization, computerization, and other Latinate nominatives are fated to remain in a state of incompletion. How can one finish when when the process, not the result, is the single most important objective.
I heard that some units of Angela Merkel’s government are now using traditional typewriters. Ah, progress.
Stephen E Arnold, November 11, 2015
Watson Weakly: IBM Case Manager Includes Watson
October 28, 2015
I read “IBM Case manager Provides Tailored Content Management.” For those of you not keeping track of IBM’s product line, may I share with you that IBM Case Manager is a wrapper perched on top of FileNet?
Remember FileNet?
In 2006, yep, nine short years ago, IBM purchased FileNet for $1.6 billion. I stumbled literally upon FileNet when I was doing some work for one of the financial outfits who once found me mildly amusing. FileNet was founded in 1982. The company scanned checks and other documents, stored the images on optical discs, and made the contents searchable—sort of.
The hardware was pure 1982: Big machines, big scanners, and lots of humans doing tasks. Over time, FileNet updated its human dependent system to become more automated. FileNet was a proprietary set up and required lots of engineers, programmers, and specialists to set up the system and keep it humming along at 2 am when most back office operations were performed in the 1980s.
Enter IBM.
FileNet is still available. But IBM has created applications which are designed to make the system more saleable in today’s market. The IBM Case Manager includes FileNet and workflow, collaboration, and compliance tools. You can now run FileNet from a mobile device. When I first stubbed my toe on a giant scanning system, folks were using nifty green screens. Progress.
The 1980s are gone. IBM now delivers a case manager. Keep in mind, gentle reader, that case management is a solution keenly desired by many in law enforcement and certain intelligence disciplines. The US government continues to search for a case management system that meets its various units’ requirements. I would suggest that some of the products available as commercial off the shelf software do not do the job. But let’s focus on what the article reveals about IBM Case Manager.
The article points out that IBM Case Manager includes these components:
- A unified interface. Always good for a busy user.
- A data capture and parsing module.
- Information life cycle tools. The idea is that one can comply with Federal regulations and make darned sure information has a “kill on” date.
- A content manager which “provides features for capture, workflow, collaboration, and compliance on both mobile and desktop [devices].”
- SoftLayer which makes IBM Case Manager a cloud application. But licensees can install the system on premises or use a hybrid approach which can be exciting for engineers and investigators.
But the big news in the article is contained in this passage, which I circled in dollar bill green:
Analytics, which is a set of packages that includes IBM Watson, which can glean insight from business content, present that insight in the right context, and identify patterns and trends.
I did not know that IBM Case Management included Watson. My understanding was that Watson was the new IBM; therefore, Watson includes IBM Case Management.
Perhaps this is a minor point, but since we are dealing with technology from the 1980s, open source code, and wrappers which add a range of user experience features, I think getting the horse and cart lined up correctly can be helpful at times.
Another remarkable revelation in the article is that IBM Case Manager is for “enterprise of all sizes.” There you go. The local Harrod’s Creek, Kentucky, car wash and grocery can use IBM Case Management with Watson to help the proprietors deal with their information demands.
May I suggest that FileNet, regardless of its name, is appropriate for outfits like banks, hospitals, and meaty government agencies.
I also learned:
IBM Case Manager can be used to monitor social media sites to get a reading on public sentiment on a given subject or brand. Case Manager can also provide collaboration with social media platforms.
I have updated my Watson files and noted that IBM Case Manager includes Watson or is it the other way around?
Stephen E Arnold, October 28, 2015
Netflix: Slow Subscriber Growth Due to Things Netflix Cannot Control
October 15, 2015
I like the idea of blaming what some MBA whiz called exogenous events. The idea is that hapless, yet otherwise capable senior managers, are unable to deal with the ups and downs of running a business. In short, an exogenous event is a variation on “it’s not my fault,” “there’s little I can do,” and “let’s just muddle forward.” The problem is that hunting for scapegoats is not a way to generate revenue. Wait. One can raise subscription fees.
I read “Netflix Is Blaming Slow US Growth on the Switch to Chip-Based Credit Cards.” The write up references a letter, allegedly written by the Netflix top dog. I noted this passage:
In his letter to investors, Netflix CEO Reed Hastings partially blamed America’s recent switch to chip-enabled credit cards. As credit card companies send new cards to their customers, some have been issuing new numbers, as well. And if people forget to update their credit card number with Netflix, they can’t pay their bill and become what Hastings called “involuntary churn.”
I like that involuntary churn. I remember working on a project for a telecommunications company in which churn was a burr under the saddle of some executives. Those pesky customers. Darn it.
The write up ignores the responsibility of management to deal with exogenous events. When a search system fails, is it the responsibility of customers to fix the system. Nah, users just go to a service that seems to work.
I interpreted this alleged explanation and the article’s willingness to allow Netflix’s management to say, in effect, “Hey, this is not something I can do anything about.” If not the top dog, who takes responsibility? Perhaps the reason is not chip enabled credit cards? Perhaps users are sending Netflix a signal about sometimes unfindable content, clunky search, and a lack of features. Not everyone is a binge watcher. Some folks look for Jack Benny films or other entertainment delights. When these are not available, perhaps some look elsewhere. See and you shall find often delivers the goods.
Stephen E Arnold, October 15, 2015
Amazon Updates Sneaker Net
October 8, 2015
I remember creating a document, copying the file to a floppy, and then walking up one flight of steps to give the floppy to my boss. He took the floppy, loaded it into his computer, and made changes. A short time later he would walk down one flight of steps, hand me the floppy with his file on it, and I would review the changes.
I thought this was the cat’s pajamas for two reasons:
- I did not have to use ledger paper, sharpen a pencil, and cramp my fingers
- Multiple copies existed so I no longer had to panic when I spilled my Fresca across my desk.
Based on the baloney I read every day about the super wonderful high speed, real time cloud technology, I was shocked when I read “Snowball’s Chance in Hell? Amazon Just Launched a Physical Data Transfer Service.” The news struck me as more important than the yap and yammer about Amazon disrupting cloud business and adding partners.
Here’s the main point I highlighted in pragmatic black:
A Snowball device is ordered through the AWS Management Console and is delivered to site within a few days; customers can order multiple devices and devices can be run in parallel. Described as coming in its “own shipping container” (it doesn’t require packing or unpacking) the Snowball is entirely self-contained, complete with 110 Volt power, a 10 GB network connection on the back and an E Ink display/control panel on the front. Once received it’s simply a matter of plugging the device in, connecting it to a network, configuring the IP address, and installing the AWS Snowball client; a job manifest and 25 character unlock code complete the task. When the transfer of data is complete the device is disconnected and a shipping label will automatically appear on the E Ink display; once shipped back to Amazon (currently only the Oregon data center is supporting the service, with others to follow) the data will be decrypted and copied to S3 bucket(s) as specified by the customer.
There you go. Sneaker net updated with FedEx, UPS, or another shipping service. Definitely better than carrying an appliance up and down stairs. I was hoping that individuals participating in the Mechanical Turk system would be available to pick up an appliance and deliver it to the Amazon customer and then return the gizmo to Amazon. If Amazon can do Etsy-type stuff, it can certainly do Uber-type functions, right?
When will the future arrive? No word on how the appliance will interact with Amazon’s outstanding search system. I wish I knew how to NOT out unpublished books or locate mysteries by Japanese authors available in English. Hey, there is a sneaker net. Focus on the important innovations.
Stephen E Arnold, October 8, 2015
-
- Subscribe to Beyond Search
Feature archive
News archive
-
