When AI Goes Off the Rails: Who Gets Harmed?
September 13, 2021
One of the worst things about modern job hunting is the application process. Hiring systems require potential applicants to upload their resume, then retype their resume into specified fields. It is a harrowing process that would annoy anyone. What is even worse is that most resume are rejected thanks to resume-scanning software. The Verge details how bad automation harms job seekers in the story, “Automated Hiring Software Is Mistakenly Rejecting Millions of Viable Job Candidates.”
Automated resume-scanning software rejects viable candidates. The software accidentally rejecting the candidates created a new pocket of qualified workers, who are locked out of the job market. Seventy-five percent of US employers use resume software and it is one of the biggest factors harming job applicants. There are many problems with resume software and they appear to stem from how they are programmed to “evaluate” candidates:
“For example, some systems automatically reject candidates with gaps of longer than six months in their employment history, without ever asking the cause of this absence. It might be due to a pregnancy, because they were caring for an ill family member, or simply because of difficulty finding a job in a recession. More specific examples cited by one of the study’s author, Joseph Fuller, in an interview with The Wall Street Journal include hospitals who only accepted candidates with experience in “computer programming” on their CV, when all they needed were workers to enter patient data into a computer. Or, a company that rejected applicants for a retail clerk position if they didn’t list “floor-buffing” as one of their skills, even when candidates’ resumes matched every other desired criteria.”
Employers use rigid criteria to filter job applicants. On one hand, resume software was supposed to make hiring easier, but employers are inundated with hundreds of resumes with an average of 250 applicants per job. Automation in job hiring is not slowing down and the industry is projected to be worth $3.1 billion by 2025.
How will off-the-rails AI apps be avoided or ameliorated? My hunch is that they cannot.
Whitney Grace, September 13, 2021
Silicon Valley: Fraud or Fake Is an Incorrect Characterization
September 10, 2021
I read “Elizabeth Holmes: Has the Theranos Scandal Changed Silicon Valley?” The write up contains a passage I found interesting; to wit:
In Silicon Valley, hyping up your product – over-promising – isn’t unusual…
Marketing is more important than the technology sold by the cash hype artists. Notice that I don’t use the word “entrepreneur,” “innovator,” “programmer,” or the new moniker “AIOps” (that’s artificial intelligence operations).
The Theranos story went wrong because there was not a “good enough” method provided. The fact that Theranos could not cook up a marginally better way of testing blood is less interesting than the fact about the money. She had plenty of money, and her failure is what I call the transition from PowerPoint to “good enough.”
Why not pull a me-too and change the packaging? Why not license a method from Eastern Europe or Thailand and rebrand it? Why not white label a system known to work, offer a discount, and convince the almost clueless Walgreen’s-type operation that the Zirconia was dug out of a hole in a far-off country.
Each of these methods has been used to allow an exit strategy with honor and not a career-ending Tesla-like electric battery fire which burns for days.
The write up explains:
Particularly at an early stage, when a start-up is in its infancy, investors are often looking at people and ideas rather than substantive technology anyway. General wisdom holds that the technology will come with the right concept – and the right people to make it work. Ms Holmes was brilliant at selling that dream, exercising a very Silicon Valley practice: ‘fake it until you make it’. Her problem was she couldn’t make it work.
The transgression, in my opinion, was a failure to use a me-too model. That points to what I call a denial of reality.
Here are some examples of how a not-so-good solution has delivered to users a disappointing product or service yet flourished. How many of these have entered your personal ionosphere?
- Proprietary app stores which offer mobile software which is malware? The purpose of the proprietary app store is to prevent malfeasance, right?
- Operating systems which cannot provide security? My newsfeed is stuffed full of breaches, intrusions, phishing scams, and cloud vulnerabilities. How about that Microsoft Exchange and Azure security or the booming business of NSO Group-types of surveillance functionality?
- Self-driving vehicles anyone? Sorry, not for me.
- Smart software which is tuned to deliver irrelevant advertising despite a service’s access to browser history, user location, and email mail? If I see one more ad for Grammarly or Ke Chava when I watch a Thomas Gast French Foreign Legion video in German, I may have a stroke. (Smart software is great, isn’t it? Just like ad-supported Web search results!)
- Palantir-type systems are the business intelligence solutions for everyone with a question and deep pockets.
The article is interesting, but it sidesteps the principal reason why Theranos has become a touchstone for some people. The primum movens from my vantage point is:
There are no meaningful consequences: For the funders. For the educational institutions. For the “innovators.”
The people who get hurt are not part of the technology club. Maybe Ms. Holmes, the “face” of Theranos will go to jail, be slapped with a digital scarlet A, and end up begging in Berkeley?
I can’t predict the future, but I can visualize a Michael Milkin-type or Kevin Mitnick-type of phoenixing after walking out of jail.
Theranos is a consequence of the have and have not technology social construct. Technology is a tool. Ms. Holmes cut off her finger in woodworking class. That’s sort of embarrassing. Repurposing is so darned obvious and easy.
More adept pioneers have done the marketing thing and made a me-too approach to innovation work. But it does not matter. This year has been a good one for start ups. Get your digital currency. Embrace AIOps. Lease a self driving vehicle. Use TikTok. No problem.
Stephen E Arnold, September 10. 2021
More AI Bias? Seems Possible
September 10, 2021
Freddie Mac and Fannie Mae are stuck in the past—the mid-1990s, to be specific, when the Classic FICO loan-approval software was developed. Since those two quasi-government groups basically set the rules for the mortgage industry, their reluctance to change is bad news for many would-be home buyers and their families. The Markup examines “The Secret Bias Hidden in Mortgage-Approval Algorithms.” Reporters Emmanuel Martinez and Lauren Kirchner reveal what their organization’s research has uncovered:
“An investigation by The Markup has found that lenders in 2019 were more likely to deny home loans to people of color than to white people with similar financial characteristics — even when we controlled for newly available financial factors the mortgage industry for years has said would explain racial disparities in lending. Holding 17 different factors steady in a complex statistical analysis of more than two million conventional mortgage applications for home purchases, we found that lenders were 40 percent more likely to turn down Latino applicants for loans, 50 percent more likely to deny Asian/Pacific Islander applicants, and 70 percent more likely to deny Native American applicants than similar White applicants. Lenders were 80 percent more likely to reject Black applicants than similar White applicants. These are national rates. In every case, the prospective borrowers of color looked almost exactly the same on paper as the White applicants, except for their race.”
Algorithmic bias is a known and devastating problem in several crucial arenas, but recent years have seen efforts to mitigate it with better data sets and tweaked machine-learning processes. Advocates as well as professionals in the mortgage and housing industries have been entreating Fannie and Freddie to update their algorithm since 2014. Several viable alternatives have been developed but the Federal Housing Finance Agency, which oversees those entities, continues to drag its heels. No big deal, insists the mortgage industry—bias is just an illusion caused by incomplete data, representatives wheedle. The Markup’s research indicates otherwise. We learn:
“The industry had criticized previous similar analyses for not including financial factors they said would explain disparities in lending rates but were not public at the time: debts as a percentage of income, how much of the property’s assessed worth the person is asking to borrow, and the applicant’s credit score. The first two are now public in the Home Mortgage Disclosure Act data. Including these financial data points in our analysis not only failed to eliminate racial disparities in loan denials, it highlighted new, devastating ones.”
For example, researchers found high-earning Black applicants with less debt get rejected more often than white applicants with similar income but more debt. See the article for more industry excuses and the authors’ responses, as well some specifics on mechanisms of systemic racism and how location affects results. There are laws on the books that should make such discrimination a thing of the past, but they are difficult to enforce. An outdated algorithm shrouded in secrecy makes it even more so. The Federal Housing Finance Agency has been studying its AI’s bias and considering alternatives for five years now. When will it finally make a change? Families are waiting.
Cynthia Murrell, September 10, 2021
Researcher Suggests Alternative to Criminalization to Curb Fake News
September 10, 2021
Let us stop treating purveyors of fake news like criminals and instead create an atmosphere where misinformation cannot thrive. That is the idea behind one academic’s proposal, The Register explains in, “Online Disinformation Is an Industry that Needs Regulation, Says Boffin.” (Boffin is British for “scientist or technical expert.”) Dr. Ross Tapsell, director of the Australian National University’s Malaysia Institute, looked at Malaysia’s efforts to address online misinformation by criminalizing its spread. That approach has not gone so well for that nation, one in which much of its civil discourse occurs online. Reporter Laura Dobberstein writes:
“In 2018, Malaysia introduced an anti-fake news bill, the first of its kind in the world. According to the law, those publishing or circulating misleading information could spend up to six years in prison. The law put online service providers on the hook for third-party content and anyone could make an accusation. This is problematic as fake news is often not concrete or definable, existing in an ever-changing grey area. Any fake news regulation brings a whole host of freedom of speech issues with it and raises questions as to how the law might be used nefariously – for example to silence political opponents. … The law was repealed in 2019 after becoming seen as an instrument to suppress political opponents rather than protecting Malaysians from harmful information.”
Earlier this year, though, lawmakers reversed course again in the face of COVID—wielding fines of up to RM100,000 ($23,800 US) and the threat of prison for those who spread false information about the disease. Tapsell urges them to consider an alternate approach. He writes:
“Rather than adopting the common narrative of social media ‘weaponisation’, I will argue that the challenges of a contemporary ‘infodemic’ are part of a growing digital media industry and rapidly shifting information society” that is best addressed “through creating and developing a robust, critical and trustworthy digital media landscape.”
Nice idea. Tapsell points to watchdog agencies, which have already taken over digital campaigns during Malaysian elections, as one way to create this shift. His main push, though, seems to be for big tech companies like Facebook and Twitter to take action. For example, they can publicly call out purveyors of false info. After all, it is harder to retaliate against them than against local researchers and journalists, the researcher notes. He recognizes social media companies have made some efforts to halt coordinated disinformation campaigns and to make them less profitable, but insists there is more they can do. What, specifically, is unclear. We wonder—does Tapsell really mean to leave it to Big Tech to determine which news is real and which is fake? We are not sure that is the best plan.
Cynthia Murrell, September 10, 2021
Another Angle for Protecting Kids Online
September 10, 2021
Nonprofit group Campaign for Accountability has Apple playing defense for seemingly putting kids at risk. MacRumors reports, “Watchdog Investigation Finds ‘Major Weaknesses’ in Apple’s App Store Child Safety Measures.” Writer Joe Rossignol cites the group’s report as he writes:
“As part of its Tech Transparency Project, the watchdog group said it set up an Apple ID for a fictitious 14-year-old user and used it to download and test 75 apps in the App Store across several adult-oriented genres: dating, hookups, online chat, and casino/gambling. Despite all of these apps being designated as 17+ on the App Store, the investigation found the underage user could easily evade the apps’ age restrictions. Among the findings presented included a dating app that presented pornography before asking the user’s age, adult chat apps with explicit images that never asked the user’s age, and a gambling app that allowed the minor to deposit and withdraw money. The investigation also identified broader flaws in Apple’s approach to child safety, claiming that Apple and many apps ‘essentially pass the buck to each other’ when it comes to blocking underage users. The report added that a number of apps design their age verification mechanisms ‘in a way that minimizes the chance of learning the user is underage,’ and claimed that Apple takes no discernible steps to prevent this.”
Ah, buck passing, a time-honored practice. Why does Apple itself not block such content when it knows a user is underaged? That is what the Campaign for Accountability’s executive director would like to know. Curious readers can see more details from the report and the organization’s methodology at its Tech Transparency website.
For its part, Apple points to its parent control features built in to its iOS and iPadOS. These settings let guardians choose what apps can be downloaded as well as the time children may spend on each app or website. The Campaign for Accountability did not have these controls activated for its hypothetical 14-year-old. Don’t parents still bear ultimate responsibility for what their kids are exposed to? Trying to outsource that burden to tech companies and app developers is probably a bad idea.
Cynthia Murrell, September 10, 2021
Smart Software: Boiling Down to a Binary Decision?
September 9, 2021
I read a write up which contained a nuance which is pretty much a zero or a one; that is, a binary decision. The article is “Amid a Pandemic, a Health Care Algorithm Shows Promise and Peril.” Okay, good news and bad news. The subtitle introduces the transparency issue:
A machine learning-based score designed to aid triage decisions is gaining in popularity — but lacking in transparency.
The good news? A zippy name: The Deterioration Index. I like it.
The idea is that some proprietary smart software includes explicit black boxes. The vendor identifies the basics of the method, but does not disclose the “componentized” or “containerized” features. The analogy I use in my lectures is that no one pays attention to a resistor; it just does its job. Move on.
The write up explains:
The use of algorithms to support clinical decision making isn’t new. But historically, these tools have been put into use only after a rigorous peer review of the raw data and statistical analyses used to develop them. Epic’s Deterioration Index, on the other hand, remains proprietary despite its widespread deployment. Although physicians are provided with a list of the variables used to calculate the index and a rough estimate of each variable’s impact on the score, we aren’t allowed under the hood to evaluate the raw data and calculations.
From my point of view this is now becoming a standard smart software practice. In fact, when I think of “black boxes” I conjure an image of Stanford University and the University of Washington professors, graduate students, and Google-AI types which share these outfits’ DNA. Keep the mushrooms in the cave, not out in the sun’s brilliance. I could be wrong, of course, but I think this write up touches upon what may be a matter that some want to forget.
And what is this marginalized issue?
I call it the Timnit Gebru syndrome. A tiny issue buried deep in a data set or method assumed to be A-Okay may not be. What’s the fix? An ostrich-type reaction, a chuckle from someone with droit de seigneur? Moving forward because regulators and newly-minted government initiatives designed to examine bias in AI are moving with pre-Internet speed?
I think this article provides an interest case example about zeros and ones. Where’s the judgment? In a black box? Embedded and out of reach.
Stephen E Arnold, September 9, 2021
Alleged DHS Monitoring of Naturalized Citizens
September 9, 2021
Are the fates of millions of naturalized immigrants are at the mercy of one secretive algorithm run by the Department of Homeland Security and, unsurprisingly, powered by Amazon Web Services?
The Intercept examined a number of documents acquired by the Open Society Justice Initiative and Muslim Advocates through FOIA lawsuits and reports, “Little-Known Federal Software Can Trigger Revocation of Citizenship.” Dubbed ATLAS, the software runs immigrants’ information through assorted federal databases looking for any sign of dishonesty or danger. Journalists Sam Biddle and Maryam Saleh write:
“ATLAS helps DHS investigate immigrants’ personal relationships and backgrounds, examining biometric information like fingerprints and, in certain circumstances, considering an immigrant’s race, ethnicity, and national origin. It draws information from a variety of unknown sources, plus two that have been criticized as being poorly managed: the FBI’s Terrorist Screening Database, also known as the terrorist watchlist, and the National Crime Information Center. Powered by servers at tech giant Amazon, the system in 2019 alone conducted 16.5 million screenings and flagged more than 120,000 cases of potential fraud or threats to national security and public safety. Ultimately, humans at DHS are involved in determining how to handle immigrants flagged by ATLAS. But the software threatens to amplify the harm caused by bureaucratic mistakes within the immigration system, mistakes that already drive many denaturalization and deportation cases.”
DHS appears reluctant to reveal details of how ATLAS works or what information it uses, which makes it impossible to assess the program’s accuracy. It also seems the humans who act on the algorithm’s recommendations have misplaced faith in the accuracy of the data behind it. The article cites a 2020 document:
“It also notes that the accuracy of ATLAS’s input is taken as a given: ‘USCIS presumes the information submitted is accurate. … ATLAS relies on the accuracy of the information as it is collected from the immigration requestor and from the other government source systems. As such, the accuracy of the information in ATLAS is equivalent to the accuracy of the source information at the point in time when it is collected by ATLAS.’ The document further notes that ‘ATLAS does not employ any mechanisms that allow individuals to amend erroneous information’ and suggests that individuals directly contact the offices maintaining the various databases ATLAS uses if they wish to correct an error.”
We are sure that process must be a piece of cake. The authors also report:
“Denaturalization experts say that putting an immigrant’s paper trail through the algorithmic wringer can lead to automated punitive measures based not on that immigrant’s past conduct but the government’s own incompetence. … According to [Muslim Advocates’ Deborah] Choi, in some cases ‘denaturalization is sought on the basis of the mistakes of others, such as bad attorneys and translators, or even the government’s failures in record-keeping or the failures of the immigration system.’ Bureaucratic blundering can easily be construed as a sign of fraud on an immigrant’s part, especially if decades have passed since filling out the paperwork in question.”
Worth monitoring. Atlas may carry important payloads, or blow up on the launch pad.
Cynthia Murrell, September 9, 2021
Techno-Psych: Perception, Remembering a First Date, and Money
September 9, 2021
Navigate to “Investor Memory of Past Performance Is Positively Biased and Predicts Overconfidence.” Download the PDF of the complete technical paper at this link. What will you find? Scientific verification of a truism; specifically, people remember good times and embellish those memory with sprinkles.
The write up explains:
First, we find that investors’ memories for past performance are positively biased. They tend to recall returns as better than achieved and are more likely to recall winners than losers. No published paper has shown these effects with investors. Second, we find that these positive memory biases are associated with overconfidence and trading frequency. Third, we validated a new methodology for reducing overconfidence and trading frequency by exposing investors to their past returns.
The issue at hand is investors who know they are financial poobahs. Mix this distortion of reality with technology and what does one get? My answer to this question is, “NFTs for burned Banksy art.”
The best line in the academic study, in my view, is:
Overconfidence is hazardous to your wealth.
Who knew? My answer is the 2004 paper called “Overconfidence and the Big Five.” I also think my 89-year-old great grandmother who told me when I was 13, “Don’t be over confident.”
I wonder if the Facebook artificial intelligence wizards were a bit too overconfident in the company’s smart software. There was, if I recall, a question about metatagging a human as a gorilla.
Stephen E Arnold, September 9, 2021
Has TikTok Set Off an Another Alarm in Washington, DC?
September 9, 2021
Perhaps TikTok was hoping the recent change to its privacy policy would slip under the radar. The Daily Dot reports that “Senators are ‘Alarmed’ at What TikTok Might Be Doing with your Biometric Data.” The video-sharing platform’s new policy specifies it now “may collect biometric identifiers and biometric information,” like “faceprints and voiceprints.” Why are we not surprised? Two US senators expressed alarm at the new policy which, they emphasize, affects nearly 130 million users while revealing few details. Writer Andrew Wyrich reports,
“That change has sparked Sen. Amy Klobuchar (D-Minn.) and Sen. John Thune (R-S.D.) to ask TikTok for more information on how the app plans to use that data they said they’d begin collecting. Klobuchar and Thune wrote a letter to TikTok earlier this month, which they made public this week. In it, they ask the company to define what constitutes a ‘faceprint’ and a ‘voiceprint’ and how exactly that collected data will be used. They also asked whether that data would be shared with third parties and how long the data will be held by TikTok. … Klobuchar and Thune also asked the company to tell them whether it was collecting biometric data on users under 18 years old; whether it will ‘make any inferences about its users based on faceprints and voiceprints;’ and whether the company would use machine learning to determine a user’s age, gender, race, or ethnicity based on the collected faceprints or voiceprints.”
Our guess is yes to all three, though we are unsure whether the company will admit as much. Nevertheless, the legislators make it clear they expect answers to these questions as well as a list of all entities that will have access to the data. We recommend you do not hold your breath, Senators.
Cynthia Murrell, September 9, 3021
Change Is Coming But What about Un-Change?
September 8, 2021
My research team is working on a short DarkCyber video about automating work processes related to smart software. The idea is that one smart software system can generate an output to update another smart output system. The trend was evident more than a decade ago in the work of Dr. Zbigniew Michalewicz, his son, and collaborators. He is the author of How to Solve It: Modern Heuristics. There were predecessors and today many others following smart approaches to operations for artificial intelligence or what is called by thumbtypers AIOps. The DarkCyber video will become available on October 5, 2021. We’ll try to keep the video peppy because smart software methods are definitely exciting and mostly invisible. And like other embedded components, some of these “modules” will become components, commoditized, and just used “as is.” That’s important because who worries about a component in a larger system? Do you wonder if the microwave is operating at peak efficiency with every component chugging along up to spec? Nope and nope.
I read a wonderful example of Silicon Valley MBA thinking called “It’s Time to Say “Ok, Boomer!” to Old School Change Management.” At first glance, the ideas about efficiency and keeping pace with technical updates make sense. The write up states:
There are a variety of dated methods when it comes to change management. Tl;dr it’s lots of paper and lots of meetings. These practices are widely regarded as effective across the industry, but research shows this is a common delusion and change management itself needs to change.
Hasta la vista Messrs. Drucker and the McKinsey framework.
The write up points out that a solution is at hand:
DevOps teams push lots of changes and this is creating a bottleneck as manual change management processes struggle to keep up. But, the great thing about DevOps is that it solves the problem it creates. One of the key aspects where DevOps can be of great help in change management is in the implementation of compliance. If the old school ways of managing change are too slow why not automate them like everything else? We already do this for building, testing and qualifying, so why not change? We can use the same automation to record change events in real time and implement release controls in the pipelines instead of gluing them on at the end.
Does this seem like circular reasoning?
I want to point out that if one of the automation components operates using probability and the thresholds are incorrect, the data poisoned (corrupted by intent or chance) or the “averaging” which is a feature of some systems triggers a butterfly effect, excitement may ensue. The idea is that a small change may have a large impact downstream; for example, a wing flap in Biloxi could create a flood in the 28th Street Flatiron stop.
Several observations:
- AIOps are already in operation at outfits like the Google and will be componentized in an AWS-style package
- Embedded stuff, like popular libraries, are just used and not thought about. The practice brings joy to bad actors who corrupt some library offerings
- Once a component is up and running and assumed to be okay, those modules themselves resist change. When 20 somethings encounter mainframe code, their surprise is consistent. Are we gonna change this puppy or slap on a wrapper? What’s your answer, gentle reader?
Net net: AIOps sets the stage for more Timnit Gebru shoot outs about bias and discrimination as well as the type of cautions produced by Cathy O’Neil in Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.
Okay, thumbtyper.
Stephen E Arnold, September 8, 2021
-
- Subscribe to Beyond Search
Feature archive
News archive
-
