Synthetic Data: It Sucks, Says Designers
April 25, 2025
Some would argue taking actual customers out of market research is a bad idea. Smashing Magazine supports that reasonable perspective in, “How to Argue Against AI-First Research.” Yes, some AI enthusiasts praise synthetic user testing as a valuable new tool. The practice is exactly what it sounds like—using LLMs to build fake customers and performing market research on them. Admittedly, it is much faster and cheaper than surveying actual humans. But what good is that if the results are bad? Writer Vitaly Friedman explains:
“When ‘producing’ user insights, LLMs can’t generate unexpected things beyond what we’re already asking about. In comparison, researchers are only able to define what’s relevant as the process unfolds. In actual user testing, insights can help shift priorities or radically reimagine the problem we’re trying to solve, as well as potential business outcomes. Real insights come from unexpected behavior, from reading behavioral clues and emotions, from observing a person doing the opposite of what they said. We can’t replicate it with LLMs.”
But budgets are tight. Isn’t synthetic user data better than nothing? No. No it is not. We learn:
“Pavel Samsonov articulates that things that sound like customers might say them are worthless. But things that customers actually have said, done, or experienced carry inherent value (although they could be exaggerated). We just need to interpret them correctly. AI user research isn’t ‘better than nothing’ or ‘more effective.’ It creates an illusion of customer experiences that never happened and are at best good guesses but at worst misleading and non-applicable.”
Not only that, cutting real customers out of the equation means not catching AI errors. And there will be errors. Furthermore, emphasizes Friedman:
“Synthetic testing assumes that people fit in well-defined boxes, which is rarely true. Human behavior is shaped by our experiences, situations, habits that can’t be replicated by text generation alone. AI strengthens biases, supports hunches, and amplifies stereotypes.”
All of which could send marketing dollars down the wrong, unprofitable track. As suspicious as we are of AI hype, even we can admit the tech is good for some things. Market research perhaps is not a core competency.
Cynthia Murrell, April 25, 2025
AI Crawlers Are Bullying Open Source: Stop Grousing and Go Away
April 25, 2025
AI algorithms are built on open source technology. Unfortunately generative AI is harming its mother code explains TechDirt: “AI Crawlers Are Harming Wikimedia, Bringing Open Source Sites To Their Knees, And Putting The Open Web At Risk.” To make generative AI work you need a lot of computer power, smart coding, and mounds of training data. Money can buy coding and power, but (quality) training data is incredibly difficult to obtain.
AI crawlers were unleashed on the Internet to scrap information and use it for training models. The biggest information providers for crawlers are Wikimedia projects and it’s a big problem. Wikimedia, which claims to be “the largest collection of open knowledge in the world,” says most of its traffic is from crawlers and it is eating into costs:
“Since January 2024, we have seen the bandwidth used for downloading multimedia content grow by 50%. This increase is not coming from human readers, but largely from automated programs that scrape the Wikimedia Commons image catalog of openly licensed images to feed images to AI models. Our infrastructure is built to sustain sudden traffic spikes from humans during high-interest events, but the amount of traffic generated by scraper bots is unprecedented and presents growing risks and costs.”
This is bad because it is straining the Wikimedia datacenter and budgetary resources. Wikimedia isn’t the only information source feeling the burn from AI crawlers. News sites and more are being wrung by crawlers for every decimal of information:
“It’s increasingly clear that the reckless and selfish way in which AI crawlers are being deployed by companies eager to tap into today’s AI hype is bringing many sites around the Internet to their knees. As a result, AI crawlers are beginning to threaten the open Web itself, and thus the frictionless access to knowledge that it has provided to general users for the last 30 years.”
Silicon Valley might have good intentions but dollars are more important. (Oh, I am not sure about the “good intentions.”)
Whitney Grace, April 25, 2025
Microsoft and Its Modern Management Method: Waffling
April 23, 2025
No AI, just the dinobaby himself.
The Harvard Business School (which I assume will remain open for “business”) has not addressed its case writers to focus on Microsoft’s modern management method. To me, changing direction is not a pivot; it is a variant of waffling. “Waffling” means saying one thing like “We love OpenAI.” Then hiring people who don’t love OpenAI and cutting deals with other AI outfits. The whipped cream on the waffle is killing off investments in data centers.
If you are not following this, think of the old song “The first time is the last time,” and you might get a sense of the confusion that results from changes in strategic and tactical direction. You may find this GenX, Y and Z approach just fine. I think it is a hoot.
PC Gamer, definitely not the Harvard Business Review, tackles one example of Microsoft’s waffling in “Microsoft Pulls Out of Two Big Data Centre Deals Because It Reportedly Doesn’t Want to Support More OpenAI Training Workloads.”
The write up says:
Microsoft has pulled out of deals to lease its data centres for additional training of OpenAI’s language model ChatGPT. This news seems surprising given the perceived popularity of the model, but the field of AI technology is a contentious one, for a lot of good reasons. The combination of high running cost, relatively low returns, and increasing competition—plus working on it’s own sickening AI-made Quake 2 demo—have proven enough reason for Microsoft to bow out of two gigawatt worth of projects across the US and Europe.
I love the scholarly “sickening.” Listen up, HBR editors. That’s a management term for 2025.
The article adds:
Microsoft, as well as its investors, have witnessed this relatively slow payoff alongside the rise of competitor models such as China’s Deepseek.
Yep, “payoff.” The Harvard Business School’s professors are probably not familiar with the concept of a payoff.
The news report points out that Microsoft is definitely, 100 percent going to spend $80 billion on infrastructure in 2025. With eight months left in the year, the Softies have to get in gear. The Google is spending as well. The other big time high tech AI juggernauts are also spending.
Will these investments payoff? Sure. Accountants and chief financial officers learn how to perform number magic. Guess where? Schools like the HBS. Don’t waffle. Go to class. Learn and then implement big time waffling.
Stephen E Arnold, April 23, 2025
ArXiv: Will Other Smart Software Systems Get “Free” Access? Yeah, Sure
April 21, 2025
Believe it or not, no smart software. Just a dumb and skeptical dinobaby.
Before commenting on Cornell University’s apparent shift of the ArXiv service to the Google Cloud, let me point you to this page:
The page was updated 15 years ago. Now check out the access to
NCSTRL, the Networked Computer Science Technical Reference Library.
CoRR, the Computing Research Repository.
The Open Archives Initiative.
ETRDL, the ERCIM Technical Reference Digital Library.
Cornell University Library Historical Math Book Collection
Cornell University Library Making of America Collection
Hein online Retrospective Law Journals
Yep, 404s, some content behind paywalls, and other data just disappeared because Bing, Google, and Yandex don’t index certain information no matter what people believe or the marketers say.
This orphaned Cornell University Dienst service has “gorged out”; that is, jumped off a bridge to the rocks below. The act is something students know about but the admissions department seems to not be aware of the bound phrase.
I read “Careers at ArXiv.” The post seems to say to me, “We are moving the ArXiv “gray” papers to Google Cloud. Here’s a snippet of the “career” advertisement / news announcement:
We are already underway on the arXiv CE ("Cloud Edition") project. This is a project to re-home all arXiv services from VMs at Cornell to a cloud provider (Google Cloud). There are a number of reasons for this transition, including improving arXiv’s scalability while modernizing our infrastructure. This will not be a simple port of the existing arXiv code base because this project will:
- replace the portion of our backends still written in perl and PHP
- re-architect our article processing to be fully asynchronous, and provide better insight into the processing workflows
- containerize all, or nearly all arXiv services so we can deploy via Kubernetes or services like Google Cloud Run
- improve our monitoring and logging facilities so we can more quickly identify and manage production issues with arxiv.org
- create a robust CI/CD pipeline to give us more confidence that changes we deploy will not cause services to regress
The cloud transition is a pre-requisite to modernizing arXiv as a service. The modernization will enable: – arXiv to expand the subject areas that we cover – improve the metadata we collect and make available for articles, adding fields that the research community has requested such as funder identification – deal with the problem of ambiguous author identities – improve accessibility to support users with impairments, particularly visual impairments – improve usability for the entire arXiv community.
I know Google is into “free.” The company is giving college students its quantumly supreme smart software for absolutely nothing. Maybe a Google account will be required? Maybe the Chrome browser may be needed to give those knowledge hungry college students the best experience possible? Maybe Google’s beacons, bugs, and cookies will be the students’ constant companions? Yeah, maybe.
But will ArXiv exist in the future? Will Google’s hungry knowledge munchers chew through the data and then pull a Dienst maneuver?
As a dinobaby, I liked the ArXiv service, but I also liked the Dienst math repository before it became unfindable.
It seems to me that Cornell University is:
- Saving money at the library and maybe the Theory Center
- Avoiding future legal dust ups about access to content which to some government professionals may reveal information to America’s adversaries
- Intentionally or inadvertently giving the Google control over knowledge flow related to matters of technical and competitive interest to everyone’s favorite online advertising company
- Running a variation of its Dienst game plan.
But I am a dinobaby, and I know zero about Cornell other than the “gorging out” approach to termination. I know even less about the blue chip consulting type thinking in which the Google engages. I don’t even know if I agree that Google’s recent court loss is really a “win” for the Google.
But the future of the ArXiv? Hey, where is that bridge? Do some students jump, fall, or get pushed to their death on the rocks below?
PS. In case your German is rusty “dienst” means duty and possibly “a position of authority” like a leader at Google.
Stephen E Arnold, April xx, 2025
AI and Movies: Better and Cheaper!
April 21, 2025
Believe it or not, no smart software. Just a dumb and skeptical dinobaby.
I am not a movie oriented dinobaby. I do see occasional stories about the motion picture industry. My knowledge is shallow, but several things seem to be stuck in my mind:
- Today’s movies are not too good
- Today’s big budget films are recycles of sequels, pre-quels, and less than equals
- Today’s blockbusters are expensive.
I did a project for a little-time B movie fellow. I have even been to an LA party held in a mansion in La Jolla. I sat in the corner in my brown suit and waited until I could make my escape.
End of Hollywood knowledge.
I read “Ted Sarandos Responds To James Cameron’s Vision Of AI Making Movies Cheaper: “There’s An Even Bigger Opportunity To Make Movies 10% Better.” No, I really did red the article. I cam away confused. Most of my pre-retirement work involved projects whose goal was to make a lot of money. The idea was be clever, do a minimum of “real” work, and then fix up the problems when people complained. The magic formula for some Silicon Valley and high-technology outfits located outside of the Plastic Fantastic World.
This article pits better versus cheaper. I learned:
Citing recent comments by James Cameron, Netflix Co-CEO Ted Sarandos said he hopes AI can make films “10% better,” not just “50% cheaper.”
Well, there you go. Better and cheaper. Is that the winning formula for creative work? The write up quotes Ted Sarandos (a movie expert, I assume) as saying:
Today, you can use these AI-powered tools to enable smaller-budget projects to have access to big VFX on screen.
From my point of view “better” means more VFX which is, I assume, movie talk for visual effects. These are the everyday things I see at my local grocery store. There are super heroes stopping crimes in progress. There are giant alien creatures shooting energy beams at military personnel. There are machines that have a great voice that some AI experts found particularly enchanting.
The cheaper means that the individuals who sit in front of computer screens fooling around with Blackmagic’s Fusion and the super-wonderful Adobe software will be able to let smart software do some of the work. If 100 people work on a big budget film’s VFX and smart software can do the work cheaper, the question arises, “Do we need these 100 people?” Based on my training, the answer is, “Nope. Let them find their future elsewhere.”
The article sidesteps two important questions: Question 1. What does better mean? Question 2. What does cheaper mean?
Better is subjective. Cheaper is a victim of scope creep. Big jobs don’t get cheaper. Big jobs get more expensive.
What smart software will do the motion picture industry is hasten its “re-invention.”
The new video stars are those who attract eyeballs on TikTok- and YouTube-type platforms. The traditional motion picture industry which created yesterday’s stars or “influencers” is long gone. AI is going to do three things:
- Replace skilled technicians with software
- Allow today’s “influencers” to become the next Clark Gabel and Marilyn Monroe (How did she die?)
- Reduce the barrier for innovations that do not come from recycling Superman-type pre-quels, sequels, and less than equals.
To sum up, both of these movie experts are right and wrong. I suppose both can be reskilled. Does Mr. Beast offer a for fee class on video innovation which includes cheaper production and better outputs?
Stephen E Arnold, April 21, 2025
When
Bugs, Debugs, and Rebugs: AI Does Great Work
April 21, 2025
AI algorithms have already been assimilated into everyday technology, but AI still has problems or you could say there’s a bug in their code. TechCrunch tells us that, “AI Models Still Struggle To Debug Software, Microsoft Study Shows.” Large AI Models such as Anthropic, OpenAI, and more are used for programming. Mark Zuckerberg of Facebook plans to deploy AI coding models at his company, while Sundar Pichai, the Google CEO, said that 25% of code is AI generated.
AI algorithms are great at automating tasks, but they shouldn’t be relied on 100% for all programming projects. Microsoft Research released a new study that discovered AI models like Cause 3.7 Sonnet and 03-mini fail to debug problems in SWE-bench Lite, a software development benchmark. Humans still beat technology when it comes to coding. Here’s what the study did and found:
“The study’s co-authors tested nine different models as the backbone for a “single prompt-based agent” that had access to a number of debugging tools, including a Python debugger. They tasked this agent with solving a curated set of 300 software debugging tasks from SWE-bench Lite. According to the co-authors, even when equipped with stronger and more recent models, their agent rarely completed more than half of the debugging tasks successfully. Claude 3.7 Sonnet had the highest average success rate (48.4%), followed by OpenAI’s o1 (30.2%), and o3-mini (22.1%).”
What is the problem? It’s one that AI has faced since it was first programmed: lack of data for training.
More studies show that AI generated code creates security vulnerabilities too. Is anyone surprised? (Just the AI marketers who do not understand why their assertions don’t match reality.)
Whitney Grace, April 21, 2025
Google Is Just Like Santa with Free Goodies: Get “High” Grades, of Course
April 18, 2025
No AI, just the dinobaby himself.
Google wants to be [a] viewed as the smartest quantumly supreme outfit in the world and [b] like Santa. The “smart” part is part of the company’s culture. The CLEVER approach worked in Web search. Now the company faces what might charitably be called headwinds. There are those pesky legal hassles in the US and some gaining strength in other countries. Also, the competitive world of smart software continues to bedevil the very company that “invented” the transformer. Google gave away some technology, and now everyone from the update champs in Redmond, Washington, to Sam AI-Man is blowing smoke about Google’s systems and methods.
What a state of affairs?
The fix is to give away access to Google’s most advanced smart software to college students. How Santa like. According to “Google Is Gifting a Year of Gemini advanced to Every College Student in the US” reports:
Google has announced today that it’s giving all US college students free access to Gemini Advanced, and not just for a month or two—the offer is good for a full year of service. With Gemini Advanced, you get access to the more capable Pro models, as well as unlimited use of the Deep Research tool based on it. Subscribers also get a smattering of other AI tools, like the Veo 2 video generator, NotebookLM, and Gemini Live. The offer is for the Google One AI Premium plan, so it includes more than premium AI models, like Gemini features in Google Drive and 2TB of Drive storage.
The approach is not new. LexisNexis was one of the first online services to make online legal research available to law school students. It worked. Lawyers are among the savviest of the work fast, bill more professionals. When did Lexis Nexis move this forward? I recall speaking to a LexisNexis professional named Don Wilson in 1980, and he was eager to tell me about this “new” approach.
I asked Mr. Wilson (who as I recall was a big wheel at LexisNexis then), “That’s a bit like drug dealers giving the curious a ‘taste’?”
He smiled and said, “Exactly.”
In the last 45 years, lawyers have embraced new technology with a passion. I am not going to go through the litany of search, analysis, summarization, and other tools that heralded the success of smart software for the legal folks. I recall the early days of LegalTech when the most common question was, “How?” My few conversations with the professionals laboring in the jungle of law, rules, and regulations have shifted to “which system” and “how much.”
The marketing professionals at Google have “invented” their own approach to hook college students on smart software. My instinct is that Google does not know much about Don Wilson’s big idea. (As an aside, I remember one of Mr. Wilson’s technical colleague sometimes sported a silver jumpsuit which anticipated some of the fashion choices of Googlers by half a century.)
The write up says:
Google’s intention is to give students an entire school year of Gemini Advanced from now through finals next year. At the end of the term, you can bet Google will try to convert students to paying subscribers.
I am not sure I agree with this. If the program gets traction, Sam AI-Man and others will be standing by with special offers, deals, and free samples. The chemical structure of certain substances is similar to today’s many variants of smart software. Hey, whatever works, right? Whatever is free, right?
Several observations:
- Google’s originality is quantumly supreme
- Some people at the Google dress like Mr. Wilson’s technical wizard, jumpsuit and all
- The competition is going to do their own version of this “original” marketing idea; for example, didn’t Bing offer to pay people to use that outstanding Web search-and-retrieval system?
Net net: Hey, want a taste? It won’t hurt anything. Try it. You will be mentally sharper. You will be more informed. You will have more time to watch YouTube. Trust the Google.
Stephen E Arnold, April 18, 2025
Google Gemini 2.5: A Somewhat Interesting Content Marketing Write Up
April 18, 2025
Just a still alive dinobaby . No smart software involved.
How about this headline: “Google’s Gemini 2.5 Pro Is the Smartest Model You’re Not Using – and 4 Reasons It Matters for Enterprise AI”?
OpenAI scroogled the Google again. First, it was the January 2023 starting gun for AI hype. Now it was the release of a Japanese cartoon style for ChatGPT. Who knew that Japanese cartoons could have blasted the Google Gemini 2.5 Pro launch more effectively than a detonation of a failed SpaceX rocket?
The write up pants:
Gemini 2.5 Pro marks a significant leap forward for Google in the foundational model race – not just in benchmarks, but in usability. Based on early experiments, benchmark data, and hands-on developer reactions, it’s a model worth serious attention from enterprise technical decision-makers, particularly those who’ve historically defaulted to OpenAI or Claude for production-grade reasoning.
Yeah, whatever.
Announcements about Google AI are about as satisfying as pizza with glued-on cheese or Apple’s AI fantasy PR about “intelligence.”
But I like this statement:
Bonus: It’s Just Useful
The headline and this “just useful” make it clear none of Google’s previous AI efforts are winning the social media buzz game. Plus, the author points out that billions of Google dollars have not made the smart software speedy. And if you want to have smart software write that history paper about Germany after WW 2, stick with other models which feature “conversational smoothness.”
Quite an advertisement. A headline that says, “No one is using this” and” it is sluggish and writes in a way that a student will get flagged for cheating.
Stick to ads maybe?
And what about “why it matters to for enterprise AI.” Yeah, nice omission.
Stephen E Arnold, April 18, 2025
Why Is Meta Experimenting With AI To Write Comments?
April 18, 2025
Who knows why Meta does anything original? Amazon uses AI to write snapshots of book series. Therefore, Meta is using AI to write comments. We were not surprised to read “Meta Is Experimenting With AI-Generated Comments, For Some Reason."
Meta is using AI to write Instagram comments. It sounds like a very stupid idea, but Meta is doing it. Some Instagram accounts can see a new icon to the left of the text field after choosing to leave a comment. The icon is a pencil with a star. When the icon is tapped, a new Meta AI menu pops up, and offers a selection of comment choices. These comments are presumed to be based off whatever content the comment corresponds to in the post.
It doesn’t take much effort to write a simple Instagram comment, but offloading the task appears to take more effort than completing the task yourself. Plus, Instagram is already plagued with chatbot comments already. Does it need more? Nope.
Here’s what the author Jake Peterson requests of his readers:
“Writing comments isn’t hard, and yet, someone at Meta thought there was a usefulness—a market—for AI-generated comments. They probably want more training data for their AI machine, which tracks, considering companies are running out of internet for models to learn from. But that doesn’t mean we should be okay with outsourcing all human tasks to AI.
Mr. Peterson suggest that what bugs him the most is users happily allowing hallucinating software to perform cognitive tasks and make decision for people like me. Right on, Mr. Peterson.
Whitney Grace, April 18, 2025
Trust: Zuck, Meta, and Llama 4
April 17, 2025
Sorry, no AI used to create this item.
CNET published a very nice article that says to me: “Hey, we don’t trust you.” Navigate to “Meta Llama 4 Benchmarking Confusion: How Good Are the New AI Models?” The write up is like a wimpy version of the old PC Perspective podcast with Ryan Shrout. Before the embrace of Intel’s intellectual blanket, the podcast would raise questions about video card benchmarks. Most of the questions addressed: “Is this video card that fast?” In some cases, yes, the video card benchmarks were close to the real world. In other cases, video card manufacturers did what the butcher on Knoxville Avenue did in 1951. Mr. Wilson put his thumb on the scale. My grandmother watched friendly Mr. Wilson who drove a new Buick in a very, very modest neighborhood, closely. He did not smile as broadly when my grandmother and I would enter the store for a chicken.
Would someone put an AI professional benchmarked to this type of test? Of course not. But the idea has a certain charm. Plus, if the person dies, he was fooling. If the person survives, that individual is definitely a witch. This was a winner method to some enlightened leaders at one time.
The CNET story says about the Zuck’s most recent non-virtual reality investment:
Meta’s Llama 4 models Maverick and Scout are out now, but they might not be the best models on the market.
That’s a good way to say, “Liar, liar, pants on fire.”
The article adds:
the model that Meta actually submitted to the LMArena tests is not the model that is available for people to use now. The model submitted for testing is called “llama-4-maverick-03-26-experimental.” In a footnote on a chart on Llama’s website (not the announcement), in tiny font in the final bullet point, Meta clarifies that the model submitted to LMArena was ‘optimized for conversationality.”
Isn’t this a GenZ way to say, “You put your thumb on the scale, Mr. Wilson”?
Let’s review why one should think about the desire to make something better than it is:
- Meta’s decision is just marketing. Think about the self driving Teslas. Consequences? Not for fibbing.
- The Meta engineers have to deliver good news. Who wants to tell the Zuck that the Llama innovations are like making the VR thing a big winner? Answer: No one who wants to get a bonus and curry favor.
- Meta does not have the ability to distinguish good from bad. The model swap is what Meta is going to do anyway. So why not just use it? No big deal. Is this a moral and ethical dead zone?
What’s interesting is that from my point of view, Meta and the Zuck have a standard operating procedure. I am not sure that aligns with what some people expect. But as long as the revenue flows and meaningful regulation of social media remains a windmill for today’s Don Quixotes, Meta is the best — until another AI leader puts out a quantumly supreme news release.
Stephen E Arnold, April 17, 2025
-
- Subscribe to Beyond Search
Feature archive
News archive
-
