A Digital Flea Market Tests Smart Software
November 14, 2024
Sales platform eBay has learned some lessons about deploying AI. The company tested three methods and shares its insights in the post, “Cutting Through the Noise: Three Things We’ve Learned About Generative AI and Developer Productivity.” Writer Senthil Padmanabhan explains:
“Through our AI work at eBay, we believe we’ve unlocked three major tracks to developer productivity: utilizing a commercial offering, fine-tuning an existing Large Language Model (LLM), and leveraging an internal network. Each of these tracks requires additional resources to integrate, but it’s not a matter of ranking them ‘good, better, or best.’ Each can be used separately or in any combination, and bring their own benefits and drawbacks.”
The company could have chosen from several existing commercial AI offerings. It settled on GitHub Copilot for its popularity with developers. That and the eBay codebase was already on GitHub. They found the tool boosted productivity and produced mostly accurate documents (70%) and code (60%). The only problem: Copilot’s limited data processing ability makes it impractical for some applications. For now.
To tweak and train an open source LLM, the team chose Code Llama 13B. They trained the camelid on eBay’s codebase and documentation. The resulting tool reduced the time and labor required to perform certain tasks, particularly software upkeep. It could also sidestep a problem for off-the-shelf options: because it can be trained to access data across internal services and within non-dependent libraries, it can get to data the commercial solutions cannot find. Thereby, code duplication can be avoided. Theoretically.
Finally, the team used an Retrieval Augmented Generation to synthesize documentation across disparate sources into one internal knowledge base. Each piece of information entered into systems like Slack, Google Docs, and Wikis automatically received its own vector, which was stored in a vector database. When they queried their internal GPT, it quickly pulled together an answer from all available sources. This reduced the time and frustration of manually searching through multiple systems looking for an answer. Just one little problem: Sometimes the AI’s responses were nonsensical. Were any just plain wrong? Padmanabhan does not say.
The post concludes:
“These three tracks form the backbone for generative AI developer productivity, and they keep a clear view of what they are and how they benefit each project. The way we develop software is changing. More importantly, the gains we realize from generative AI have a cumulative effect on daily work. The boost in developer productivity is at the beginning of an exponential curve, which we often underestimate, as the trouble with exponential growth is that the curve feels flat in the beginning.”
Okay, sure. It is all up from here. Just beware of hallucinations along the way. After all, that is one little detail that still needs to be ironed out.
Cynthia Murrell, November 14, 2024
Comments
Got something to say?
-
- Subscribe to Beyond Search
Feature archive
News archive
-
