Ambassador Labs Founder Richard Li’s blog post about his experience building an AI Application summarized below:
The advent of ChatGPT has undeniably revolutionized the software landscape, prompting many to delve into the world of artificial intelligence. One such individual embarked on a year-long project to develop an AI assistant for their past CEO self, a pedagogical exercise designed to provide answers, status reports, and summaries. Reflecting on this experience, they shared seven crucial lessons learned.
Early Scale-Up Challenges and the Impact of AI Tools:
A significant early finding was that problems typically associated with scaling up emerged much sooner than anticipated. However, the use of AI-powered tools like Copilot proved to be a notable productivity enhancer, not necessarily making development faster or slower, but fundamentally different.
Seven Core Takeaways:
- AI Programming is Stochastic: Unlike traditional deterministic programming, working with AI involves a stochastic process of experimentation. “Programming” an AI entails numerous trials, adjusting inputs and parameters to improve performance. These adjustments fall into four main categories:
- Prompt Engineering: Refining the initial prompt using techniques like few-shot prompting, retrieval augmented generation, and chain-of-thought prompting.
- Task/Domain Fine-Tuning: Utilizing domain-specific data to fine-tune the model, often using methods like LoRA.
- Preference Tuning: Steering the model’s behavior by optimizing for human-preferred outputs.
- Hyperparameter Tuning: Adjusting parameters like learning rates and batch sizes to improve training efficiency and performance.
- Data Quality is Paramount: While fine-tuning and preference tuning are mechanically straightforward, creating a high-quality training dataset is a significant undertaking. This is not a one-time task but requires a continuous pipeline for data transformation, formatting, and quality evaluation.
- Model Evaluation is Crucial: Just as software quality depends on test coverage, an AI model’s effectiveness hinges on its evaluation strategy. Traditional validation sets often fail to capture real-world complexities like edge cases and outliers. Off-the-shelf evaluation solutions were found to be immature, leading to the development of a bespoke system combining human review and LLM-as-judge, though even this approach was not entirely satisfactory.
- Trust and Quality are the Top Priorities: Echoing the sentiment of Ritendra Datta of Databricks, quality, followed by performance and reliability, is the most critical long-term factor for AI product success. While creating a compelling LLM demo is relatively easy, ensuring real-world performance is far more challenging. Even tech giants like Apple have encountered issues with hallucinations in deployed AI features, highlighting the need for continuous experimentation and evaluation.
- The Training Pipeline is the Core IP: The focus has shifted from the AI model itself to the training pipeline as the true intellectual property. This pipeline encompasses everything from data acquisition and transformation to fine-tuning and evaluation. As issues arise, new strategies are incorporated into this pipeline, making its rapid iteration crucial for AI success.
- AI is Another Distributed System: An AI application typically comprises a database, a logic application, and an LLM. This constitutes a distributed system, subject to the inherent challenges and fallacies of such systems. The high latency of LLMs, especially with retrieval augmented generation, necessitates a different approach to distributed system design. Traditional resilience patterns like timeouts and retries are insufficient. An asynchronous architecture with a robust task management system is better suited for handling LLM latency, ensuring application responsiveness and providing built-in resilience.
- Beware the AI Library Hype: While numerous libraries promise to simplify AI development, they often fall short in practical application. Issues like missing implementations and poor ecosystem integration can hinder progress. It’s often more efficient to implement functionality directly using lower-level abstractions like PyTorch or Transformers.
Final Thoughts:
The AI landscape is evolving at an unprecedented pace, far exceeding the speed of cloud-native technologies. The most effective way to learn about AI is through hands-on experience. With accessible tools like ChatGPT and Claude, building AI applications is more attainable than ever, offering surprising progress with relatively little initial effort.
