In a fast-evolving interview on the Free Code Camp podcast, software engineer Landon Gray analyzed the challenges and nuances of working with large language models (LLMs). Gray, often recognized for his contributions to porting Python machine learning libraries to Ruby and advocating for its potential in AI projects, shared key insights into the complex, dynamic field of AI engineering.

Understanding harnesses for large language models

Gray introduced the concept of a "harness" in the context of LLMs, describing it as the infrastructure and tooling that surrounds the model to enhance its usability and output. While raw LLM models like GPT or Claude generate outputs based on prompts, a harness improves reliability, coherence, and efficiency by addressing issues such as hallucination or latency.

For instance, Gray cited Perplexity, a research-oriented tool, as an example of how a robust harness delivers better results than directly interfacing with an LLM. Companies like Perplexity leverage the core of models like Claude but enhance the user experience through sophisticated infrastructure and functionalities built around the model.

“Think of the LLM as raw fuel,” Gray explained. “The harness is the engine that processes that fuel to deliver specific, user-friendly outcomes. Without it, you’re just left with raw, unrefined outputs.”

He noted that relying solely on the base model for API calls could be limiting—users expecting fully refined results without harness infrastructure often find themselves disappointed. Companies that excel provide not just the model but tools for structuring outputs, reducing errors, and optimizing workflows.

Layers of optimization: From tweaking models to creating rigorous pipelines

Gray distinguished between improving underlying LLM models and building accessible tools around them to refine results. Improving foundational models requires immense time and effort, often involving costly retraining processes. In contrast, tweaking or fine-tuning existing models offers faster and more affordable enhancements, which organizations can combine with powerful harnesses to improve user experiences significantly.

Using an analogy, Gray discussed simple code wrappers—layers implemented over an LLM API—to automate query processes or organize outputs. These wrappers can become highly complex systems that elevate performance without requiring substantial knowledge of the underlying model. This quickly deployable approach demonstrates the benefit of tools built around LLMs, ensuring flexibility while controlling costs.

AI engineering vs. data-centric roles

One of the interview’s focal points was Gray’s reflection on the broader shift from data engineering and science into AI engineering. With concepts like machine learning, inference, and prompt engineering gaining traction, Gray addressed misconceptions about the profession.

Data scientists traditionally focus on statistical approaches, like regression analysis or Bayesian methods, often working with structured algorithms to solve problems. Data engineers, meanwhile, handle the "plumbing work" of preparing datasets—cleaning, transferring, and making sense of raw data. Machine learning engineers train models, often creating the foundational LLMs AI systems depend upon.

In contrast, AI engineers, like Gray, take software engineering expertise and adapt these models into actionable applications. Gray noted that “AI engineering” remains vaguely defined and inconsistently applied as a title. At times, it overlaps with machine learning or data engineering, depending on the hiring company.

For Gray, transitioning into AI work meant building on his software development skillset first, then exploring advanced knowledge of machine learning systems. He emphasized that while understanding the theory behind models is vital, AI engineers can still contribute without extensive background in mathematics or retraining foundational models.

Why learn the inner workings of machine learning?

For developers, LLMs often present as black boxes—systems accessible via an API call with little explanation of what happens inside. But according to Gray, this lack of understanding can become a critical limitation when systems underperform.

He recalled a bottleneck experience with a company’s chat application, where high latency broke user expectations. Relying on LLM tools alone failed to solve the problem. Instead, Gray’s knowledge of underlying mechanics, coupled with exploration of academic knowledge and technical debugging, enabled progress where generative models fell short.

“If you only use a model at face value, you’ll eventually reach a point where you’re stuck,” he warned. Without understanding latency or inference issues, developers may fail to diagnose complex problems—or worse, find themselves unable to provide solutions when the model performs poorly in production.

Understanding concepts like fine-tuning and latency optimization enables engineers to go beyond asking the model for solutions. Gray emphasized that this deeper expertise doesn’t just resolve issues; it prevents costly errors during a project’s lifecycle.

The future potential of Ruby in AI

Gray’s passion for applying Ruby to AI engineering opens another conversation entirely. Typically sidelined in favor of Python, Ruby has seen limited adoption in machine learning. However, Gray believes the AI boom, which pivots on advanced APIs, could elevate Ruby’s standing, given its accessible syntax and adaptability. Thanks to initiatives like Gray’s pipeline work, others have started exploring Ruby as a viable alternative in performance-critical environments.

Key takeaways for developers entering AI engineering

Gray stressed that entering the AI engineering field doesn’t always require rewriting foundational models. Developers can instead focus on leveraging and refining existing resources. By understanding key concepts like harnesses, they can augment outputs without the need for costly overhauls.

For anyone pivoting from traditional software engineering: “Start with what feels adjacent to your existing strengths, and reinforce them from there,” Gray advised, encouraging individuals to identify and bridge crucial gaps in their skillset. Large-scale courses like those offered by organizations such as Free Code Camp or specific university extensions could facilitate this growth for newcomers.

Gray’s points reaffirm the importance of balancing hands-on development with technical learning. AI engineering ultimately represents an iterative process between software design, model improvement, and creative problem-solving—skills highly dependent on agile adaptation.

Why understanding the nuances matters

The podcast leaves tech-savvy listeners with a thought-provoking answer to its central question: What happens when the model can’t fix the problem? Gray’s experiences highlight the gap between naïve reliance on tools and enduring creativity embedded in engineering expertise.

Developers looking to advance in AI must recognize that scraping answers by brute force limits scalability. Guided by curiosity and foundational understanding, however, engineers hold the power to refine and redefine entire systems independently—tools that unlock increasingly sophisticated possibilities for automation, customization, and user satisfaction.