In the last decade, the AI research and implementation community has made great fast-moving progress from advanced AI language models to AI-powered vision technology. However, when it comes to deploying these models at scale, the challenges are far from academic. The path from a well-performing model in a lab to a reliable, highly effective and continuously learning system in real time production many a times gets caught up with infrastructural, operational, and ethical complexities.
“There is a wide gap between training a model using common test datasets and deploying it in the real world where secure decision making and interpretability matter just as much as accuracy,” says Srinidhi M, Machine Learning (ML) Acceleration engineer at Cruise.
Complexity at the forefront
Deployment is no longer just about wrapping a model packaged for real-time use over the web. Today’s production-grade AI involves complex pipelines that include- model performance tracking and lifecycle management, optimizing the usage of the hardware and unavailability of built-in protections to handle failures instantly.
In autonomous vehicles, a few milliseconds here and there could mean the difference between a safe stop and a critical failure. Srinidhi, a Gold Winner at the Globee® Awards for Artificial Intelligence explains, “In safety-critical systems like self-driving cars, inference time and robustness are just as important as model accuracy. We spend a lot of time optimizing for low-latency deployment across distributed systems.
Infrastructure
AI models are getting larger, but that doesn’t mean the deployment infrastructure can grow at the same pace. Engineers are increasingly tasked with making models faster and smaller, without sacrificing performance. “There’s a growing emphasis on techniques like model quantization, distillation, and compiler-level optimizations. It’s not just about getting the right prediction, it’s about getting it at the right time and at scale,” he adds.
Srinidhi, who has worked across cloud platforms like IBM and AWS and now focuses on ML acceleration at Cruise, highlights how infrastructure decisions can make or break a deployment. “One must think and create holistically, especially to topics like the computer, memory, bandwidth and how your model interacts with the broader system. Optimizing any one part in isolation rarely solves the problem,” he notes.
His recent book, Carbon-Light Compute: A National Playbook for Energy-Efficient AI, outlines actionable strategies for building AI systems that are not just fast and accurate, but also sustainable. It reflects his broader mission to align performance engineering with energy efficiency—a topic gaining urgency as AI scales globally.
Bridging the divide
The whole AI ecosystem is unfortunately having to grapple with a misalignment between those designing cutting-edge algorithms and those engineering them for real-world use. “The industry needs more people who can speak both languages — of deep learning and distributed systems. That’s where the magic of real-world AI happens,” Srinidhi, a session chair at IEEE conferences, says.
As AI becomes important for businesses to grow and function, the focus is shifting from standalone models to strong, scalable systems that work across teams. Deploying AI at scale needs more than smart models. It takes efficient systems, real-world awareness and cross-functional collaboration.
