Why I Moved ML Workloads to AWS EKS

·1 min read·

EC2, then ECS, then EKS. The evolution of how I run ML inference at scale.

When we started running ML inference at Sabivox, we ran models on EC2. It worked. It was also operationally expensive.

We moved to ECS. Better. Then to EKS for the ML workloads specifically: GPU instance scheduling, granular resource quotas for different memory profiles, and HPA integrating with custom metrics (inference queue depth).

EKS is not simple. The learning curve and operational overhead are non-trivial. But for production LLM inference at scale, it was the right answer.

Choose your orchestration layer based on workload characteristics, not marketing.

— Dick Bassey | DevDick | 2024

Share this article
← Back to all writing