Aleks Kuzminskyi
InfraHouse
Self-Hosting an LLM on ECS: Serving, Distribution, Disk, and Autoscaling
Not so long ago I had a customer who needed to serve an open model inside their own AWS account. They already ran ECS for everything, so the question was whether that was enough, or whether they had to bring in Kubernetes or a managed ML platform first. It was enough. In this talk I will show how we put Qwen2.5-7B behind an ALB on ECS GPU instances with vLLM, and what broke on the way to production. The build was the easy part. Three things surprised us, and in each case the measurement disagreed with the obvious answer. Distribution: a scale-out has to get 15 GB of weights onto every node. A peer-to-peer swarm looked like the clever answer, so I built a test rig and ran it at 4, 8, 16 and 32 nodes. Plain S3 in the same region beat it by 11.7x at 32 nodes. Disk: downloads plateaued at 1.05 Gbit/s and we blamed the network. The network was fine at 4.4 Gbit/s. A default gp3 root volume caps at 125 MB/s, and that number decides your cold start. Autoscaling: the GPU scaling policy sat in INSUFFICIENT_DATA and never fired. On Amazon Linux 2023 a CloudWatch agent running as a sidecar cannot see the GPU, so nvidia-smi never runs and the metric never publishes. I will show the numbers behind each one and the fixes we shipped in our open source Terraform module, and close with a checklist you can run against your own service.
