Self-Hosting an LLM on ECS: Serving, Distribution, Disk, and Autoscaling

Not so long ago I had a customer who needed to serve an open model inside their own AWS account. They already ran ECS for everything, so the question was whether that was enough, or whether they had to bring in Kubernetes or a managed ML platform first. It was enough. In this talk I will show how we put Qwen2.5-7B behind an ALB on ECS GPU instances with vLLM, and what broke on the way to production. The build was the easy part. Three things surprised us, and in each case the measurement disagreed with the obvious answer. Distribution. A scale-out has to get 15 GB of weights onto every node. A peer-to-peer swarm looked like the clever answer, so I built a test rig and ran it at 4, 8, 16 and 32 nodes. Plain S3 in the same region beat it by 11.7x at 32 nodes. Disk. Downloads plateaued at 1.05 Gbit/s and we blamed the network. The network was fine at 4.4 Gbit/s. A default gp3 root volume caps at 125 MB/s, and that number decides your cold start. Autoscaling. The GPU scaling policy sat in INSUFFICIENT_DATA and never fired. On Amazon Linux 2023 a CloudWatch agent running as a sidecar cannot see the GPU, so nvidia-smi never runs and the metric never publishes. I will show the numbers behind each one and the fixes we shipped in our open source Terraform module. I will close with a checklist you can run against your own service: how to confirm your GPU metric is real, how to tell a disk-bound load from a network-bound one, and when local NVMe storage is worth the trouble. This talk is for anyone running model inference on AWS, or about to.

How I Automated an Enterprise Data Lake on AWS

Most data platforms assume someone will operate them. Plenty of organizations do not have that person. No dedicated platform team, no on-call rotation, and no budget to create either. Whatever gets built has to run itself or it stops running. That constraint shaped every decision I made. This talk covers how I automated the data lifecycle of an enterprise data lake on AWS, what I deliberately left manual, and what broke along the way. The environment was the common one, not the ideal one. Source systems that predate most of the people maintaining them. Departments with no reason to coordinate with each other. Retention obligations that vary by record type and are regulatory rather than advisory. And data of very different sensitivity landing in the same place. I walk through what got automated, in the order I built it. Ingestion that absorbs schema changes from systems nobody controls. Cataloging and classification on landing, so data arrives governed instead of getting governed later. Partitioning and compaction on a schedule rather than on a ticket. And retention and expiry driven by the actual records schedule, which is the piece almost nobody automates and the one that ended up mattering most. Then the honest part: what I chose not to automate, and why some approvals should stay human no matter how good the tooling gets. Attendees leave with a reference architecture built on AWS native services, a clear view of which lifecycle stages are worth automating first, and a realistic account of what a very small team can actually keep running.

AI Powered Fashion Trend Forecasting System

Fashion color trend forecasting has traditionally relied heavily on qualitative expert judgment, making it difficult to consistently determine which emerging colors will achieve broader consumer adoption. This project delivers an AI-powered fashion color trend forecasting system that combines computer vision, Natural Language Processing (NLP), machine learning, and automated visual intelligence to support earlier, data-driven retail decisions. The system processes runway imagery and fashion-related digital content to identify and track emerging color signals. PyTorch-based computer vision detects garments and extracts dominant colors, which are standardized using the CIELAB color space and mapped to a unified color lexicon. NLP analyzes Instagram and X (Twitter) content to identify color mentions, themes, engagement patterns, and social momentum, while Google Trends and market validation sources provide additional indicators of consumer interest and adoption. The solution is implemented on Amazon Web Services (AWS), with Amazon S3 serving as the data lake for images, social media datasets, raw source data, and analytical artifacts. Amazon RDS for PostgreSQL stores structured metadata, standardized colors, engineered features, validation data, and model outputs, enabling modular and reproducible AI pipelines. Machine learning models, including Logistic Regression, Random Forest, and Gradient Boosted Trees, analyze runway prominence, engagement-weighted growth, search activity, time-to-peak, persistence, and cross-platform diffusion to estimate the probability that an emerging color will become a high-impact trend. The system also automatically generates AI-powered mood boards by combining predicted colors, harmonized palettes, extracted themes, and visually relevant assets. Together, predictive forecasting, explainable recommendations, and automated visual storytelling provide retailers with an end-to-end AI solution for identifying promising colors earlier, improving seasonal assortment decisions, optimizing inventory commitments, and reducing markdown risk.

How to Design a Petabyte Scale Agentic Lakehouse Using Only AWS Services

You can build a petabyte scale lakehouse that AI agents can safely query using nothing but AWS services. This session covers how we did it at UC Irvine, what AWS gave us out of the box, and the gaps we had to fill ourselves. The environment made the constraint interesting — student records, research data, and operational systems land in the same lakehouse, owned by departments that do not report to each other, under rules that differ per dataset. We walk through cataloging and classification in AWS Glue, Apache Iceberg on S3, fine-grained row and column permissions in Lake Formation, and how partitioning and compaction had to be reworked around how agents query rather than how dashboards were built. We close with the honest part: where AWS native services ended, what we built to cover the gap, and the readiness checks we run before any agent touches sensitive data.

Specs, Not Prompts: Building a Penny-Perfect Financial System with Kiro

One AWS account buys the Savings Plans and Reserved Instances for an organisation of 177. The discount lands wherever the usage is; the invoice lands in one place. Allocating that fairly has an unforgiving property: a number that's three cents wrong doesn't look wrong. It looks like a number, and it goes to Finance. This is how I built that allocation system with Kiro over ten months, using its spec workflow rather than conversational prompting. Every change went through requirements.md, design.md and tasks.md — numbered testable acceptance criteria, a design that keeps the rejected alternatives and the reasons they lost, and a task plan where every task cites the criteria it satisfies and reports back the figures it measured against live AWS through MCP. 16 specs and 356 tasks later, eight months of allocations have published balanced to the cent. I'll show real artifacts from the repo: an acceptance criterion that forbids one specific wrong implementation by name, a design decision that got reversed because the spec was precise enough to catch a claim inside it being false by six parts per million, and a bugfix spec whose first task disproved my own root-cause diagnosis before any code changed. Then the honest half. Six AWS billing traps that were live in production and balanced perfectly while being wrong — Cost Explorer and CUR using different vocabularies for the same bill, zero standing in for null, a field name that lied about what it held. And the one the process missed: an "integrate with existing" requirement that shipped as a hardcoded map covering 3 accounts out of 217. That failure became a Kiro steering rule, and eight months later the same rule caught the same mistake one level down, before it shipped.

AI SDLC in large complex codebases

Every AI demo starts from scratch. That's the easy part. How do you use AI to build new features in a large, legacy, complex codebase? After all, that is what most production software is: years of accumulated decisions, sparse documentation, and context that only exists in the heads of the people who wrote it. This session covers tips, tricks and the thinking behind getting agents to do useful work. We'll explore what goes wrong when an agent lacks the right context, or is given more decisions than it can handle. Let's get things back in balance.

A Composable Serverless System Spec Sharable between Humans & Agents

The Topology Composer (tc) open source tooling started its life in the Before Agentic AI Times to better design, build, and deploy complex serverless systems. This allowed us to describe a system that is composable as a tree of topologies as well as the elements within a topology. These abstractions represent AWS serverless elements, including API Gateway (Routes), EventBridge Rules (Events), SQS (Queues), AppSync Resolvers/Mutations, AppSync Events (Channels), Step Functions (State), Lambdas (Functions), and DynamoDB/S3 (Stores). It turns out that this level of abstraction and composability fits into the new Agentic AI Age we are now entering. These workflows and abstractions give us a way to describe systems to Agents in a way that helps both the developers and the agents to communicate about the system design. And like strongly typed languages help to keep agents from straying from best practices, the Topology Composer's abstractions keep the agent honest and inherently follow the intent of the developers. The separation of an abstract representation from the mechanics of deployment, along with advanced LLMs, means we can deploy via any type of IaC ranging from Terraform, CloudFormation/CDK, or even direct generation of AWS SDK code. Plus, you get all the other features of tc which include sandboxing, namespacing, and a range of build options for Lambdas. The presentation will give a quick overview of the Topology Composer, its abstractions, and workflows. Then dive into how tc can work with LLMs and agentic systems to create and reason about sophisticated serverless systems and their lifecycles.

From Agent Traces to Agent Simulations: Closing the AgentOps Loop on AWS

AI agents fail differently from traditional software. Production traces can tell us what happened, but they cannot tell us whether changing a model, prompt, tool, or agent harness will fix the problem without introducing another regression. In this talk, we will move from passive agent observability to executable simulations and build a continuous AgentOps evaluation loop on AWS. We will start with patterns for turning real-world agent failures into repeatable benchmark tasks with isolated environments and explicit verifiers. We will then show how to represent those tasks using the open-source Harbor task format and run them locally or at scale on AWS. Finally, we will map the complete lifecycle onto an AWS reference architecture: Amazon Bedrock AgentCore Runtime for production agents, AgentCore Observability and Amazon CloudWatch for production traces, Harbor with isolated Amazon EC2 environments for simulations, and versioned agent configurations for controlled experiments and releases. We will walk through two feedback loops: continuously expanding benchmarks from production failures, and evaluating model, prompt, tool, and harness changes before promotion to production. Attendees will leave with practical patterns for building an evaluation system that grows alongside their agents rather than relying on static test sets or manual trace review.

Your Agent Demo Works. Now Make It Survive Production: A Defense-in-Depth Playbook on AWS

Every agent demo works. The demo is not the hard part. The hard part is the Tuesday three weeks later when the same agent, on a real user's request, calls the wrong tool, hallucinates a confident answer, loops forever, or quietly does something it wasn't authorized to do. We'll share a defense-in-depth playbook for reliable agents on AWS, structured as two reinforcing halves. First, containment: content safety with Amazon Bedrock Guardrails, tool-use authorization so an agent can only do what the user could, and why you guide an agent back on track rather than hard-block it into a dead end. Second, verification: LLM-as-judge for the semantic checks guardrails can't express, the "fail-open vs fail-closed" call that separates a safety control from an availability control, and the part nobody demos — evaluation. Offline eval sets, online sampling, and how teams move from a 15-20% false-reject rate to 95%+ by measuring instead of guessing. You'll leave with a layered reliability checklist you can apply to any agent — on Amazon Bedrock, AgentCore, or your own stack — and a candid map of what's production-ready today versus still frontier.