15 min read
From a Single EC2 Instance to Production-Grade EKS: Scaling the Infrastructure Behind One of Our Builds
A look at how we phased infrastructure for a production application, from a single EC2 instance through a lightweight Kubernetes stepping stone to a fully modular AWS EKS setup, along with the backend, observability, and frontend architecture built on top of it.
TL;DR
- We phased infrastructure in three stages: a single EC2 instance running Docker Compose to reach production fast and cheap, a lightweight self-managed Kubernetes cluster as a stepping stone, and finally a fully modular AWS EKS setup built from dedicated Terraform modules.
- The backend is a NestJS and TypeScript service with a GraphQL API, clean and hexagonal architecture, Kafka-driven events, and ability-based authorization, generated consistently across modules with custom code generators rather than copied by hand each time.
- OpenTelemetry instrumentation runs through HTTP, GraphQL, the database, and Kafka, exporting to Jaeger, so production behavior is observable rather than something to reconstruct after an incident.
- None of the phases required tearing anything down. Each one built on the last, and the local development environment mirrors the production topology closely enough that most of this can be exercised before it ever reaches AWS.
Corsair Media Group
Why we phased the infrastructure instead of building for scale on day one
We started this application on a single EC2 instance running Docker Compose, with encrypted S3 backups and least-privilege IAM roles from day one. The goal of that first stage was specific: get to production fast and cheap, without spending early engineering time on infrastructure the application did not need yet. A single instance is easy to reason about, easy to back up, and easy to hand off, and it let the team focus on the application itself while it was still taking shape.
From there, we moved to a lightweight, self-managed Kubernetes cluster using k3s running on EC2. k3s is a stripped-down Kubernetes distribution that keeps the core orchestration concepts, deployments, services, scaling, without the operational weight of a full managed cluster. We treated it as a stepping stone rather than a destination: a way to start thinking in Kubernetes primitives and prove out the application in that shape before taking on the cost of a fully managed setup.
The final stage was a migration to a fully modular, production-grade AWS EKS setup. Each of the three stages matched the infrastructure investment to what the application actually needed at that point in time, rather than paying the operational cost of the most capable setup before there was a real reason to.
What the EKS setup actually looks like
The EKS environment is built from dedicated Terraform modules, one each for networking, the cluster itself, managed Postgres through Amazon RDS, managed cache and streaming through ElastiCache Serverless and MSK Serverless, secrets, and cluster add-ons. Splitting these into separate modules means a change to networking does not require touching the module that manages the database, and a new environment can be stood up by composing the same modules rather than writing new infrastructure code from scratch.
The add-ons layer is what makes the cluster behave like a production platform rather than a bare set of nodes. It includes the AWS Load Balancer Controller for routing external traffic in, ExternalDNS for keeping DNS records in sync with what is actually running, the External Secrets Operator for pulling secrets from AWS rather than storing them in the cluster directly, Karpenter for autoscaling nodes based on what the cluster actually needs at a given moment, metrics-server for basic resource metrics, and an OpenTelemetry collector running as the AWS Distro for OpenTelemetry, or ADOT, to gather traces and metrics from everything running on the cluster. We run Karpenter self-managed rather than through EKS Auto Mode, AWS’s newer bundled version of the same autoscaler. Auto Mode is a reasonable default for a team that wants autoscaling to work with less setup, but it trades away control over node configuration and custom AMIs, and this build already had a modular Terraform setup built to own that layer directly.
The application itself deploys through Helm, using IAM Roles for Service Accounts, or IRSA, so each workload gets AWS permissions scoped to exactly what it needs, tied to its own identity, rather than a shared credential that every workload on the cluster would otherwise have access to.
The backend: a service built to stay consistent as it grows
The backend is a NestJS and TypeScript service exposing a GraphQL API through Apollo, with real-time subscriptions running over Redis. The codebase follows clean and hexagonal architecture: application logic is organized into modules and use-cases that stay independent of the infrastructure around them, so the business logic does not know or care whether a request arrived over GraphQL, a message arrived over Kafka, or a test is calling it directly.
We built custom code generators using Plop to scaffold new modules, use-cases, and GraphQL endpoints in a consistent shape every time. That matters more than it sounds like it should on a codebase that will keep growing. Without a generator, every new module ends up structured slightly differently depending on who wrote it and what they were looking at that week. With one, a new module starts from the same shape as every module before it, which keeps the codebase readable as more people touch it.
Asynchronous workflows and decoupled services run through Kafka, following an event-driven design. Authorization is ability-based through CASL, which checks what a specific user is allowed to do with a specific piece of data rather than relying on a coarse role check alone, and actions that matter for compliance are written to an encrypted audit log.
Seeing what is actually happening in production
OpenTelemetry instrumentation runs through the HTTP layer, the GraphQL layer, the database, and Kafka, exporting traces to Jaeger over OTLP. Health checks and structured logging are wired through the whole stack rather than added function by function as problems come up. The point of instrumenting this broadly is straightforward: when something behaves unexpectedly in production, the trace already exists. Nobody has to add logging after the fact and wait for the problem to happen again.
Integrations that had to work correctly the first time
A few integrations in this build carry a higher cost of getting wrong than most application code, so they were built with that in mind. Payment processing includes automated setup and reconciliation tooling, so account state stays correct without someone manually cross-checking transactions. AI-assisted features run through an LLM API integration. Third-party authentication runs through Auth0, with the resulting JWT propagated through an Envoy edge proxy so downstream services can trust the identity of a request without each one talking to Auth0 directly. Cloud object storage uses direct-to-S3 uploads with presigned URLs, so large files move straight from the client to storage instead of routing through the application server as a bottleneck.
The frontend: typed data and a component library tested like code
The frontend is a Next.js and React application using Apollo Client against the GraphQL API, styled with Material UI and Tailwind. It includes interactive mapping and animated 3D visualizations, which pushed the component work well past standard forms and tables and into territory where visual regressions are easy to introduce and easy to miss in a normal code review.
We built the component library with Storybook, using Chromatic for visual regression testing, so a change that shifts a component’s appearance gets flagged automatically instead of discovered by a user later. Test coverage runs through Vitest for unit tests, Playwright for end-to-end tests, and Jest, giving the frontend the same layered testing approach most teams reserve for the backend alone.
Keeping the frontend and backend honest with each other
GraphQL codegen generates types on both sides of the API directly from the schema, so a field that changes on the backend shows up as a type error on the frontend at build time, not as a runtime bug a user finds first. That kind of end-to-end type safety is one of the more underrated benefits of GraphQL when it is set up correctly, and it only holds if the codegen step is treated as part of the build, not an optional convenience.
The local development environment is fully containerized with Docker Compose and mirrors the production topology closely: a proxy, a database, a cache, a message broker, tracing, a mail sandbox, and mock cloud storage all run locally. An engineer can exercise the full request path, including the parts that would normally depend on a third-party service, without touching a real payment processor, a real mail provider, or a real S3 bucket during development. Most confidence about how the system behaves gets built before a change ever reaches a shared environment.
What this adds up to
None of the three infrastructure phases required tearing anything down to move to the next one. Each stage built on top of what came before it, and the same discipline shows up in the rest of the stack: modules generated the same way every time, instrumentation that already exists before an incident happens, and a local environment that mirrors production closely enough to catch problems before they ship. If your infrastructure investment is currently ahead of or behind what your application actually needs, then that mismatch is usually fixable without a full rebuild, provided the underlying architecture was built to allow it. Is your infrastructure matched to where your application actually is today, or to where it was when it was first built? If you want a second opinion on that, then will you share what you are running through our contact page?
Outgrowing your current infrastructure, or paying for more than you need?
Talk with CorsairContinued reading
Keep exploring related topics that connect strategy, implementation, and long-term maintenance.
AWS Lambda in practice: where it earns its keep, where it bites back, and how to keep your code portable
One pipeline where Lambda performed well in production, one project where it became the wrong tool, and the architectural pattern we use to keep business logic portable across runtimes.
Software projects and real-world complexity
Fixed-price bids, shifting requirements, and on-call gaps are the recurring patterns that push software projects off track. This article describes what they look like and how to address them.