A serverless AWS data platform that turns multi-TB IoT data into analytics — for the price of a team lunch

Context

The client operates in the smart-buildings and facilities space, collecting telemetry from sensors deployed across their building portfolio. That data was piling up — multiple terabytes of it — but sitting largely untouched. They wanted to start an analytics initiative to actually make use of it, and they wanted to do it without standing up a data engineering department or signing up for a five-figure monthly platform bill. They brought me in to design and build that first platform: the foundation the rest of their data strategy would grow on.

The Challenge

The hard part here wasn’t the volume. Terabytes of IoT data is a solved problem if you’re willing to throw money and headcount at it. The hard part was matching the platform to where the company actually was.

This was an early-stage analytics initiative, not a mature data org. A heavyweight stack — always-on clusters, a streaming backbone, a warehouse with a per-seat license — would have been technically impressive and completely wrong. It would have burned budget on capacity nobody was using yet, and it would have demanded ongoing operational attention from a team that didn’t have a dedicated platform engineer to give it.

So the real constraints were:

• Cost has to scale with usage, not with ambition. At this stage the platform processes hundreds of GB a month against multiple TB of stored history. The bill should reflect that — not the theoretical peak.
• Maintenance load must stay near zero. No clusters to babysit, no servers to patch, no 2 a.m. pages. The client’s people needed to spend their time on analysis, not on keeping infrastructure alive.
• It has to be a foundation, not a dead end. A cheap platform that boxes you in is a false economy. Whatever I built had to be able to grow as the initiative matured, without a rebuild.

The genuinely difficult judgment was resisting the urge to over-engineer for a future that hadn’t arrived yet.

Approach

I diagnosed this as a right-sizing problem before it was an architecture problem. The question wasn’t “what’s the best data platform” — it was “what’s the leanest platform that solves today’s problem and doesn’t block tomorrow’s.”

That ruled some popular options out on purpose:

• No always-on Spark/EMR cluster. Powerful, but it bills whether or not you’re using it, and it needs operational care. For hundreds of GB a month, that’s paying rent on an empty warehouse.
• No managed data warehouse (Redshift/Snowflake-style). Excellent at interactive analytics at scale, but overkill — and over-budget — for an initiative still finding its footing. Storage and compute coupling also works against a “store everything cheaply, process occasionally” pattern.
• No streaming pipeline. The reporting needs were batch-shaped. Real-time infrastructure would have added cost and complexity to solve a problem the client didn’t have.

What I chose instead was a fully serverless, pay-per-use stack on AWS:

• Amazon S3 as the storage and data-lake layer — cheap, effectively infinite, and decoupled from compute. Store all the TBs you like; you only pay compute when you actually run something.
• Apache Iceberg as the table format on top of S3 — this is what turns a pile of files into a real analytical table. Schema evolution, time travel, efficient incremental reads and ACID guarantees, without locking the data into any one engine. It’s also the key to the platform growing later: the same Iceberg tables can be queried by heavier engines down the road without migrating data.
• AWS Glue for the ETL/processing — serverless Spark that bills per job run. No idle cost. It scales up for the occasional heavy backfill and disappears when idle.
• AWS Step Functions for orchestration — a managed state machine to coordinate the Glue jobs, handle retries and failure paths, and keep the workflow observable. No Airflow server to run and patch.

The whole shape of the architecture follows one principle: decouple storage from compute, and pay for compute only when it runs. That’s what gets the bill down to roughly $200/month while still sitting on multiple terabytes.

Implementation

Beyond the runtime platform, I focused on something that pays dividends long after I’m gone: the developer workflow.

A data platform that only the person who built it can extend is a liability. So I set up Git repository templates that encode the project’s conventions — structure, configuration, CI wiring, deployment scaffolding — so that starting a new data project is a matter of cloning a template and filling in the specifics. What used to be a multi-day setup-and-wire-everything exercise becomes a few minutes.

On top of that I built out CI, so that changes to pipelines are validated and deployed through a repeatable, automated path rather than by hand. That keeps quality consistent as more people touch the platform, and it removes the single-maintainer bottleneck — the client’s own developers can extend the platform confidently because the guardrails are built in.

The result is a platform the client owns, not one they rent my attention for.

Results

The headline is the ratio: multiple terabytes of stored IoT data, hundreds of GB processed every month, for around $200/month in running cost. For a company at the start of its analytics journey, that’s the difference between “let’s try this” and “we can’t justify the spend.”

• Cost. Roughly $200/month, scaling with actual usage rather than provisioned capacity. There’s no idle infrastructure being paid for.
• Maintenance. Effectively zero operational overhead — no clusters, servers, or orchestration hosts to manage. The serverless stack means the client’s small team spends its time on analysis, not babysitting.
• Velocity. New data projects go from idea to scaffolded-and-deploying in minutes, thanks to the repository templates and CI, instead of days of boilerplate.
• Headroom. Because the data lives in S3 as open Apache Iceberg tables, the platform isn’t a dead end. As the analytics initiative matures, heavier query engines and more sophisticated workloads can be layered on the same data without a migration or a rebuild.

The client now has working data analytics capability where before they had a growing pile of unused sensor data — and a foundation that can grow with them rather than against them.

Takeaway

The instinct in data engineering is to build for the scale you hope to reach. Often the more valuable move is to build precisely for the scale you’re at — cheaply, simply, and on open foundations that let you grow without throwing the first version away.

If your organization is sitting on data it isn’t using yet, and you want a platform that fits where you are today while leaving room for where you’re going, I’d be glad to talk.