Replacing spreadsheets with a production data platform
My summer at Quantum Opus was the most forward-deployed work I've done: sitting with engineering and operations teams, learning how data actually moved between them, and replacing a tangle of spreadsheets with a real internal platform — one shaped by their feedback, not my assumptions.
The problem
Engineering and operations were running on fragmented spreadsheets. Data was copied between teams by hand, the same records were reconciled in multiple places, and "the source of truth" depended on who you asked. The cost wasn't just wasted time — it was the quiet errors that creep in whenever humans are the integration layer.
The forward-deployed part
There was no clean spec. The real requirements lived in people's heads and in the shape of the spreadsheets they'd built to cope. So the first job wasn't code — it was sitting with the people doing the work, watching how they actually moved data, and translating those messy operational processes into scoped requirements. Then I built against that, put it in front of them, and let real feedback redraw the plan when my first read was wrong.
What I built
A full-stack internal data platform: a Go/Gin backend with a Next.js dashboard, PostgreSQL for storage, and Docker Compose to make the whole thing reproducible. The backend exposed modular REST endpoints that Python data-generation pipelines could consume independently, and I shipped automated ingestion, validation, and dynamic entity linking so data quality stopped depending on manual cleanup.
Decisions & tradeoffs
| Decision | Why | What I traded away |
|---|---|---|
| Modular REST endpoints, consumed independently by the Python pipelines | It cut redundant cross-service data movement and let each pipeline pull exactly what it needed, so integrations stayed flexible as requirements shifted. | More endpoints to design and keep coherent than a single monolithic interface. |
| Automated ingestion + validation + dynamic entity linking | The whole point was to remove manual cleanup loops; validating and linking entities at ingest meant the dashboard could be trusted. | Real upfront complexity in the pipeline versus just importing rows and hoping. |
| Scope from stakeholder conversations, then iterate from real feedback | The accurate spec only existed in how the team worked; building, demoing, and revising surfaced the edge cases a requirements doc never would. | A slower start and a lot of conversations before the first line of durable code. |
What was hard
The code was rarely the hard part. The hard part was ambiguity — undocumented exceptions, two teams using the same word for different things, processes that existed only as habits. Getting that wrong early meant re-scoping later, so I learned to ship a thin slice to real users fast and treat their reaction as the actual requirements gathering.
What I'd do differently
Invest in validation and a little observability even earlier — the moments that cost the most trust were silent data-quality issues, and catching them at the boundary is cheaper than explaining them after. And I'd push to put the very first rough version in front of one team sooner, before generalizing across both.
Internal/proprietary codebase, so this is the engineering reasoning rather than the code. The work and stack are summarized on my résumé.