The Modern Data Stack for Growing Startups

The spreadsheet moment

There is a specific moment that arrives at most growing startups, usually somewhere between the first investor update and the first serious hire. Someone asks a question that ought to have a straightforward answer: what is the lifetime value of customers who came in through paid search? Revenue lives in Stripe, acquisition channel in Google Analytics 4, customer activity in the application database, and none of these systems can see the others. Answering that single question requires three separate exports, a shared spreadsheet that nobody fully trusts, and an afternoon that nobody had available.

By the time a startup reaches Series A, it is typically running on 10 or more separate data stores: a production Postgres database, a CRM, a payment processor, a marketing analytics platform, a customer support tool, and whatever else the product and commercial teams have adopted independently since the last all-hands. The data exists; it is simply scattered across systems that were never designed to talk to each other, and the business is making decisions on incomplete information without always knowing it.

Inconvenience versus structural constraint

The distinction worth making early is between a data inconvenience and a data infrastructure problem. The former is a fact of life at any company running on SaaS tools: you will always have to look in more than one place for some things, and that is largely manageable. The latter is when fragmentation actively prevents you from answering questions that should inform strategy, pricing, product direction, or hiring. When a founder cannot say, with confidence, which customer segments are growing or which acquisition channels are paying back, that is not an inconvenience but a structural constraint on the company's ability to make good decisions, and it compounds the longer it goes unaddressed.

The ingestion layer

The first thing to get right is ingestion, because everything else in the stack depends on it. Before you can transform, model, or analyse your data, you need it in one place. Managed ingestion tools (Airbyte and Fivetran are the two credible options at this stage) maintain pre-built connectors to the SaaS platforms your business already runs on: Stripe, HubSpot, Salesforce, Google Analytics 4, Intercom, and several hundred others. The connectors handle authentication, pagination, rate limiting, schema changes, and incremental syncing. Building this yourself is possible; it is also the kind of work that expands to fill whatever engineering time is available and is never quite finished.

The practical question at this point is not which ingestion tool to use but what to connect first. The temptation is to pull everything in immediately on the basis that more data is always better. A more useful starting point is to identify the specific decisions you are currently making on partial information, then trace those back to the source systems that hold the missing half. Starting from the question rather than the data produces a prioritised list of connections and avoids the cost of maintaining syncs you are not yet using.

MotherDuck and Athena

Once ingestion is in place, you need somewhere for the data to land and a way to query across it, and that does not have to mean an immediate commitment to a full cloud data warehouse. MotherDuck and Amazon Athena are both credible starting points for companies at the earlier stages of this journey, occupying slightly different positions depending on where your infrastructure already lives.

MotherDuck is the cloud layer built on top of DuckDB, a column-oriented analytical database that is, for its size, remarkably fast and cheap to operate. It suits teams that are not yet committed to a cloud provider and want something SQL-native with minimal setup. Athena is Amazon's serverless query layer that runs SQL directly against data stored in S3, making it the natural choice for startups already living in the AWS environment. Both solve the same problem at the same stage: querying aggregated data from multiple sources without the overhead of a fully managed warehouse. The meaningful difference is that Athena, because it queries data directly from S3 in formats such as Parquet and Apache Iceberg, puts you in data lake territory from the beginning, which has consequences further up the stack when the time comes to think about a lakehouse.

When to graduate to a cloud warehouse

MotherDuck and Athena carry most companies further than they expect, but specific conditions make a managed cloud warehouse the sensible next step. The most common are team size (multiple analysts running concurrent queries against the same dataset), data volume (DuckDB in particular is not designed for the scale a Snowflake or BigQuery cluster handles comfortably), and the need for features that managed warehouses provide as standard: granular access controls, query history, cost management tooling, and the operational foundations that a data team of more than two or three people will start to require.

BigQuery and Snowflake are the two dominant options at this point, and the choice between them tends to matter less than teams expect. BigQuery suits companies already in the Google Cloud environment and bills by bytes processed, which can be either economical or expensive depending on query patterns. Snowflake separates compute and storage cleanly, giving more predictable cost management and working well across cloud providers. Both are fully managed, both scale comfortably past the volumes a Series A company is likely to encounter, and neither requires an infrastructure engineer to operate. The decision is worth thirty minutes of analysis rather than three weeks.

dbt from the start

Whichever query layer your data lands in, you need a transformation layer to turn raw ingested records into something the business can use. dbt is the right tool at every stage of this journey, and starting with it from the beginning matters more than most teams realise.

dbt works against your warehouse or lakehouse using SQL, which means any engineer who can write a query can contribute to data models without learning a new paradigm. More importantly, it imposes the kind of practices that ad hoc SQL scripts do not: version control, automated testing, documentation that lives alongside the code, and a clear dependency graph between models. A transformation written as a dbt model is something you can deploy, test, and hand to a new team member; the equivalent logic written in a Jupyter notebook that only one person runs is a liability. The portability is equally important: dbt adapters exist for MotherDuck, BigQuery, Snowflake, and Databricks, which means the models you write today run against a different compute layer when the time comes to move, without rewriting the transformation logic.

When cron jobs stop being enough

A pipeline that runs once a day on a predictable schedule does not need an orchestration tool. A cron job is perfectly adequate, and adding infrastructure to solve a problem you do not yet have is the kind of decision that returns to haunt engineering teams later. The inflection point arrives when you have enough pipeline steps that the failure of one should block the others, when you need retry logic you would rather not write yourself, and when your data engineer starts each Monday by checking which jobs failed overnight.

Prefect is what we reach for at that point. It provides dependency management between tasks, visibility into run history and failures, and configurable retry behaviour without requiring changes to the underlying pipeline code. The argument against Apache Airflow at this scale is largely operational: Airflow requires infrastructure to run and a non-trivial amount of configuration to maintain, and for a team that is not yet large enough to justify a dedicated data platform engineer, that overhead is difficult to absorb. Prefect's managed offering removes most of it.

The lakehouse as the destination

Databricks is, in our view, where a serious data function eventually lands, though the timing of that arrival matters as much as the destination. The Lakehouse architecture that Databricks pioneered combines what a traditional data warehouse does well (structured SQL analytics at scale) with what a data lake does well (storing data in open formats, handling mixed workloads, and supporting ML training pipelines alongside batch analytics). The result is an environment capable of running structured streaming for real-time pipelines, processing data at volumes where warehouse compute costs become difficult to justify, and providing governance and lineage through Unity Catalog at a level of detail that bolt-on tools rarely match.

The medallion architecture is the standard pattern for organising data within a lakehouse, and its logic becomes clearer once you have spent time debugging a pipeline where raw and transformed data live in the same tables. Data flows through three layers: bronze holds raw ingested records exactly as they arrived from the source; silver is where you clean, validate, deduplicate, and join records into coherent entities; gold contains the business-ready aggregations that analysts and decision-makers actually query. The separation between what was received and what has been processed makes reprocessing, auditing, and debugging substantially more tractable than stacks that collapse these concerns into one layer.

The conditions that make Databricks the right next step are fairly concrete. If your warehouse compute costs have become a material line item driven by query volume rather than storage, Databricks' architecture tends to produce savings at the volumes that matter. If you are running machine learning pipelines and managing compute environments outside the warehouse has become its own maintenance burden, Databricks' managed ML infrastructure addresses that directly. If you need to process data in real time rather than in batch windows, Delta Live Tables and structured streaming provide a mature, production-grade answer. For companies in regulated industries where data lineage and access governance are audit requirements rather than aspirations, Unity Catalog is the most complete solution currently available.

Build for the migration, not away from it

The decisions you make at each earlier stage of the stack determine how expensive reaching the lakehouse eventually is. Using dbt for transformation from the beginning is the most consequential of these. Because dbt runs on Databricks with a first-class adapter, the transformation logic you write against Snowflake today migrates without a rewrite. If your ingestion layer uses Airbyte and writes to an open format, the underlying data can follow without a lengthy migration project. If your orchestration is in Prefect and your pipeline logic is version-controlled, the knowledge about how your data moves lives in the code rather than in the memory of the engineer who built it.

The goal at Series A is not to build the stack you will need at Series C; it is to build something coherent and well-founded at your current scale, using tools that do not require you to throw everything away when you grow out of them. Starting with the ingestion problem, solving it with the right tool for your current infrastructure, and building transformation logic in dbt from day one puts you in a position where each subsequent step in the stack is an addition rather than a replacement.