· Gary O'Keeffe · AI & Data · 4 min read
Warehouse, Lakehouse, or Search Index?
Three architectures for making lab data queryable. Which one fits depends on what questions you're asking.
In our experience, most conversations about making data AI-ready eventually hit the same question: where should the data live?
The options have multiplied. You can build a data warehouse. You can build a lakehouse. You can build a search index. You can combine them. Vendors often claim their solution is best, and the architecture diagrams get complicated fast.
Here’s a simpler way to think about it: start with the questions you need to answer.
What kind of questions?
Structured queries over known schemas. “Show me all experiments with compound X from the last six months, with yield over 80%.” You know the columns. You know the tables. You want SQL answers.
For this, a data warehouse is hard to beat. BigQuery, Snowflake, Redshift, Azure Synapse—they all do it well. They’re optimized for joining tables, aggregating results, and handling concurrent users. If your data fits in rows and columns and you’re asking analytical questions, a warehouse is probably right.
Exploratory search over semi-structured data. “Find all the slides with HER2 staining from the breast cancer cohort.” You’re looking for files or records that match criteria, but the criteria aren’t fully structured. The data might be in file metadata, in text fields, or in headers you’ve extracted.
For this, a search index works well. Elasticsearch, OpenSearch, or a managed service. Search indexes are good at finding things when you can’t specify exactly what you’re looking for. They handle text, nested documents, and flexible schemas.
Analytics over mixed data types at scale. “Build a model using our genomics data, our clinical outcomes, and our imaging metadata.” You need to join structured and semi-structured data, process large files, and run compute-intensive workloads.
For this, a lakehouse is the modern answer. Databricks, Snowflake’s Iceberg tables, AWS with Iceberg or Delta Lake. Lakehouses give you the flexibility of a data lake with the query performance of a warehouse. They handle diverse data types and scale to large volumes.
The practical question
In our experience, most growing life science companies don’t need to pick just one architecture. They need to pick a starting point.
If you’re just trying to make lab data findable and searchable, a search index might be all you need. It can be quick to set up, handles semi-structured data well, and works for AI assistant queries.
If you’re building analytics dashboards and reports, a warehouse is the standard choice. It integrates with visualization tools and handles the queries data analysts expect.
If you’re running ML pipelines over diverse datasets, a lakehouse gives you the most flexibility. But it’s also the most complex to set up and operate.
A real example
A digital pathology company had a petabyte of slide images. They wanted scientists and account managers to search by stain type, tissue type, and customer. They didn’t need complex analytics over the images—they needed to find the right images fast.
We built a search index on AWS OpenSearch. We extracted metadata from the image headers and file paths, linked each record to the LIMS, and indexed everything. Queries return in milliseconds. The whole thing costs under $1,000 a year to run.
A lakehouse would have been overkill. A traditional warehouse would have been awkward for the semi-structured metadata. A search index was the right tool for the problem.
The wrong question
The wrong question is “which technology is best?” Technologies change. Today’s leading lakehouse may not be the leader in three years.
The right question is “what are we trying to do?” If you can answer that clearly, the technology choice often becomes clearer. If you can’t, you may not be ready to build anything yet—you’re ready for a discovery conversation.
What about AI assistants?
AI assistants can query any of these. The assistant calls an MCP server. The MCP server calls your warehouse, lakehouse, or search index. The assistant doesn’t care which one you’re using.
What matters is that the data is:
- Findable and queryable
- Linked to records that give it meaning
- Permissioned appropriately
- Behind a read-only API the assistant can call
Whether that API queries BigQuery or OpenSearch or Databricks is an implementation detail.
How to decide
If you’re unsure, start with what you already have. If you’re on GCP and use BigQuery for other things, start with BigQuery. If you’re on AWS and already run OpenSearch for logs, start with OpenSearch. If you have Databricks, start with Databricks.
The technology often matters less than having something in place. A working search index today can be more valuable than a perfect lakehouse next year.
Once you have something working, you can migrate or add layers. Start simple. Get value. Then optimize.
Not sure what architecture fits your data? Our AI Data Readiness assessment includes a target architecture recommendation based on your specific situation.
