Real-time data has become an expectation. Fraud checks, inventory, pricing, logistics and AI agents all work better on fresh data. The temptation is to make everything streaming and sort out the details later.
That usually produces an expensive platform with fragile pipelines and unclear ownership. Real time is a requirement to be justified per use case, and the lakehouse is where streaming and batch can finally share one set of tables, rules and controls.
Ask how fresh is fresh enough
Start with the decision the data supports, not the technology:
- What action depends on this data, and how quickly must it happen?
- What does a stale value cost — a bad customer experience, a financial loss, nothing much?
- Is seconds, minutes or hourly freshness actually different for the user?
Many use cases described as real time are well served by micro-batches every few minutes. Reserve true event-by-event streaming for the cases where latency changes the outcome.
One storage layer, two speeds
Open table formats such as Delta Lake, Apache Iceberg and Apache Hudi let streaming writes and batch reads work against the same tables. That removes the old split where a separate speed layer disagreed with the warehouse.
A common shape:
- Events arrive through a log or message bus such as Kafka
- Stream processing cleans, deduplicates and enriches records
- Bronze, silver and gold layers hold raw, conformed and consumption-ready data
- The same gold tables serve dashboards, APIs, feature stores and retrieval for AI
The point is consistency. A metric should not have one value in the real-time view and another in the monthly report.
Data products, not pipelines
A pipeline is plumbing. A data product is something a team owns and others depend on. Each data product should have:
- A named owner who answers for quality and changes
- A schema and contract consumers can rely on
- Freshness and completeness targets, measured
- Documentation of meaning, not only column names
- Access rules and lineage
When streaming feeds a data product with these properties, consumers can build on it with confidence. When it feeds an anonymous table, every downstream team writes its own defensive checks.
Streaming has its own failure modes
Batch jobs fail loudly. Streams often fail quietly. Plan for:
- Late and out-of-order events — define watermarks and how late data is handled
- Duplicates — design for idempotent writes and exactly-once where it matters
- Schema changes — use a schema registry and contract checks before producers deploy
- Backpressure — know what happens when a downstream system slows
- Replay — be able to reprocess a time window after a bug fix
Each of these needs a test and a runbook, not just a design diagram.
Observability and cost
Monitor consumer lag, end-to-end latency, error rates and data quality checks on the stream itself. Alert on the freshness target the business agreed, not on infrastructure metrics alone.
Streaming compute runs all the time, so cost is continuous. Right-size clusters, compact small files, and review whether each stream still needs its latency. A pipeline that nobody reads in real time should go back to batch.
Governance applies at speed
Personal data in an event stream is still personal data. Masking, retention, residency and access control need to apply at ingestion, not after data lands. Lineage should cover streaming jobs as well as batch ones, so audit can trace how a value was produced.
Where to start
- Choose one decision that clearly benefits from fresher data
- Agree the freshness target and the cost of missing it
- Build one streaming data product end to end, with an owner and a contract
- Add monitoring for lag, freshness and quality before adding a second stream
- Review latency needs quarterly and downgrade streams that do not earn it
Incubics builds streaming pipelines and data products on the lakehouse with ownership and observability included from the first release.