A data engineering pipeline is an automated workflow that ingests raw data, transforms it into a usable structure, stores it reliably, and serves it to dashboards, models, or applications. Start with a batch pipeline unless a specific business case demands sub-minute freshness; batch is cheaper to build, easier to debug, and covers most analytics needs. Requestum and platforms like Apache Airflow, Dagster, and Apache Spark all reflect this same four-layer foundation.
TL;DR:
- Choosing a batch pipeline is generally better for most analytics needs due to its lower cost, simplicity, and robustness unless real-time data is critical.
- Separating pipeline layers into ingestion, transformation, storage, and serving helps maintainability and fault isolation, with each layer using specialized tools like Kafka Connect or dbt.
- Orchestration tools such as Airflow and Dagster improve failure handling and dependency management, with Dagster offering better lineage and asset tracking for debugging.
- Implementing medallion architecture with bronze, silver, and gold layers enhances data traceability and simplifies error root-cause analysis across the pipeline.
- Proper security, access controls, data lineage, monitoring, and explicit service-level agreements at each layer are essential to building reliable, scalable data pipelines.
Table of Contents
- Data Engineering Pipeline Architecture: The Four Layers That Matter
- Batch, Streaming, or Hybrid: Which Fits Your Use Case?
- Why Does Orchestration Live Separately From the Pipeline Itself?
- Medallion, Lambda, and Kappa: Which Design Pattern Should You Use?
- How Do You Actually Build a Production Data Pipeline?
- What Should You Monitor to Keep a Pipeline Reliable?
- What Mistakes Break Data Pipelines Most Often?
- How Requestum Applies These Patterns in Real Projects
- How Do You Secure a Data Pipeline End to End?
- Why Does Data Lineage Matter More Than Most Teams Realize?
- The Real Bottleneck Isn't the Tools
- Get Help Building a Pipeline That Won't Break in Six Months
- Sources
- FAQ
Data Engineering Pipeline Architecture: The Four Layers That Matter
Every reliable data pipeline architecture breaks into four layers, and treating each one as a separate contract, rather than one continuous script, is what keeps a pipeline maintainable once three teams depend on it.
Ingestion pulls data from source systems (APIs, databases, event streams) and lands it somewhere durable, unmodified. Its SLA is about availability and completeness: did every record from the source arrive, and how stale can it be before someone gets paged? Transformation reshapes raw records into clean, modeled tables. Its blast radius is the widest; a bad join here corrupts everything downstream. Storage holds both raw and processed data at rest, with SLAs around durability and cost per terabyte rather than latency. Serving exposes final datasets to dashboards, APIs, or machine learning features, where the SLA is query latency and concurrency.
Typical tooling by layer looks like this:
- Ingestion: Kafka Connect, Fivetran, Airbyte
- Transformation: dbt, Apache Spark, Apache Flink
- Storage: Amazon S3, Google Cloud Storage, Snowflake, BigQuery, Redshift
- Serving: Looker, Tableau, Redis, Feast
Each layer failing independently, instead of taking the whole pipeline down with it, is the entire point of separating them this way.
Batch, Streaming, or Hybrid: Which Fits Your Use Case?
Choosing between batch, streaming, and a hybrid setup comes down to how fast the business actually needs an answer, not how fast the technology allows.
- Batch pipelines run on a schedule (hourly, nightly) and process data in bulk. They're the default for analytics workloads because they're simpler to test, cheaper to run, and forgiving of late-arriving data.
- Streaming pipelines process events as they arrive, usually through a broker like Apache Kafka. They solve real-time streaming architecture needs like fraud detection or live inventory counts, but they introduce ordering guarantees, late-data handling, and exactly-once semantics that batch never has to deal with.
- ETL transforms data before loading it into storage; ELT loads raw data first and transforms it inside the warehouse, which works well when warehouse compute is elastic and storage is inexpensive.
- Hybrid architectures run a batch backbone for historical accuracy alongside a thin streaming layer for the handful of metrics that truly need freshness.
A retail sales dashboard belongs on batch. A payment fraud detector needs streaming. Most companies need both, running side by side, not one replacing the other.
Why Does Orchestration Live Separately From the Pipeline Itself?
Orchestration platforms exist because cron jobs don't retry gracefully, don't report failures clearly, and don't understand dependencies between tasks. Tools like Airflow, Dagster, and Prefect replace that fragility with scheduling, monitoring, and dependency management built for pipelines with dozens of interdependent steps.
The split that matters most is task-centric versus asset-centric orchestration. Airflow defines pipelines as Python DAGs, where the unit of work is a task and its dependencies. Dagster flips that model: it treats each dataset as a first-class asset, attaching lineage, freshness, and health status directly to the data rather than just to a job's pass/fail status. That difference shows up the moment something breaks, since Dagster can tell you exactly which downstream table went stale, while a purely task-centric view only tells you a job failed.
Representative tools by role:
- Orchestration: Airflow, Dagster, Prefect
- Ingestion: Fivetran, Airbyte, Kafka Connect
- Transformation: dbt, Spark, Flink
- Storage and serving: Snowflake, BigQuery, Redshift, Looker
Pro Tip: If your team already knows Python and needs a mature operator ecosystem, Airflow will feel familiar. If you need built-in data lineage and asset freshness tracking without bolting on separate tools, Dagster's model saves real debugging time later.
Medallion, Lambda, and Kappa: Which Design Pattern Should You Use?
The medallion architecture organizes data into bronze (raw), silver (cleaned and validated), and gold (business-ready, aggregated) layers. It matters because it makes lineage traceable: when a number looks wrong in a gold table, you can walk backward through silver to bronze and find exactly where it broke, instead of re-running one monolithic transformation and hoping.
Lambda architecture runs separate batch and streaming code paths that eventually reconcile, giving you speed and completeness at the cost of maintaining two systems. Kappa architecture treats everything as a stream, including historical reprocessing, which simplifies the codebase but demands a stream processor capable of replaying large volumes efficiently.
A few concrete guidelines carry over regardless of which pattern you choose:
- Use columnar formats like Parquet for analytical storage; use Avro when schema evolution on write matters more than read speed.
- Partition by a field your queries actually filter on (usually date), not by an arbitrary hash.
- Compact small files regularly. Streaming ingestion in particular produces thousands of tiny files that quietly wreck query performance over time.
How Do You Actually Build a Production Data Pipeline?
Building a pipeline that survives contact with real production traffic follows a fairly consistent sequence, even though every team's tooling differs.
- Discovery first. Define the data contract for each source: expected schema, arrival frequency, and who owns fixing it when it breaks. Write the SLA down; don't leave it implied.
- Land raw data untouched. Ingest into a raw zone before any transformation touches it, and build in schema-evolution handling so a new column upstream doesn't silently break the job.
- Design transformations modularly. Break logic into small, testable steps rather than one giant SQL script. Add unit tests for business logic, integration tests across the full flow, and snapshot tests to catch unexpected output drift.
- Wire up orchestration. Configure retries with exponential backoff, route unprocessable records to a dead-letter queue instead of dropping them, and build a backfill path from day one, because you will need it. Production pipelines require this operational layer to avoid silent data quality failures.
- Deploy through CI/CD. Treat pipeline code like application code: peer review, automated tests in the deploy pipeline, and staged rollouts.
- Check it after launch. Confirm freshness, row counts, and downstream consumer health, and write a short runbook covering what to do when the on-call engineer gets paged at 2 a.m.
Pro Tip: Build the dead-letter queue and backfill script before launch, not after the first incident. Retrofitting error handling under pressure is how minor bugs turn into three-day outages.
What Should You Monitor to Keep a Pipeline Reliable?
Observability in a data pipeline means catching problems before an analyst notices a dashboard looks wrong. Four signals matter most: freshness (is the latest data as recent as the SLA promises), volume (did row counts drop or spike unexpectedly), schema drift (did an upstream field change type or disappear), and lineage (which downstream tables does a broken upstream table actually affect).
A solid testing matrix combines several test types, each catching a different class of failure:
- Unit tests on individual transformation functions
- Integration tests across the full ingest to serve flow
- Golden or snapshot tests comparing output against a known good result
- Synthetic data tests that inject edge cases real production data hasn't shown yet
Operational controls close the loop: retry and backoff policies for transient failures, dead-letter queues for records that consistently fail, alerting thresholds tuned to avoid both silence and noise, and a runbook that tells whoever is on call what to check first.
What Mistakes Break Data Pipelines Most Often?
Most pipeline failures trace back to a handful of repeat offenders, and each has a known fix.
- Non-idempotent writes. Rerunning a job that appends instead of upserts doubles your data. Design every write to be safely repeatable.
- Loose schema contracts. An upstream team renames a column and your pipeline breaks at 3 a.m. Enforce schema contracts and fail loudly, early, in ingestion.
- Small-file explosion. Streaming or micro-batch jobs writing thousands of tiny files will quietly degrade query performance over weeks until someone notices.
- Uncontrolled retention. Keeping every raw file forever inflates storage costs; set retention policies tied to actual compliance and debugging needs.
- Right-size compute, not maximum compute. Oversized clusters running on partial workloads are the most common hidden cost in a data platform.
Growing query latencies and a rising count of small files are the two clearest signals it's time to revisit the architecture rather than patch around it again.
How Requestum Applies These Patterns in Real Projects
Requestum's data science services team builds pipelines that follow the same layered discipline covered above: raw ingestion, validated transformation, and a serving layer sized to whatever consumes the data, whether that's a dashboard or a predictive model. Projects documented in Requestum's AI and data science case studies apply medallion-style layering and orchestration to keep forecasting and analytics pipelines auditable rather than opaque.
The value of a well-built pipeline shows up quietly, in the incidents that never happen: the schema change that gets caught in a raw layer instead of corrupting a finance report three hops downstream.
How Do You Secure a Data Pipeline End to End?
Security in a pipeline is layered, the same way the architecture itself is layered, and each layer needs its own controls rather than one perimeter firewall pretending to cover everything.
At ingestion, credentials for source systems should live in a secrets manager, never in pipeline code or config files checked into version control. Encrypt data in transit between every layer and at rest in object storage and warehouses alike. Role-based access control matters most at the storage and serving layers, where analysts, models, and external applications all need different slices of the same data. A common failure mode is granting broad warehouse access during a rushed rollout and never revisiting it, which leaves far more people with far more access than the original use case required.
Column-level and row-level security matter for any pipeline touching personal or financial data. Masking sensitive fields in lower environments (staging, development) prevents a common and often overlooked leak: engineers debugging with real customer data in a test database that has none of production's access controls.
Audit logging closes the loop. Every layer should log who accessed what and when, both for compliance and for incident response when something does go wrong. Treat access reviews as a recurring task, not a one-time setup step, since pipeline access tends to accumulate the same way file permissions do on any shared system: quietly, and in one direction.
Why Does Data Lineage Matter More Than Most Teams Realize?
Data lineage traces a piece of data from its source through every transformation to its final destination, and metadata management is the practice of recording enough context about each dataset for someone else to trust it without re-deriving it themselves.
The practical case for lineage shows up the first time a number in a gold table looks wrong. Without lineage, tracing the error means manually reading transformation code across every step. With lineage tooling in place, either through an orchestrator like Dagster that attaches lineage to assets automatically, or a dedicated metadata catalog, you can trace the exact table and transformation step where the number diverged in minutes rather than hours.
Metadata management covers more than lineage graphs. It includes column-level descriptions, ownership records, data freshness timestamps, and quality scores attached to each table. Analysts querying a warehouse full of tables named things like tmp_report_v3 waste enormous time guessing which table is authoritative. A metadata catalog with clear ownership and descriptions solves that guesswork directly.
The medallion architecture pairs naturally with lineage tracking, since each bronze to silver to gold hop is a natural checkpoint to record what changed and why. Teams that skip metadata management early almost always end up building it retroactively, under pressure, after a data quality incident forces the question.
The Real Bottleneck Isn't the Tools
Most advice on data pipelines focuses on picking the right orchestrator or the trendiest storage format, and that misses where pipelines actually fail. They fail because nobody wrote down the contract between layers: what freshness a downstream team can expect, what happens when a record fails validation, who gets paged when ingestion goes silent. Airflow versus Dagster is a real decision, but it's a secondary one.
The conventional wisdom oversells streaming, too. Plenty of teams reach for real-time infrastructure because it sounds more sophisticated, then spend months fighting late-arriving events and exactly-once semantics for a dashboard that genuinely could have run on an hourly batch job. Streaming earns its complexity when the business impact of staleness is measurable in minutes, not when it just feels more modern.
If you take one thing from this guide, make it this: write the SLA for each layer before you write a line of transformation code. Everything else, the orchestrator, the storage format, the design pattern, is a detail you can change later. The contract between layers is the part that's expensive to fix after the fact.
— Dmitry
Get Help Building a Pipeline That Won't Break in Six Months
Requestum builds pipelines the way this guide describes: with explicit layer contracts, orchestration that surfaces failures instead of hiding them, and testing baked in before launch rather than bolted on after an incident. Our teams handle custom engineering, data science modeling, predictive analytics, and the CI/CD and QA support that keeps a pipeline stable as data volume grows.
If your team is evaluating tools for AI-assisted pipeline scaffolding, partners like AmmarAI's developer resources offer a useful starting reference for AI-aided workflows. When you're ready for hands-on help, Requestum's predictive analytics services and broader SaaS development work cover everything from a pipeline audit to a full production build. Reach out through Requestum's contact page to schedule a discovery call and get a clear scope before you commit to an architecture.
Sources
- Data pipeline architecture guide — DataDriven
- Airflow vs Dagster vs Prefect comparison — Modern Data Tools
- Extract Transform Load (ETL) — Databricks
FAQ
Will AI Replace ETL Engineers?
AI tools can generate transformation code and suggest schema mappings, but someone still has to define data contracts, judge data quality, and own the SLA when a pipeline breaks. AI is changing how ETL work gets written, not eliminating the need for engineers who understand the architecture.
Is a Data Pipeline the Same Thing as ETL?
No. ETL is one subtype of data pipeline; a data pipeline is the broader category covering any automated flow of ingestion, transformation, storage, and serving, including ELT and streaming pipelines that don't follow the classic ETL sequence.
Is Data Engineering Just ETL?
Data engineering includes ETL work but goes well beyond it, covering orchestration, infrastructure, data quality systems, security and access control, lineage and metadata management, and the platform decisions that keep pipelines running reliably at scale.
What Is an ETL Pipeline in Databricks?
A Databricks ETL pipeline typically uses SQL or Spark for transformation logic, running on Databricks' compute engine with orchestration handled either natively or through a tool like Airflow, and it follows the same extract, transform, load sequence as any other ETL system.
Should I Start With Batch or Streaming?
Start with batch unless you have a specific, measurable business need for sub-minute data freshness, since batch pipelines are simpler to build, test, and operate than streaming systems.