Data Engineering & Big Data
11 clusters, spanning half-life layers 0–3: Distributed processing, Spark job interfaces, Single-node analytics, Streaming & messaging, Workflow orchestration, Data warehouses, Lakehouse & formats, ELT / transformation, Quality, lineage & governance, Analytics & BI, Core concepts.
Clusters
Section titled “Clusters”Distributed processing
Partitioning, shuffle, lazy evaluation and the cost of data movement transfer across every engine and are the actual skill. Batch↔streaming transfer…
Spark job interfaces
The submit/session distinction and where the driver runs transfer across all three. Everything else is API surface.
Single-node analytics
Columnar execution and lazy vs eager evaluation transfer between DuckDB and Polars. pandas is the odd member — its eager, index-centric model…
Streaming & messaging
The log abstraction, partitions, consumer groups, offsets and delivery guarantees transfer across all of them and are Layer 1 distributed-systems…
Workflow orchestration
DAGs, idempotent tasks, backfills, retries and scheduling semantics transfer across every member. Authoring APIs differ enough that a migration is a…
Data warehouses
Columnar storage, partitioning, clustering and the separation of storage from compute transfer across all of them. Pricing models differ enough to…
Lakehouse & formats
Table formats over object storage, ACID via a metadata log, schema evolution and time travel transfer across all three. Parquet is the substrate…
ELT / transformation
Declarative models, lineage, incremental strategies and testing transformations transfer, principally from dbt outward — dbt taught the rest of the…
Quality, lineage & governance
Expectation-based validation and column-level lineage are the durable ideas. Every tool implements them differently and none interoperates cleanly.
Analytics & BI
Semantic modelling and the grain of a metric transfer across all of them and are the real skill. Everything visual is per-tool.
Core concepts
Layer 0. Dimensional modelling and CDC reasoning are older than every product in this domain and will outlive them.