Career roadmap
Big Data Engineer
Process data at a scale where the naive approach stops working: distributed compute, streaming and lakehouse architecture.
Before you start Big Data
- Python or Scala at a solid level
- SQL and data modelling fundamentals
- Understanding of distributed systems basics
Distributed processing
Spark is the centre of this role, and its execution model is the interview.
Understanding stages, shuffles and partitions is what makes tuning possible.
- Driver, executors and cluster managers
- Jobs, stages, tasks and the DAG
- Narrow versus wide transformations
- Lazy evaluation and actions
The API you will actually write, plus the optimiser behind it.
- DataFrame API and Spark SQL
- Catalyst optimiser and predicate pushdown
- UDFs and why to avoid them
- Adaptive query execution
Nearly every slow Spark job is a shuffle problem or a skew problem.
- Why shuffles are expensive
- Partition sizing and repartitioning
- Detecting and fixing data skew
- Broadcast joins and their limits
Reading the Spark UI under time pressure is a standard interview exercise.
- Reading the Spark UI
- Memory configuration and spill
- Caching decisions
- Cluster sizing and cost
Format choice changes job runtime by an order of magnitude.
- Parquet, ORC and columnar layout
- Compression codecs and trade-offs
- Partitioning and the small file problem
- Schema evolution
BuildProcess a dataset too large for one machine, and tune the job from first run to production-ready.
Lakehouse architecture
Table formats brought transactions to the data lake and changed the architecture.
Delta, Iceberg and Hudi are the current architectural centre of gravity.
- ACID on object storage
- Delta Lake, Iceberg and Hudi compared
- Time travel and versioning
- Metadata and manifest handling
Bronze, silver and gold layers as a reliability pattern, not a fashion.
- Raw, cleaned and curated layers
- Idempotent reprocessing
- Backfill without duplication
- Retention per layer
CDC into a lake is where most real complexity lives.
- MERGE and upsert patterns
- Change data capture ingestion
- Deduplication and late data
- Deletes and compliance requirements
Lakehouses degrade without compaction. Interviews ask how you manage it.
- Compaction and file sizing
- Vacuum and retention
- Z-ordering and clustering
- Statistics and manifest health
Discovery and access control across a lake full of tables.
- Metastore and catalogue options
- Table and column-level access
- Lineage capture
- Data classification
BuildBuild a lakehouse with a table format, including time travel, upserts and compaction.
Streaming
Real-time processing has different correctness rules, and interviews probe them hard.
The backbone of nearly every streaming architecture.
- Topics, partitions and consumer groups
- Offsets, retention and compaction
- Delivery semantics
- Schema registry and evolution
Structured Streaming or Flink, with the same underlying concepts.
- Micro-batch versus continuous processing
- Checkpointing and state stores
- Exactly-once semantics in practice
- Backpressure handling
Event time versus processing time is the classic streaming interview question.
- Event time versus processing time
- Watermarks and allowed lateness
- Tumbling, sliding and session windows
- Handling out-of-order events
Joining streams to streams and to slowly changing reference data.
- Stream-stream joins and state size
- Stream-table enrichment
- State TTL and cleanup
- Lookup patterns and caching
Reconciling two pipelines that should agree but often do not.
- Lambda versus kappa architecture
- Unified batch and streaming code
- Reprocessing history
- Consistency between paths
BuildA streaming pipeline with exactly-once semantics, windowing, and a documented late-data policy.
Operating data platforms
At this scale, cost and reliability engineering become the day job.
Hundreds of interdependent jobs need real dependency management.
- Airflow or Dagster patterns
- Dependency and sensor design
- Backfill orchestration
- SLA monitoring per pipeline
Silent corruption is worse than a failed job.
- Expectation frameworks
- Anomaly detection on volume and distribution
- Quarantine patterns for bad records
- Quality SLAs with consumers
Knowing which of two hundred jobs caused the wrong number downstream.
- Lineage across the platform
- Job metrics and runtime trends
- Alerting on freshness
- Incident response for data
Big data platforms are expensive, and cost work gets noticed by leadership.
- Cluster sizing and autoscaling
- Spot instances for batch
- Storage tiering
- Attribution per pipeline
Deleting one person's data from a petabyte lake is a genuine engineering problem.
- GDPR deletion in immutable storage
- PII detection and masking
- Audit logging
- Retention enforcement
BuildInstrument a pipeline with data quality checks, SLAs and cost tracking, then cut its cost by a third.
Interview preparation
Interviews combine Spark internals, streaming semantics and pipeline design.
Internals questions and a tuning exercise are near-universal.
- Explain a shuffle and its cost
- Diagnose a skewed job
- Choose a join strategy with reasons
- Read a Spark UI screenshot
Correctness semantics, asked precisely.
- Exactly-once: what it really guarantees
- Watermark and late data handling
- State growth and mitigation
- Failure recovery behaviour
Design ingestion and processing for a stated volume and latency.
- Volume and latency requirements first
- Storage and format choices
- Batch versus streaming decision
- Cost estimate and trade-offs
Usually PySpark or SQL against a realistic transformation problem.
- Transformations in PySpark
- Complex SQL aggregation
- Deduplication and windowing
- Writing testable pipeline code
Public pipelines with performance numbers are strong evidence.
- Batch and streaming implementations
- A tuning case study with timings
- Cost optimisation write-up
- Data quality framework you built
BuildA public repository with a batch and a streaming pipeline over the same data, and a comparison write-up.
Big Data tools on your CV
- Apache Spark
- Kafka
- Delta Lake / Iceberg
- Flink
- Airflow
- Databricks
- Python / Scala
- Parquet
What Big Data employers ask to see
- A Spark tuning case study with before/after runtimes
- A streaming pipeline with documented exactly-once handling
- A lakehouse implementation with compaction and time travel
- A pipeline cost reduction with measured savings
Specialised end of data engineering, concentrated in large enterprises, ad-tech, fintech and anywhere with genuine volume. Fewer roles than general data engineering, and better paid.
Content last reviewed 2026-08-31. Guidance only — no institute or paid placement is endorsed anywhere in this book.