JPJobPrepfull-stack interview
RoadmapsJS CompilerStar on GitHub
Career Roadmaps · Data Engineer

Career roadmap

Data Engineer

Build the pipelines and models every analyst, dashboard, and AI feature depends on. Own freshness and trust.

Time
6-10 months part-time
Entry bar
Good entry route for analysts and backend developers. Low maths requirement.
Stages
5 · 23 topics
0/23 studied0%
Read the full guide →All roadmaps

Before you start Data Engineer

  • SQL beyond SELECT
  • Python scripting
  • Linux command line
  • Git

SQL & data modelling

4-6 weeks · 0/4 topics

Model data so the next question is cheap to answer.

  1. Window functions and CTEs are the daily tools, not exotica.

    Ch 38 — Indexing
    • Window functions
    • CTEs and recursive CTEs
    • Set operations
    • Query plans and indexes

  2. Facts and dimensions, chosen so analysts stop asking you for help.

    • Star vs snowflake
    • Grain of a fact table
    • Conformed dimensions
    • Surrogate keys

  3. History is a requirement the first time someone asks 'what was the price then?'

    • SCD type 1 vs 2
    • Effective dating
    • Point-in-time joins

  4. Pick the store from the access pattern, not from familiarity.

    Ch 41 — SQL vs NoSQL
    • OLTP vs OLAP
    • Document vs relational
    • When a warehouse is the wrong answer

BuildA star schema over a public dataset, with the ten business questions it answers in one query each.

Pipelines

6-8 weeks · 0/5 topics

Move data on a schedule without babysitting it.

  1. Land raw, transform in-warehouse, keep the raw layer replayable.

    • Raw / staging / marts layering
    • When to transform before landing
    • Replayability

  2. Airflow, Dagster, or Prefect — one of them, properly.

    • DAGs and dependencies
    • Retries and SLAs
    • Backfills
    • Sensors vs schedules

  3. A rerun must produce the same result, not duplicate rows.

    • Merge on a unique key
    • Delete-and-insert by partition
    • Watermarks
    • Late-arriving data windows

  4. Version-controlled SQL with tests attached. The industry default.

    • Incremental models
    • Schema and data tests
    • Sources and freshness
    • Documentation and lineage

  5. A loud failure is fine. A pipeline that silently stops updating is what gets people fired.

    • Freshness SLAs
    • Row-count anomaly checks
    • Alert routing and on-call

BuildA scheduled pipeline with dbt tests, a backfill command, and a freshness alert that actually fires.

Storage & scale

6-8 weeks · 0/5 topics

Handle data too big for one machine, without a surprise cloud bill.

  1. BigQuery, Snowflake, or Redshift — learn its pricing model, not just its SQL.

    • Storage vs compute separation
    • Slots / warehouses / credits
    • Result caching
    • Cost attribution

  2. Why analytics stores columns, and what that buys you.

    • Row vs columnar layout
    • Compression and encoding
    • Predicate pushdown
    • Small-file problems

  3. Iceberg or Delta: ACID and time travel on object storage.

    • Table formats
    • Schema evolution
    • Time travel and snapshots
    • Compaction

  4. The single biggest lever on warehouse cost.

    • Choosing a partition key
    • Clustering / sort keys
    • Pruning and why it breaks
    • Avoiding SELECT *

  5. Needed when a single warehouse query is the wrong tool.

    • Lazy evaluation and stages
    • Shuffles and skew
    • Broadcast joins
    • Tuning partitions

BuildThe same query on unpartitioned and partitioned data, with the cost and runtime difference measured.

Streaming

4-6 weeks · 0/4 topics

Some questions cannot wait for tomorrow's batch.

  1. Topics, partitions, consumer groups — and what ordering actually guarantees.

    • Partitioning and keys
    • Consumer groups and offsets
    • Retention and compaction
    • Schema registry

  2. At-least-once plus idempotent writes is how most teams get 'exactly once'.

    • At-most / at-least / exactly once
    • Deduplication keys
    • Transactional writes

  3. Replicate a production database without adding load to it.

    • Debezium and logical replication
    • Snapshot plus stream
    • Schema changes mid-stream
    • Lag monitoring

  4. Event time is not arrival time, and pretending otherwise corrupts metrics.

    • Tumbling vs sliding windows
    • Watermarks
    • Reprocessing late events

BuildA CDC stream from Postgres to the warehouse with a measured end-to-end lag figure.

Reliability & governance

4-6 weeks · 0/5 topics

Being trusted is the job. Wrong data is worse than no data.

  1. Producers commit to a schema; consumers stop guessing.

    • Schema ownership
    • Breaking vs additive changes
    • Enforcement in CI

  2. Tests on data, not just on code.

    • Not-null, unique, referential checks
    • Distribution and volume anomalies
    • Great Expectations / dbt tests

  3. Before you change a column, know who breaks.

    • Column-level lineage
    • Downstream impact reports
    • Deprecation process

  4. Who can see what, provably.

    • PII classification
    • Row and column level security
    • Masking
    • Retention policy

  5. Data incidents need the same discipline as service incidents.

    • Runbooks per pipeline
    • Incident write-ups
    • Backfill after an outage

BuildA data contract plus quality checks that block a bad upstream change before it reaches a dashboard.

Data Engineer tools on your CV

  • SQL and Python
  • dbt
  • Airflow or Dagster
  • BigQuery or Snowflake
  • Kafka
  • Spark
  • Terraform basics

What Data Engineer employers ask to see

  • One pipeline running on a schedule you did not fix by hand
  • Tests and a freshness SLA someone else could rely on
  • A cost or runtime optimisation with numbers

Steadiest demand of these tracks and the most transferable. Every company with AI ambitions discovers its data layer first — half of 'AI projects' are data projects in a costume. Strong path in from SQL analyst or backend work.

Content last reviewed 2026-08-31. Guidance only — no institute or paid placement is endorsed anywhere in this book.