Who Is Hiring? Search
open the tool →

Who Is Hiring? Search › Top 5 data engineering

Which Data Engineering Skills Are Most in Demand? (Kafka, Spark, Airflow, Snowflake)

The data-pipeline stack companies hire for in Hacker News 'Who is hiring?' posts: Kafka, Spark, Hadoop, Airflow and Snowflake, charted by share of postings since 2011.

Explore Top 5 data engineering in the Who Is Hiring? tool

Ranked by demand (all-time)

  1. 1.Kafka2,209 postings · peak May 2018
  2. 2.Spark2,108 postings · peak Feb 2017
  3. 3.Hadoop1,343 postings · peak Jul 2015
  4. 4.Airflow498 postings · peak May 2021
  5. 5.Snowflake371 postings · peak May 2022
2,209
2,108
1,343
498
371
tip: use | to OR several terms into one bar, e.g. backend|sre|devops

Job postings

Hover (or click) a bar segment to read the job postings that mention that term in that month.

Each calendar month since 2011 as one bar. Switch to share % to stack the bands to 100% and see each one’s slice of the category’s postings, narrow the window, or click a month to read the postings behind the bar.

What the chart shows

The data-engineering chart tracks how the pipeline stack modernized. Hadoop and Spark lead the early big-data years; Kafka becomes the near-default streaming backbone; Airflow rises as orchestration standardizes; Snowflake climbs as the cloud-warehouse era arrives and Hadoop fades. Read the relative bands as a timeline of how teams moved data, and the sample postings to see which tools cluster together - Kafka and Spark on the streaming side, Airflow and Snowflake on the warehouse side.

Popular Kafka postings (the category leader)

  1. lars_francke · read the thread · view on HN

    …Platform - SDP) bundles 12+ open source data tools (Trino, Apache Kafka, Apache Airflow, Apache Superset, etc.) and makes them work together. Right now everything is Infrastructure-as-Code only. We need someone to build a UI layer on top. No mockups, no Figma, no product roadmap…

  2. nicksnyder · read the thread · view on HN

    …schemas throughout their stack, from gRPC APIs to streaming data in Kafka. We’re deployed everywhere from small startups to the world’s largest enterprises, and we’re hiring to scale our growing business. Tech stack: Go, TypeScript, Kubernetes Open roles: https: buf.build careers

Each skill on its own

Build your own comparison on the full Who Is Hiring? chart, or see how these terms trend across all of Hacker News.