Senior Staff Software Engineer - Lakeflow Pipelines Datasets

Databricks builds the world's leading data and AI platform, used by tens of thousands of organizations to turn raw data into decisions that matter. The Lakeflow Engineering team owns the entirety of the ETL product line: Declarative Dataflow Graphs, Materialized Views, Structured Streaming, and Flows. We run what is arguably the world's largest data engineering platform, processing exabytes of data daily across tens of thousands of customers. The problems we solve (fault-tolerant incremental computation, stream-batch unification, declarative pipeline orchestration at massive scale) sit at the frontier of what distributed data systems can do. We're looking for a Senior Staff Engineer to serve as a technical architect and engineering leader for this platform. You'll own the long-term technical direction for how pipelines are defined, optimized, executed, and operated, shaping the systems that data teams worldwide depend on every day. You will work across one or more of the following areas to design and build the next generation of data infrastructure: • Declarative Dataflow Processing • Resource management • Efficient storage structures • Stream Processing and Complex Event Processing • Autonomic Computing and Self-Managing Systems The Impact you will have: • Architect and deliver a highly scalable, fault-tolerant platform for declarative pipelines and materialized views serving thousands of production customers. • Turn ambiguous, large-scale technical challenges into clear execution plans that ship incrementally and hold up at exabyte scale. • Bridge the gap between research and production by bringing ideas from the academic literature or internal R&D into systems that solve real customer problems. • Lead deep systems work: performance diagnosis on large production clust…