Staff Software Engineer II - Apache Iceberg

Lensa is the leading career site for job seekers at every stage of their career. Our client, Confluent, is seeking professionals in Madison, WI. Apply via Lensa today!

With Confluent, organizations can harness the full power of continuously flowing data to innovate and win in the modern digital world. We have a purpose that drives us to do better every day – we're creating an entirely new category within data infrastructure - data streaming. This technology will allow every organization to create experiences and use the power of data in ways that profoundly impact the way we all live. This impact is our purpose and drives us to do better every day.

One Confluent. One team. One Data Streaming Platform.

Data Connects Us.

About The Role

We are a team of passionate engineers who love solving complex distributed storage systems and infrastructure problems. Through the Tableflow (https://www.confluent.io/product/tableflow/) project, we are building the next generation of storage solutions for Kora, transforming it into a true multi-modal storage engine. Our goal is to enable streaming data to be consumed both as a table and as a stream seamlessly. If you're interested in learning more about the challenges we are tackling, check out our blog post: Tableflow: The Stream/Table, Kafka/Iceberg Duality. (https://jack-vanlightly.com/blog/2024/3/19/tableflow-the-stream-table-kafka-iceberg-duality)

As a Staff Software Engineer on the team, you will focus on advancing key layers of the Tableflow project, including its multi-tenant, highly available compute infrastructure, and the core components responsible for table materialization and background maintenance. You will tackle challenges such as developing strategies for efficient table materialization and maintenance at massive scale across all of Confluent Cloud, ensuring high availability and scalability for background storage tasks. Additionally, you will drive contributions to open-source projects such as Apache Iceberg, while mentoring junior engineers and helping shape the direction of the team.

What You Will Do

• Storage Engine Development: Design and implement innovative strategies for table schematization, materialization, and compaction at massive scale.
• Platform Engineering: Build and maintain multi-tenant, highly available infrastructure that powers background computational tasks for Kora’s long-term storage. Ensure that the platform is optimized for scalability, reliability, and resilience, in alignment with organizational objectives.
• Open-Source Leadership: Guide and mentor team members in contributing to relevant open-source projects such as Apache Iceberg. Advocate for best practices in open-source collaboration and foster a culture of giving back to the community.
• Engineering Leadership: As a staff engineer, set the standard for engineering excellence. Act as a trusted advisor for both internal and cross-functional teams, providing guidance on complex technical problems and contributing to high-level decision-making processes.Cross-Functional Collaboration: Collaborate with product management, design, and other engineering teams to ensure seamless integration of storage features and infrastructure with the broader organization.
  
  

What You Will Bring

• BS, MS, or PhD in computer science or a related field, or equivalent work experience
• Proven experience designing and building large-scale, distributed storage systems.
• Deep knowledge of distributed systems concepts, including fault tolerance, consistency, and scalability.
• Experience with open-source contributions, particularly in the areas of storage, data lakes, or stream processing (e.g., Apache Iceberg, Apache Kafka, Apache Flink).
• Expertise in modern cloud-based infrastructure and multi-tenant environments.
• Strong leadership skills with a demonstrated ability to mentor and guide engineers to success.
• Excellent communication and collaboration skills, with the ability to influence and guide stakeholders at all levels.
  
  

What Gives You An Edge

• Familiarity with or has experience contributing to the following open-source technologies: Apache Kafka, Apache Iceberg, Apache Flink
• Experience/knowledge with public clouds (AWS, Azure or GCP)
• Interest in evangelism (giving talks at tech conferences, writing blog posts evangelizing your work)
  
  

Come As You Are

At Confluent, equality is a core tenet of our culture. We are committed to building an inclusive global team that represents a variety of backgrounds, perspectives, beliefs, and experiences. The more diverse we are, the richer our community and the broader our impact. Employment decisions are made on the basis of job-related criteria without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other classification protected by applicable law.

At Confluent, we are committed to providing competitive pay and benefits that are in line with industry standards. We analyze and carefully consider several factors when determining compensation, including work history, education, professional experience, and location. This position has an annual estimated salary of $252,000- $296,100 and a competitive equity package. The actual pay may vary depending on your skills, qualifications, experience, and work location. In addition, Confluent offers a wide range of employee benefits. To learn more about our benefits click HERE (https://confluentbenefits.com/) .

Click HERE (https://www.confluent.io/legal/confluent-candidate-privacy-notice/) to review our Candidate Privacy Notice which describes how and when Confluent, Inc., and its group companies, collects, uses, and shares certain personal information of California job applicants and prospective employees.

Lensa is the leading career site for job seekers at every stage of their career. Our client, Confluent, is seeking professionals in Madison, WI. Apply via Lensa today!