Data Engineer III (Databricks)

LivePerson (NASDAQ: LPSN) is the global leader in enterprise conversations. Hundreds of the world's leading brands — including HSBC, Chipotle, and Virgin Media — use our award-winning Conversational Cloud platform to connect with millions of consumers. We power nearly a billion conversational interactions every month, providing a uniquely rich data set and safety tools to unlock the power of Conversational AI for better customer experiences. At LivePerson, we foster an inclusive workplace culture that encourages meaningful connection, collaboration, and innovation. Everyone is invited to ask questions, actively seek new ways to achieve success, and reach their full potential. We are continually looking for ways to improve our products and make things better. This means spotting opportunities, solving ambiguities, and seeking effective solutions to the problems our customers care about. Overview: We are looking for an experienced Data Engineer to provide data engineering expertise and support to various analytical products of LivePerson, and assist in migrating our existing data processing ecosystem from Hadoop (Spark, MapReduce, Java, and Scala) to Databricks on GCP. The goal is to leverage Databricks' scalability, performance, and ease of use to enhance our current workflows. You will: Assessment and Planning: Review the existing Hadoop infrastructure, including Spark and MapReduce jobs. Analyze Java and Scala codebases for compatibility with Databricks. Identify dependencies, libraries, and configurations that may require modification. Propose a migration plan with clear timelines and milestones. Code Migration: Refactor Spark jobs to run efficiently on Databricks. Migrate MapReduce jobs where applicable or rewrite them using Spark DataFrame/Dataset API. Update Java and Scala code to comply with Databricks' runtime environment. Testing and Validation: Develop unit and integration tests to ensure parity between the existing and new systems. Compare performance metrics before and after migration. Implement error handling and logging consistent with best practices in Databricks. Optimization and Performance Tuning: Fine-tune Spark configurations for performance improvements on Databricks. Optimize data ingestion and transformation processes. Deployment and Documentation: Deploy migrated jobs to production in Databricks. Document changes, configurations, and processes thoroughly. Provide knowledge transfer to internal teams if required.

LivePerson (NASDAQ: LPSN) is the global leader in enterprise conversations. Hundreds of the world's leading brands — including HSBC, Chipotle, and Virgin Media — use our award-winning Conversational Cloud platform to connect with millions of consumers. We power nearly a billion conversational interactions every month, providing a uniquely rich data set and safety tools to unlock the power of Conversational AI for better customer experiences.