Brillio
Website:
brillio.com
Company:
https://www.linkedin.com/company/brillio
Seniority: Not Applicable
Industries: Artificial Intelligence
Job details:
Primary Skills
Databricks, Pyspark,sql
Specialization
- Databricks Engineering: Lead Data Engineer
Job requirements
Experience Range: With at least 5-6 years of experience in data engineering, including substantial hands-on expertise in Databricks and PySpark
Key Responsibilities:
- Design and implement robust ETL pipelines using Databricks and PySpark to efficiently process large-scale datasets
- Develop, optimize, and maintain complex SQL queries, stored procedures, and scalable data models to support analytics and reporting
- Collaborate with data architects and business stakeholders to translate requirements into high-performance data solutions
- Integrate data from diverse sources into data warehouses and modern data platforms, ensuring data quality and consistency
- Troubleshoot and resolve issues related to data ingestion, transformation, and storage within Databricks environments
- Enforce best practices in data warehousing, data modeling, and Databricks platform fundamentals
- Monitor and tune performance of data processing jobs for optimal throughput and reliability
- Document data engineering processes and provide technical guidance to team members on Databricks and PySpark usage
Required Skills:
- Advanced proficiency in Databricks platform
- Expertise in PySpark for distributed data processing and transformation
- Strong SQL skills for analytics and data manipulation
- Experience with data warehousing architectures
- Hands-on experience designing and maintaining stored procedures
- Proficiency in Apache Spark for large-scale data processing
Preferred Skills:
- Experience with Azure Databricks or AWS Glue
- Familiarity with CI/CD pipelines for Databricks workflows
- Knowledge of Databricks performance optimization techniques
- Exposure to data governance and security best practices in cloud environments
- Experience with real-time data streaming frameworks such as Apache Kafka
Desired Qualifications:
- Bachelor's degree in Computer Science, Information Technology, or a closely related discipline
- Certification in Databricks or Apache Spark
- Certification in SQL or data warehousing technologies
Click on Apply to know more.
_1548133325694.png){kind=logo}