Innovya Technologies
Website:
innovyatech.com
Job details:
We are looking for an experienced Databricks Engineer to lead the design and development of high-performance and scalable enterprise solutions using PySpark,, Databricks etc. The ideal candidate will have over 10 years of experience in software development role, designing enterprise-grade, fault-tolerant, even-driven applications. You will work closely with cross-functional teams to design, build and deploy complex systems, providing technical direction and expertise to ensure the delivery of robust, efficient, and scalable solutions.
Key Responsibilities:
- Architect, design, develop and deploy high-performance and scalable data pipe lines -based applications that meet business complex requirements.
- Familiarity with streaming architectures and patterns such as event-driven pipelines, near real-time scoring, and anomaly monitoring.
- Experience working with high-volume, sensitive data while adhering to security, compliance, and privacy guidelines.
- Proficiency in Python for data processing, automation, API integration, anomaly-detection scripts, and model-ready dataset preparation
- Lead a team of engineers and work with cross-functional team to timely deliver high volume data pipeline and streaming solutions.
- Provide technical leadership across all aspects of the software development lifecycle, from initial design through production deployment.
- Design and implement data pipelines using PySpark, DataBricks, Java and related tech stack.
- Ensure high availability and scalability of systems using Kubernetes, containerization, and cloud infrastructure.
- Implement and manage schedulers, event-driven architecture, and asynchronous processes.
- Collaborate with DevOps and infrastructure teams to automate deployment, scaling, and monitoring of applications.
- Drive the adoption of best practices in coding, design, testing, and deployment to improve team productivity.
- Strong SQL skills, including query optimization, performance tuning, and working with both relational and non-relational stores.
Required Skills:
- 10+ years of total experience in software development, with at least 5 years in a design lead role.
- Deep experience with PySpark for distributed data processing, data quality validation, data enrichment, and feature engineering
- Excellent problem-solving, analytical, and interpersonal skills.
- Extensive hands-on experience on spring-boot, Kafka, and API development activities.
- Experience in designing scalable, distributed systems and microservices architecture.
- Familiarity with schedulers, event-driven architecture, and messaging systems (e.g., Kafka, RabbitMQ).
- Proficiency in working with cloud platforms such as AWS and Azure.
- Hands-on experience with caching strategies (ECH), performance tuning, and security best practices.
- Experience with version control systems (Git), CI/CD pipelines, and Agile methodologies.
- Experience of working with relational and NoSQL databases.
- .
Soft Skills
- Strong communication and collaboration abilities.
- Ability to work independently and as part of a team.
- A proactive mindset focused on quality and continuous improvement.
- Strong analytical and debugging skills.
What We Offer
- Competitive salary and benefits plan.
- Opportunity to work with modern test automation tools and cloud technologies.
- Collaborative and innovative work environment.
- Opportunities for professional growth and advancement.
Click on Apply to know more.
