Mastech Digital
Website:
mastechdigital.com
Job details:
Job Title: StreamSets Developer/Engineer
Location: Chennai/Bangalore/Hyderabad (Hybrid)
Duration: Full time
Notice period: Immediate joiner only/Currently serving notice with less than 45 days/Notice is less than 30 days
Relevant : 3+ Years
StreamSets Developer/Engineer:
➢ Designing, developing, and deploying data pipelines using StreamSets
➢ Configuring data sources, transformations, and destinations within pipelines.
➢ Monitoring and troubleshooting pipeline performance and errors.
➢ Collaborating with data architects and stakeholders to understand data requirements and ensure pipeline efficiency.
➢ Implementing best practices for data governance, security, and compliance.
➢ Basic understanding of Apache Kafka architecture, including topics, partitions, brokers, and consumer groups.
Responsibilities:
Pipeline Development:
➢ Designing and developing data pipelines using StreamSets based on business requirements.
➢ Implementing data transformations, validations, and enrichments within pipelines.
➢ Testing pipelines for performance, reliability, and data quality. Pipeline Monitoring and Maintenance:
➢ Monitoring pipeline performance metrics such as throughput, latency, and error rates.
➢ Troubleshooting and resolving issues related to pipeline failures or data inconsistencies.
➢ Optimizing pipelines for efficiency and scalability as data volumes grow. Security and Compliance:
➢ Ensuring that data handling practices within StreamSets comply with organizational security policies and regulatory requirements.
➢ Implementing encryption, access controls, and auditing mechanisms to protect sensitive data. Documentation and Knowledge Sharing:
➢ Documenting pipeline configurations, dependencies, and operational procedures.
➢ Sharing knowledge and best practices with team members through training sessions, documentation, and collaboration tools.
Performance Optimization:
➢ Analyzing pipeline performance bottlenecks and optimizing configurations for improved efficiency.
➢ Scaling pipelines to handle increasing data volumes and processing requirements. Collaboration and Communication:
➢ Collaborating with cross-functional teams including developers, data scientists, and business analysts to understand data requirements and ensure alignment with business objectives.
➢ Communicating updates, issues, and resolutions effectively with stakeholders.
Click on Apply to know more.