Scope of Work
1. Full time role with Brillio
2. Experience Range: 10 years of experience in data architecture and AWS data engineering Key
Responsibilities
1. Design and implement scalable data architectures on AWS, leveraging services such as EMR, Redshift, DynamoDB, and Kinesis
2. Develop robust data pipelines using Spark (Scala), Oozie, and AWS Glue to support analytics and operational workloads
3. Configure and optimize AWS services including Athena, Open Search, CloudWatch, Macie, and CloudFormation for efficient data processing and monitoring
4. Manage and secure data flows with Amazon SNS, SQS, API Gateway, and DMS, ensuring data integrity and compliance
5. Collaborate with cross-functional teams to translate business requirements into technical solutions, focusing on performance, security, and reliability
6. Troubleshoot and resolve complex issues related to data ingestion, transformation, and storage within AWS environments
7. Implement best practices for data governance, security, and privacy, utilizing AWS Macie and related tools
8. Monitor system performance and proactively identify opportunities to improve data processing efficiency and reduce costs
Required Skills
1. Advanced proficiency with AWS services including Athena, SNS, SQS, CloudWatch, Macie, Kinesis, CloudFormation, EMR, Open Search, DynamoDB, Amazon API Gateway, SCT, Redshift, and DMS
2. Expertise in Spark with Scala for large-scale data processing
3. Hands-on experience with Oozie for workflow orchestration
4. Strong knowledge of data modelling and architecture in cloud environments
5. Proficiency in designing and optimizing ETL pipelines
6. Experience with real-time and batch data processing solutions
7. Ability to configure and manage AWS security and monitoring tools
8. Skilled in deploying infrastructure as code using AWS CloudFormation
9. Competence in integrating AWS services for end-to-end data solutions
10. Solid understanding of data governance and compliance in AWS
Preferred Skills
1. Experience with machine learning pipelines on AWS
2. Knowledge of advanced AWS analytics services such as AWS Glue and Quick Sight
3. Familiarity with containerization technologies like Docker and Kubernetes
4. Expertise in automating data quality and validation processes
5. Background in implementing data cataloguing and lineage solutions
6. Experience with migration tools and strategies for legacy data systems
Desired Qualifications
1. Bachelor’s degree in computer science, Information Technology, Engineering, or a related field
2. AWS Certified Data Analytics – Specialty or AWS Certified Solutions Architect certification preferred
Role of Resource
Collaborating with clients, teams and stack holders and involved in late night calls around 10:30AM
_1548133325694.png){kind=logo}
