AWS Glue Engineer

Data Pipeline Development: Design, implement, and maintain ETL (Extract, Transform, Load) pipelines using AWS Glue to process and transform data from various sources (e.g., relational databases, data lakes, streaming services). Work with AWS Glue Crawlers to automate data discovery and schema inference for large-scale datasets. Develop and optimize data transformation jobs using AWS Glue Spark for complex transformations, aggregations, and computations. Cloud Data Architecture: Architect and build scalable data processing solutions using AWS Glue, Amazon S3, AWS Lambda, Amazon Redshift, Amazon RDS, and other AWS services. Integrate AWS Glue with other AWS ecosystem tools, such as AWS S3, AWS Data Pipeline, and AWS Athena, for seamless data operations and analytics. Design and manage data lakes, ensuring efficient data storage, transformation, and retrieval. Data Integration & Automation: Automate the process of data loading, transformation, and extraction to and from cloud storage and databases using AWS Glue. Develop scripts and workflows to schedule, monitor, and automate ETL jobs using AWS Glue workflows and AWS Lambda. Implement data quality and validation checks to ensure that data is accurate and reliable. Performance Tuning & Optimization: Optimize performance for large-scale data processing workflows within AWS Glue, ensuring minimal downtime and quick processing times. Profile, troubleshoot, and resolve issues with AWS Glue jobs and other data workflows. Monitor and improve the efficiency and cost-effectiveness of ETL processes by leveraging AWS CloudWatch and other monitoring tools. Collaboration & Stakeholder Management: Collaborate with data scientists, analysts, and business teams to define data requirements and deliver actionable insights. Provide technical leadership and guidance on best practices for data integration and automation in AWS. Ensure effective communication and documentation of data workflows and system architecture. Security & Compliance: Implement and manage data security, privacy, and compliance standards within the AWS Glue ecosystem. Ensure compliance with industry standards (GDPR, HIPAA, etc.) in data processing and storage. Apply best practices for data encryption, data masking, and auditing to protect sensitive data.

We are excited to partner with a renowned and forward-thinking organization committed to delivering cutting-edge software solutions. Known for its innovation and excellence, our client is seeking a highly skilled AWS Glue Engineer to join their growing team. This is an excellent opportunity to contribute to the development of high-quality, scalable, and efficient software applications in a dynamic and collaborative environment.