IntraEdge
Website:
intraedge.com
Company:
https://www.linkedin.com/company/intraedge
Seniority: Associate
Industries: Software Development, IT Services and IT Consulting, and Banking
Job details:
ETL SQL / Enterprise Data Framework Developer
Location: Hyderabad
Experience: 3–6 Years
Employment Type: Full-Time
Job Summary
We are seeking an experienced ETL SQL / Data Engineer with strong expertise in SQL development, ETL data pipelines, Stored Procedures, Views, Autosys scheduling, and Enterprise Data Framework (EDF).
The candidate will be responsible for developing and maintaining data pipelines that extract data from multiple source systems, process and validate the data through EDF modules, and load it into target platforms such as Data Lake and Snowflake.
The ideal candidate should have strong SQL coding skills, hands-on experience with DBeaver, and a good understanding of data integration using flat files and REST APIs. Experience working with enterprise-scale data pipelines and scheduling frameworks is highly desirable.
Key Responsibilities
ETL & SQL Development
- Develop, maintain, and optimize ETL SQL code for enterprise data pipelines.
- Write complex SQL queries for data extraction, transformation, validation, reconciliation, and loading.
- Develop and maintain:
- Stored Procedures
- Views
- SQL scripts
- Data transformation logic
- Data validation and reconciliation queries
- Analyze source data and determine appropriate transformation and mapping logic.
- Optimize SQL queries for performance and scalability.
- Perform data quality checks and troubleshoot data discrepancies.
- Ensure data pipelines meet defined business and technical requirements.
Enterprise Data Framework (EDF)
- Develop new Enterprise Data Framework (EDF) jobs to support enterprise data processing requirements.
- Work extensively with EDF to design, build, test, deploy, and support data pipelines.
- Understand and utilize EDF modules including:
- Data Processing Pipeline (D2P)
- Data Control and Anomaly Detection Framework (DCAF)
- General-Purpose Data Reconciliation (GPR)
- Build EDF jobs to extract data from source systems and process the data through the appropriate EDF modules.
- Configure data processing, validation, anomaly detection, and reconciliation rules.
- Ensure successful movement of data across different stages of the pipeline.
- Troubleshoot EDF job failures and data processing issues.
- Develop reusable and maintainable EDF components wherever possible.
Data Extraction & Integration
- Build data extraction processes from multiple source systems.
- Work with flat files as source data, including file-based ingestion and processing.
- Develop integrations using REST APIs for data ingestion.
- Understand API request/response structures and troubleshoot API-related data ingestion issues.
- Validate incoming data for completeness, accuracy, format, and quality.
- Handle different data formats and transformation requirements.
- Ensure reliable movement of data from source systems through the EDF pipeline.
Data Processing & Transformation
- Process extracted data through EDF's D2P, DCAF, and GPR modules.
- Implement transformation and business rules required for downstream processing.
- Develop data validation and anomaly detection mechanisms.
- Implement reconciliation logic to compare source and target datasets.
- Investigate data quality issues and work with upstream teams to resolve them.
- Ensure data is accurately transformed before loading into target environments.
Target Data Platforms
- Build and support pipelines that load processed data into target environments such as:
- Data Lake
- Snowflake
- Validate successful data loading and perform post-load data quality checks.
- Reconcile source and target records to ensure completeness and accuracy.
- Troubleshoot data load failures and performance issues.
- Work with Data Engineering and platform teams to resolve target-system issues.
Autosys Scheduling
- Develop, configure, and maintain Autosys jobs for data pipeline scheduling.
- Define job dependencies, calendars, conditions, and execution sequences.
- Monitor scheduled ETL and EDF jobs.
- Troubleshoot failed or delayed jobs.
- Manage dependencies between upstream and downstream data processes.
- Support production scheduling and batch-processing requirements.
Data Quality & Reconciliation
- Implement data quality checks throughout the ETL lifecycle.
- Utilize DCAF capabilities for anomaly detection and data control.
- Develop reconciliation processes using GPR.
- Investigate data mismatches between source and target environments.
- Identify data anomalies and work with relevant teams to resolve them.
- Ensure data pipelines meet defined quality and completeness standards.
Development, Testing & Deployment
- Participate in the complete development lifecycle:
- Requirement analysis
- Technical design
- Development
- Unit testing
- Integration testing
- Deployment
- Production support
- Develop test cases for SQL, ETL, and EDF jobs.
- Perform unit and integration testing of data pipelines.
- Validate data transformation and reconciliation results.
- Support UAT and production deployments.
- Troubleshoot defects and implement fixes within agreed timelines.
Production Support
- Monitor ETL and EDF pipelines and proactively identify failures.
- Analyze job logs and error messages to determine root causes.
- Resolve production issues related to SQL, ETL, Autosys, APIs, flat files, and EDF.
- Perform root cause analysis for recurring data pipeline failures.
- Coordinate with application, infrastructure, source-system, and database teams.
- Participate in incident and problem management activities.
Required Technical Skills
SQL
- Strong hands-on SQL coding experience.
- Experience developing complex SQL queries.
- Strong knowledge of:
- Stored Procedures
- Views
- Joins
- Subqueries
- CTEs
- Window Functions
- Aggregations
- Data validation
- Data reconciliation
- Ability to optimize SQL queries and troubleshoot performance issues.
DBeaver
- Hands-on experience using DBeaver for SQL development and database analysis.
- Ability to analyze database objects, execute queries, troubleshoot SQL issues, and validate data.
Enterprise Data Framework
- Experience developing or supporting Enterprise Data Framework (EDF) jobs is highly preferred.
- Understanding of:
- D2P – Data Processing Pipeline
- DCAF – Data Control and Anomaly Detection Framework
- GPR – General-Purpose Data Reconciliation
- Ability to develop new EDF jobs and troubleshoot existing pipelines.
Data Sources
Strong understanding of data extraction from:
- Flat files
- REST APIs
- Relational databases
- Enterprise source systems
Data Targets
Experience working with:
- Data Lake
- Snowflake
- Cloud-based data platforms
Scheduling
- Hands-on experience with Autosys.
- Knowledge of job scheduling, dependencies, calendars, triggers, and batch processing.
Preferred Skills
- Experience with Snowflake.
- Experience working with cloud-based Data Lakes.
- Knowledge of Python or shell scripting.
- Experience with REST API integration.
- Understanding of JSON/XML data formats.
- Experience with data quality and data governance concepts.
- Knowledge of CI/CD and source-control tools such as Git.
- Exposure to Agile/Scrum development methodologies.
- Experience working with enterprise-scale data platforms.
Key Responsibilities at a Glance
The selected candidate will primarily be responsible for:
- Developing ETL SQL code, Stored Procedures, and Views.
- Building new EDF jobs for enterprise data pipelines.
- Extracting data from flat files and REST APIs.
- Processing data through D2P, DCAF, and GPR modules.
- Implementing data validation, anomaly detection, and reconciliation.
- Loading processed data into Data Lake or Snowflake.
- Creating and managing Autosys schedules and dependencies.
- Performing data quality checks and troubleshooting pipeline issues.
- Supporting testing, deployment, and production operations.
- Working with cross-functional teams to ensure reliable and accurate data delivery.
Candidate Profile
- 3–6 years of experience in Data Engineering, ETL Development, SQL Development, or Data Integration.
- Strong SQL programming and data analysis skills.
- Hands-on experience with DBeaver.
- Strong understanding of ETL concepts and data pipeline development.
- Experience with Autosys scheduling.
- Experience with flat-file and REST API-based data ingestion is mandatory.
- Experience with EDF or a similar enterprise data pipeline framework is highly preferred.
- Strong understanding of data quality, reconciliation, and transformation.
- Good troubleshooting and analytical skills.
- Ability to work independently as well as collaboratively with distributed technical teams.
Click on Apply to know more.
