Website:
virdapt.com
Job details:
JOB DESCRIPTION
WORKPLACE TYPE: Remote (Canada)
EMPLOYMENT TYPE: Contract
PAY RATE: CAD 50/HR
IMMEDIATE PROJECT REQUIREMENT - C2C | REMOTE ACROSS CANADA
Virdapt, the technology vertical of Virdiss Inc., is hiring a Data Engineer (GCP) for a live client engagement. This is a C2C contract role, fully remote within Canada, with work already scoped and ready to start. We are moving fast - candidates available on short notice will be prioritised.
This is a hands-on delivery role on a client-facing team. You will be expected to contribute production code early, with limited ramp-up time.
Location: Canada - Remote (Toronto preferred, open across Canada) | Type: Contract, C2C | Experience: 4-5 years hands-on
Interview process: 1 internal technical interview + 2 client interviews
PLEASE READ BEFORE APPLYING
GCP experience is mandatory. Profiles with no prior hands-on, production GCP experience will not be considered - regardless of AWS or Azure background. This engagement cannot accommodate cloud-transition candidates.
ABOUT THE ROLE
You will help design, build and scale a modern, GCP-native enterprise data platform: batch and real-time streaming pipelines, Infrastructure as Code, CI/CD automation, and governed, analytics-ready data models.
KEY RESPONSIBILITIES
1. Data pipeline and workflow engineering
- Design and optimise scalable streaming pipelines using Pub/Sub and Dataflow (Apache Beam), including dynamic schema/attribute validation and resilient Dead Letter Queue (DLQ) patterns.
- Build, schedule and maintain complex workflow DAGs in Cloud Composer (managed Apache Airflow) - custom operators, sensors, backfills, retries and SLA handling.
- Develop and performance-tune PySpark jobs on Dataproc for high-volume batch and streaming workloads, through to production stability.
- Write production-grade Python and advanced SQL transformations that ingest raw landing data, apply deduplication and cleansing logic, and model it through clean enterprise layers (Data Lake to Data Warehouse to Golden Layer).
2. Data warehousing and platform architecture
- Architect and optimise BigQuery datasets, data models and partitioning/clustering strategies for performant, cost-controlled analytics.
- Monitor and resolve pipeline backlogs, autoscaling bottlenecks, worker load balancing, data skew and memory/reshuffle issues.
- Define grain, build fact and dimension models, and handle slowly changing dimensions and late-arriving data.
3. Infrastructure as Code and DevOps
- Own Terraform-managed infrastructure across dev, staging and production - modules, state and environment promotion.
- Build and maintain CI/CD pipelines for data workflows, Flex Templates and infrastructure code.
- Implement end-to-end monitoring, alerting and logging with Cloud Monitoring / Cloud Logging.
- Track and optimise GCP spend across BigQuery, Dataflow and Dataproc workloads.
4. Data governance and quality
- Contribute to governance and metadata management using Dataplex / Data Catalog.
- Implement validation, source-to-target reconciliation and integrity checks in every pipeline.
- Apply least-privilege IAM, service accounts and appropriate handling of sensitive data.
5. Collaboration and delivery
- Contribute to architecture and design discussions directly with the client's technical stakeholders.
- Participate in code review and uphold engineering standards.
- Document architecture, data lineage and runbooks so the platform is operable by others.
- Work independently in a remote, delivery-focused team with minimal supervision.
MANDATORY REQUIREMENTS
Every item below is a hard requirement. A profile missing any one of them will not be shortlisted.
EXPEREINCE
- 4-5 years hands-on data engineering (excluding internships and training), with clear personal ownership of at least one end-to-end production pipeline on GCP.
GCP
You must have built and operated production workloads on GCP and be able to discuss in detail:
- BigQuery: advanced SQL, dataset architecture, partitioning and clustering strategy, slot/reservation behaviour, cost control, materialized views, nested and repeated data (STRUCT, ARRAY), external/BigLake tables, and a specific slow or expensive query you diagnosed and fixed.
- Cloud Composer / Apache Airflow: DAG development, custom operators, dependency management, backfills, retries and debugging failed runs in a live schedule.
- Dataproc: running Spark in production - cluster sizing, autoscaling policies, ephemeral vs long-running clusters, serverless Spark.
- Dataflow / Apache Beam: streaming and batch, Flex Templates, windowing, dynamic schema validation, DLQ patterns.
- Pub/Sub: topics, subscriptions, delivery guarantees, dead-lettering, and where streaming was the right choice over batch.
- Cloud Storage: data lake layout, storage classes, lifecycle policies, file formats (Parquet, Avro, ORC) and partition layout.
- IAM and operations: service accounts, least-privilege roles, workload identity, Cloud Logging/Monitoring, alerting and cost monitoring.
TERRAFORM
You must have written and maintained Terraform yourself, not only consumed modules written by others.
- Authoring reusable modules: variables, outputs, locals, for_each / count, dynamic blocks.
- Remote state in GCS, state locking, workspaces, and safe state operations (import, move, targeted apply, drift detection).
- Provisioning GCP resources with the google / google-beta providers: BigQuery datasets, GCS buckets, service accounts and IAM bindings, Composer environments, Dataproc clusters, Pub/Sub topics.
- Environment promotion and running Terraform through CI/CD.
PYTHON
- Production-quality Python: modules and packaging, dependency management, typing, structured logging, error handling, configuration management.
- Testing discipline: pytest, fixtures, mocking external services, unit-testing transformation logic.
- Fluency with data libraries (pandas and/or the Spark Python API) and GCP client libraries.
- Git branching workflow, code review and CI. Scripting-only experience is not sufficient.
PYSPARK
- Distributed batch (and ideally streaming) pipelines processing large production volumes, not sample datasets.
- DataFrame API, Spark SQL, window functions, UDFs and why to avoid them, joins at scale (broadcast vs shuffle).
- Performance tuning: partitioning and repartitioning, shuffle behaviour, caching and persistence, skew handling, executor and memory configuration, and reading the Spark UI to find a bottleneck.
- File format and layout decisions (Parquet, compression, partition pruning, small-file problems) and incremental, idempotent load patterns.
SQL
- Advanced SQL: complex joins, window and analytic functions, set operations, CTEs, aggregation over very large tables, semi-structured data.
- Execution plan analysis and query performance tuning - you should be able to explain why a query was slow and exactly what you changed.
- Dimensional modeling: star/snowflake schemas, fact and dimension design, slowly changing dimensions, surrogate keys, grain definition.
- Data quality and reconciliation: deduplication, late-arriving data, null and referential integrity handling, source-to-target validation.
ADDITIONAL REQUIREMENTS
- Hands-on experience setting up CI/CD pipelines for data engineering workflows.
- Ability to articulate technical decisions and trade-offs clearly in English and defend them under detailed questioning.
- Authorised to work in Canada and able to engage on a C2C basis (own incorporation or valid business entity).
PREFERRED
- GCP Professional Data Engineer certification (active or prior).
- Dataplex / Data Catalog - semantic aspects, business glossaries, metadata-driven AI grounding.
- Legacy data warehouse migrations (Netezza, Teradata, Informatica to BigQuery).
- dbt; Delta Lake / Iceberg / Hudi or Databricks exposure.
- Data quality and observability tooling (Great Expectations, Soda, Dataplex); governance tools such as Collibra.
- Exposure to Gemini / BigQuery Conversational Agents, LLM integrations or agentic workflows on enterprise data platforms.
TECH STACK
GCP | BigQuery | Cloud Storage | Cloud Composer (Airflow v2/v3) | Dataproc (PySpark) | Dataflow (Apache Beam) | Pub/Sub | Terraform | CI/CD | Git | Cloud Monitoring | Dataplex | Python | SQL | Bash
WHO THIS ROLE IS NOT FOR
To save everyone's time, we will not shortlist:
- Candidates with no production GCP experience, including strong AWS or Azure engineers looking to transition.
- Candidates whose GCP, Spark or Terraform exposure was limited to training, certification labs or personal projects.
- Support, monitoring or ETL-tool-only backgrounds (Informatica, SSIS, Talend) without hands-on coding in Python and PySpark.
- Profiles where the listed skills cannot be tied to a specific project, your specific contribution and the data scale you worked at.
INTERVIEW PROCESS
- Internal technical interview (Virdapt) - deep dive on GCP, PySpark, SQL and Terraform, with live problem-solving and questions about pipelines you personally built.
- Client technical interview 1 - hands-on assessment covering data engineering fundamentals, coding and SQL.
- Client technical interview 2 - architecture, design trade-offs, troubleshooting scenarios and delivery team fit.
Expect detailed technical questioning at every stage. Come prepared to discuss specific projects, the data scale involved, the decisions you made and what you would do differently.
HOW TO APPLY
Apply through LinkedIn or email your resume to akash.batham@virdapt.com with the subject line "Data Engineer - GCP (Canada, C2C)".
Please include:
- A short summary (3-5 lines) of one GCP data pipeline you personally built - services used, data volume and your specific role.
- Your years of hands-on experience with each of: GCP, Terraform, Python, PySpark, SQL.
- Rate expectation (CAD/hr, C2C), availability and current location within Canada.
- Confirmation of work authorisation and C2C entity.
Click on Apply to know more.
