Quantiphi
Website:
quantiphi.com
Company:
https://www.linkedin.com/company/quantiphi
Seniority: Associate
Industries: IT Services and IT Consulting
Job details:
Summary
We are looking for an experienced Senior Data Engineer to join our team at Quantiphi, a leader in AI-first engineering and innovative GenAI solutions. In this pivotal role, you will design and architect robust data pipelines that power advanced AI applications, specifically focusing on building high-performance ingestion engines for unstructured data, vector databases, and scalable real-time streaming architectures. You will bridge the gap between complex data engineering and modern AI, ensuring data quality, lineage, and observability as we deploy cutting-edge Agentic AI solutions.
Responsibilities
- Design and build batch ETL pipelines that ingest large unstructured document corpora - handling PDF, RTF, and JSON file formats, consuming pre-extracted text and metadata, applying document chunking strategies, and loading processed outputs into a vector store and analytical platforms.
- Design partitioning, parallelism, and throughput strategies to sustain high-volume ingestion.
- Build real-time streaming ingestion pipelines consuming document creation and update events, processing them through chunking and embedding generation.
- Build vector and indexing pipelines targeting vector store platforms (e.g., Vertex AI Vector Search, Pinecone, Weaviate, Milvus, pgvector)
- Manage vector store index operations: bulk load, incremental upsert, index refresh, and consistency validation post-ingestion.
- Implement metadata tagging on vector records to support filtered retrieval at query time.
- Implement end-to-end lineage tracking for vector records - linking each chunk back to its source document.
- Implement document-level change tracking to detect upstream updates, deletions, and re-ingestion events and trigger appropriate vector store operations (upsert, delete, re-index) without full corpus re-processing.
- Tag all vector records with pipeline run metadata - ingestion timestamp, pipeline version, chunking strategy version, embedding model version, to support retrieval quality debugging and model refresh traceability.
Must Have Skills
- Data Engineering - Experience building production-grade batch and streaming data pipelines supporting enterprise analytics and AI workloads.
- Python- Strong proficiency in Python for data processing, pipeline development, automation, and transformation frameworks.
- Streaming Technologies - Experience building real-time ingestion pipelines using Apache Kafka, GCP Pub/Sub, Azure Event Hubs, or equivalent event streaming platforms.
- Data Platforms - Experience working with BigQuery/Synapse, Databricks, Snowflake, or equivalent cloud-native analytical platforms.
- Data Modeling & SQL - Strong SQL skills and experience designing scalable analytical and operational data models.
- Vector databases - Familiarity with Pinecone, Vertex AI Vector Search, Weaviate, Milvus, pgvector, or similar retrieval platforms.
- Document processing - Familiarity with PDF, RTF, JSON, OCR pipelines, and metadata extraction workflows.
- Data Quality & Governance - Experience implementing validation frameworks, reconciliation, lineage, metadata management, monitoring, and pipeline observability.
- Cloud Platforms - Hands-on experience building cloud-native data solutions using GCS/ADLS, BigQuery/Synapse, Dataflow/ADF, Cloud Composer/Airflow, or equivalent services.
- Regulated Data - Experience working with regulated data environments where security, auditability, governance, and compliance are critical.
Click on Apply to know more.
