Website:
smartbhujal.com
Job details:
Senior Data Engineer
Full-time · In-office · Tirupati, Andhra Pradesh
About us
Smart Bhujal was born from IIT Tirupati with a vision to make water decisions data-driven. We build AI-enabled decision-support systems for groundwater, floods, irrigation and water networks — automated, compliant data platforms for research institutes and development organisations in India and worldwide.
The role
Most of the data that matters to water decisions is public but unusable — scattered across government portals, agency websites and PDFs, in inconsistent formats and several languages. We are hiring a Senior Data Engineer to build the crawler and AI-assisted extraction pipeline that turns it into structured, provenance-tracked data. It is the most technically demanding work on the team, and everything else we build sits on top of it.
Location & engagement
Employment type: Full-time, permanent
Location: In-office · Tirupati, Andhra Pradesh
Level: Senior · 5+ years
Reports to: Technical Lead
Start: Immediate
Whom we're looking for
• B.Tech with 5+ years of experience, hands-on in data crawling/scraping and extraction. General backend experience does not substitute.
Key responsibilities
• Design and build a discovery and crawling pipeline over authoritative seed lists.
• Implement a compliance engine — robots.txt and terms-of-use handling, conservative rate limiting, caching, identifiable user-agent, graceful back-off, off-peak scheduling and proxy rotation — so crawling is polite and low-impact.
• Build LLM-assisted extraction from HTML and PDF into structured fields, with multilingual handling, confidence scoring and a low-confidence human-review queue (uncertain data is flagged, never silently written).
• Classify data sources as official vs unofficial and capture full source metadata and provenance for every value.
• Implement staged document fetching with metadata extraction and version history.
• Work to disciplined version-control, CI/CD, branching and code-review standards throughout.
Required skills & experience
• Strong Python with production crawling/scraping frameworks (Scrapy, Playwright / Puppeteer, Requests), used at scale on real sources.
• Hands-on anti-bot and proxy handling, distributed crawl scheduling, queue management and retry / back-off — with a demonstrated respect for robots.txt, terms of use and rate limits.
• LLM-based extraction from unstructured HTML and PDF (OpenAI / Claude APIs, LangChain / LlamaIndex or equivalent) and PDF parsing (pdfplumber, Unstructured, OCR).
• Experience designing confidence scoring and human-in-the-loop review workflows.
• PostgreSQL and field-level provenance / ETL modelling.
• Git, code review, Docker and CI/CD.
Preferred (nice to have)
• Experience with public-sector, development-sector or open-data sources.
• Multilingual / non-English source handling.
• Geospatial data (PostGIS / GeoJSON).
What we offer
• A high-impact, permanent role owning a core engineering system end to end.
• A senior, focused team and a clear, well-scoped mandate.
• Competitive salary, commensurate with experience.
Shortlisted candidates will complete a short technical conversation covering compliant crawling and LLM extraction.
Smart Bhujal is an equal-opportunity employer. Compensation is competitive and commensurate with experience.
#Hiring #Python #DataEngineering #WebScraping #LLM #IITTirupati #WaterTech
Click on Apply to know more.