Flag job

Report

Web Scraping Specialist

Min Experience

4 years

Location

India, remote

JobType

full-time

About the job

Info This job is sourced from a job board

About the role

We are seeking a Web Scraping Specialist [Official, Internal Title: Data Solutions Engineer] to join our growing Data Solutions team. Reporting directly to the Data Solutions Engineering Manager, you will play a pivotal role in designing, refactoring, and maintaining the web scrapers that power critical reports across our organization. Your contributions will ensure our data ingestion processes are resilient, efficient, and scalable, directly supporting multiple business units and products. As Our Data Solutions Engineer You Will: Refactor and Maintain Web Scrapers Overhaul existing scraping scripts to improve reliability, maintainability, and efficiency. Implement best coding practices (clean code, modular architecture, code reviews, etc.) to ensure quality and sustainability. Implement Advanced Scraping Techniques Utilize sophisticated fingerprinting methods (cookies, headers, user-agent rotation, proxies) to avoid detection and blocking. Handle dynamic content, navigate complex DOM structures, and manage session/cookie lifecycles effectively. Collaborate with Cross-Functional Teams Work closely with analysts and other stakeholders to gather requirements, align on targets, and ensure data quality. Support internal users of our web scraping tooling by providing troubleshooting, documentation, and best practices to ensure efficient data usage for critical reporting. Monitor and Troubleshoot Develop robust monitoring solutions, alerting frameworks to quickly identify and address failures. Continuously evaluate scraper performance, proactively diagnosing bottlenecks and scaling issues. Drive Continuous Improvement Propose new tooling, methodologies, and technologies to enhance our scraping capabilities and processes. Stay up to date with industry trends, evolving bot-detection tactics, and novel approaches to web data extraction.

About the company

YipitData is the leading market research and analytics firm for the disruptive economy and recently raised up to $475M from The Carlyle Group at a valuation over $1B. We analyze billions of alternative data points every day to provide accurate, detailed insights on ridesharing, e-commerce marketplaces, payments, and more. Our on-demand insights team uses proprietary technology to identify, license, clean, and analyze the data many of the world's largest investment funds and corporations depend on. For three years and counting, we have been recognized as one of Inc's Best Workplaces. We are a fast-growing technology company backed by The Carlyle Group and Norwest Venture Partners. Our offices are located in NYC, Austin, Miami, Denver, Mountain View, Seattle, Hong Kong, Shanghai, Beijing, Guangzhou, and Singapore. We cultivate a people-centric culture focused on mastery, ownership, and transparency.

Skills

web scraping
selenium
playwright
puppeteer
http
restful apis
html parsing
browser rendering
tls/ssl
fingerprinting
evasion strategies
browser fingerprint spoofing
request signature manipulation
cookies
headers
session states
proxy rotations
residential proxies
data center proxies
logging
metrics
alerting