Title:  Senior Data Engineer

Job Summary

We are looking for a passionate Senior Data Engineer to join our team at Compass Data & AI. You will design, develop, and maintain cutting-edge data products and AI solutions, focusing on making difficult-to-access data sources available through web scraping and document scanning. Collaborating with Data Scientists, Analytics Engineers, Front-End Engineers, ML Engineers, and AI Engineers, you will build cloud-native backend systems and infrastructure for new AI-powered applications. Your work will lay the foundations for a new product delivering Data Analytics and AI capabilities.

Now, if you were to come on board as our Senior Data Engineer, we’d ask you to do the following for us:
Design, build, and maintain scalable, Playwright-based web scraping systems to extract data from dynamic and complex web sources
Stay current with best practices in web scraping, anti-bot mitigation, headless browser automation, and AI infrastructure
Collaborate with cross-functional teams to define, design, and ship new features;
Architect efficient, reusable backends for scraping and AI workflows in the cloud
Write clean, well-documented, and maintainable code following best practices;
Conduct code reviews, provide constructive feedback and mentor junior team members;
Staying up-to-date with emerging AI technologies and industry trends to drive continuous improvement
Ensure data quality, resilience, and compliance across scraping processes
Work in a dynamic, collaborative, progressive, and high-performing team
Interactions with multiple cross-functional teams, exposure to many components of modern data technology

Think you have what it takes to be our Senior Data Engineer? We’re committed to hiring the best talent for the role. Here’s how we’ll know you’ll be successful in the role:
Bachelor's degree in a STEM field or equivalent experience
Strong experience with Large Language Models (LLMs)
2+ years of hands-on experience in developing AI and data products
Strong knowledge of Python and scraping frameworks/libraries (e.g., Playwright, BeautifulSoup, Requests)
Strong cloud engineering skills with experience taking AI ideas to production
Familiarity with proxy rotation, CAPTCHA solving, and session management in automated scraping
Familiarity with RESTful APIs
Knowledge of version control systems such as Git and agile development methodologies
Excellent problem-solving skills, attention to detail, and ability to work independently or within a team
Excellent written and verbal communication skills, with the ability to convey complex concepts to non-technical users