Full Stack Data Discovery Engineer

Responsibilities

• Own the full stack: Design, build, and optimize scalable data pipelines to discover OSINT and software usage across a wide public ecosystem.
• Pipeline development: Develop APIs, microservices, crawlers, and document fingerprinting to gather data securely and efficiently. Implement backoff/caching, data normalization, and persist to SQL/NoSQL indexes.
• Data Discovery: Conduct systematic searches across the web, public databases, developer ecosystems, and other platforms to identify potential external data repositories relevant to organizational objectives.
• Metadata and Attribution Analysis: Programmatically uncover and analyze metadata associated with identified data sources to understand data structure, content, quality, and potential use cases.
• Signals & scoring: Develop heuristics/ML‑lite ranking to identify relevant artifacts, deduplicate, and assign confidence scores.
• Data Governance: Ensure data quality, security, compliance, and governance.
• Productize discovery: Build internal tools that let non‑engineers run searches, review candidates, and export leads—fast and safely.
• Documentation and Reporting: Document data structures, origins (data lineage), and quality issues. Create clear, concise reports and presentations to communicate findings and recommendations to technical and non-technical stakeholders.
• Collaboration: Work closely with data stewards, data architects, and internal business units to define data requirements and facilitate the integration of new data sources.
• Innovation and Scale: Continuously explore new data sources, improve attribution logic, and propose ML-based enhancements to finding and classifying data.

Requirements

Education

• Bachelor's degree in Computer Science, Engineering, Library Science, Information Systems, Data Management, or a related field (Master's degree preferred).

Experience

• 1-5 years of proven experience as a full-stack developer and data engineer.
• Creating the initial inception and idea of the project.

Technical Skills

• Back-end: Python, SQL, Java, and Node.js.
• Front-end: Modern JS/TS + React, component libraries, auth patterns, state management.
• Data & search: Schema design, deduplication/near‑duplication logic, Elasticsearch/OpenSearch; building usable search/triage UIs.
• Acquisition: Scrapy/Playwright/Puppeteer; API design with rate‑limit/backoff; ethical crawling.
• Experience with cloud-native architecture and containerization.
• Familiarity with metadata standards (e.g., Dublin Core, XML) and data management tools.

Assets

• Knowledge of data visualization tools (e.g., Power BI, Tableau) to present findings.
• Experience building internal platforms/tools used by end-users or GTM teams.

Soft Skills

• Exceptional attention to detail and strong analytical thinking skills.
• Excellent written and verbal communication skills, with the ability to translate technical findings into business insights.
• Strong problem-solving aptitude and the ability to work independently and collaboratively in a fast-paced environment.

Benefits

• Competitive salary commensurate with experience & qualifications.
• A comprehensive extended benefits package including health, dental, and vision for you and your family, with company-paid offerings.
• 401K savings program with company match.
• Generous paid time off (PTO) is offered to support the ability to rest and recharge.
• A great team environment and resources, supporting you to do the best work of your life and providing unlimited career growth potential.
• Highly autonomous and entrepreneurial environment.
• Annual recurring WFH allowance for you to purchase items you need for your home office.
• Ongoing support for learning development so you can master your craft.
• Work with the hardware you're most comfortable with (Windows or Mac).
• Diverse and inclusive workplace where we all learn from each other.

Key skills/competency

• Full Stack Development
• Data Engineering
• Data Discovery
• API Development
• Web Crawling
• Data Governance
• SQL/NoSQL
• React.js
• Elasticsearch
• Cloud-native Architecture