Data Quality Engineer (Databricks Unity Catalog)

Position: Senior Data Quality Engineer

Location: Remote or Hybrid

Experience: 6+ years

ROLE OVERVIEW

We are looking for a senior Data Quality Engineer to own granular, proactive data quality monitoring across the entire project data estate on Databricks. This is a dedicated data quality engineering role distinct from data operations focused on building the frameworks and automated checks that catch anomalies before the business does.

You will own the data quality strategy for our Databricks environment, working with data teams to implement validation rules, establish governance policies, and ensure compliance across the platform. Given the scope of this mandate, this is a demanding, high-visibility role that requires someone who can operate independently at a senior level.

KEY RESPONSIBILITIES

Data Quality Rule Development

• Design and implement data quality rules in Databricks Unity Catalog
• Develop validation logic for data pipelines and ingestion processes
• Create automated quality checks that catch issues before they impact downstream systems
• Integrate quality checks into CI/CD pipelines for continuous validation
• Document rules and make them maintainable for other teams

Cross-Workspace Governance

• Build governance frameworks that work across multiple Databricks workspaces
• Establish data policies and access controls
• Ensure consistent data management practices across teams
• Implement data lineage and cataloging standards

Data Testing & Validation

• Define data quality SLAs and configure alerting thresholds so issues trigger before they reach downstream consumers
• Build test suites for data quality
• Validate data transformations and pipeline outputs
• Run regression testing when governance rules change
• Conduct root-cause analysis on quality failures and implement preventive measures
• Create dashboards/reports to track data quality metrics

Data Lifecycle Management

• Define and enforce data retention policies
• Manage data classification (sensitive, PII, public)
• Establish data archival and cleanup processes
• Monitor compliance with data policies

Collaboration & Support

• Work with data engineering teams to embed quality rules in pipelines
• Support data teams in troubleshooting quality failures
• Document standards and best practices
• Mentor teams on data governance principles

REQUIRED EXPERIENCE & SKILLS

Must Have:

• 6+ years of experience in data quality engineering or data governance, with a track record of owning quality frameworks independently
• Strong SQL can write complex queries, optimize performance, handle edge cases
• Deep hands-on experience with Databricks (Delta Lake, Spark, Notebooks)
• Experience developing and maintaining data quality rules/frameworks
• Understanding of data governance concepts (cataloging, lineage, policies, compliance)
• Ability to work across multiple workspaces and manage workspace-level governance
• Strong documentation and communication skills
• Experience with data validation and testing frameworks
• Comfortable operating independently in a fast-paced, high-accountability environment this is a demanding mandate that calls for senior-level judgment, not just execution against a spec

Technical Stack:

• Databricks Unity Catalog, Delta Live Tables, and Databricks Workflows (required this is the core of the role)
• SQL (advanced)
• Python or Scala (for custom quality checks)
• Data governance tools (any modern platform)

NICE TO HAVE

• Experience with Collibra Platform
• Experience with data mesh or federated governance models
• Familiarity with data quality tools (Great Expectations, Soda, dbt tests, etc.)
• Experience with data observability platforms (Monte Carlo, Datafold, Bigeye)
• Cloud platform experience (AWS, Azure, GCP)
• CI/CD pipeline tooling (Azure DevOps, GitHub Actions)
• Experience with streaming data architectures (Kafka, Structured Streaming)
• Experience with metadata management
• Understanding of data privacy regulations (GDPR, CCPA, etc.)