A reusable PySpark data-quality accelerator for schema extraction, validation, threshold monitoring, and alerting across lakehouse pipelines.
Impact
manual data-quality checks reduced through reusable validation patterns
Hiring signal
Data engineering scale, reusable framework design, observability, config-driven validation, and production reliability.
My role
Developer · PySpark logic, Fabric implementation, schema extraction, rule generation, validation framework, and monitoring patterns
Status
Source or architecture details are available where applicable.
The problem
Data-quality checks were manual, repetitive, and difficult to scale across large tables and multiple pipelines. Teams required a reusable framework for schema validation, rule enforcement, anomaly detection, reconciliation, and alerting without implementing bespoke checks for every dataset.
My approach
The framework uses Python and PySpark on Microsoft Fabric to inspect schemas dynamically, resolve configuration-driven rules, execute distributed validations, persist results, and enforce quality gates across Bronze, Silver, and Gold layers.
Architecture judgment
Data-quality management must scale with the lakehouse. Schema-derived checks expand baseline coverage as new tables are introduced, while configuration-driven rules keep thresholds, ownership, and policy changes explicit and reviewable.
Architecture
The complete execution path, service boundaries, data stores, security controls, validation stages, and operational telemetry.
Implementation
Technology
Interactive trace
Run the trace to see the main workflow stages.