Clean
The Clean module standardizes raw data at scale so downstream matching compares like with like.
Why Data Cleaning Matters
Healthcare data arrives in inconsistent formats across sources. That fragments patient records, causes missed matches, skews analytics, and creates compliance risk.
What You Can Do
Configure rules for whitespace, case, phone formats, and more.
Disable a rule for the next run without losing its configuration.
Apply several operations in sequence.
Monitor jobs in real time with logs.
Pick multiple source tables and one destination schema. Each table is written there as <table>_cleaned, with its own saved rules applied.
See the columns each rule touched and how many rows it changed.
Cleaning Operations
Trim whitespace, convert case, remove special characters
Strip formatting, handle country codes, apply a consistent pattern
Read many input formats and write one standard format
Title-case names and collapse stray whitespace
Title-case addresses and expand street-type and directional abbreviations
Replace values, fill nulls, and apply custom regular expressions
Key Capabilities
Rule-Based Configuration
Define rules visually without code: select columns, choose operations, set parameters.
Scalable Processing
Jobs run natively on Databricks (Spark), Snowflake (Snowpark), or Microsoft Fabric (OneLake), without moving data.
Rule Conflict Detection
Conflicting case transformations or duplicate rules on a column are flagged while you configure them, not at run time.
Run History and Recovery
A live run window shows progress and row counts and keeps past runs. Failed or lost runs are reconciled automatically so the stage never sticks in "running".
Column Profiling
After cleaning, each column is profiled for nulls, unique values, and data types.
Business Impact
Standardized data improves matching accuracy.
Automate repetitive cleanup.
Clean data yields reliable reports.
Who Benefits
- Data Stewards: Maintain quality standards
- Clinical Teams: Access accurate patient information
- Revenue Cycle: Reduce data-driven claim denials
- Compliance Officers: Meet data accuracy requirements