India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
Data Governance, Catalogue, Lineage and Data Quality Rules
Data governance and quality is the set of roles, rules and tooling that make analytical data findable, understood, traceable and fit for use: a business glossary, a catalogue with lineage, named stewards, and automated quality checks with scorecards. NTSPL implements these on Purview, Dataplex or SageMaker Catalog for organisations whose reports feed regulators, boards or incentive payouts.
Delivered on
In scope
Why Catalogue-First Governance Programmes Stall
Governance programmes usually start with a policy document and a catalogue licence, and stall within a year. The catalogue fills with 40,000 auto-scanned assets nobody curates, the glossary has 12 terms, and data quality is still discovered by the person who notices a branch reporting negative deposits. Meanwhile auditors and regulators ask for lineage from a submitted figure back to source, and the answer is a week of SQL archaeology.
NTSPL governs the data that matters first: the 50-150 critical data elements behind regulatory returns, board KPIs and incentive calculations. For each we name an owner and steward, write quality rules that run on every load, trace column-level lineage and publish a scorecard. Catalogue coverage grows outward from there, driven by usage rather than by scanning everything. Scorecards go to owners every month.
When Governance and Quality Become Urgent
Signals you need Data Governance & Quality
- Regulatory returns, board packs or incentive payouts have been restated twice or more in a year
- Auditors ask for lineage of submitted figures and it takes more than a week to answer
- Master data (customers, doctors, dealers, products) has duplicate rates above 3-5%
- A catalogue was bought but under 10% of assets have an owner or description
When to choose something else
- Your main concern is masking, consent and access to personal data in analytics: that is Data Security & Privacy.
- You need a full MDM platform with workflow and survivorship across operational systems: we scope that separately, as governance alone will not fix operational mastering.
Glossary, Lineage, Stewardship and Quality Controls
Critical data elements
Identification of the 50-150 elements behind regulatory, board and incentive outputs, each with owner, steward and definition.
Glossary and catalogue curation
Business glossary linked to physical columns, with curated descriptions, classifications and certification status.
Column-level lineage
Automated lineage from pipelines and SQL, supplemented by manual links for spreadsheet and file hops.
Quality rules and scorecards
Completeness, validity, uniqueness, consistency and timeliness rules per element, run on every load with thresholds.
Issue management
Rule breaches routed to stewards as tickets with root cause, fix owner and SLA, reviewed in a monthly data council.
How Data Governance & Quality Works, Step by Step
Critical-Data-Element Governance Loop
Critical elements are defined in the glossary, linked to catalogued columns and traced through lineage. Quality rules run on every pipeline load, publish scorecards and raise issues to stewards, whose fixes feed back into rules.
Select any component
Critical-Data-Element Governance Loop: click any of its 8 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
Governance Tooling Across AWS, Azure and Google Cloud
Glue Data Quality rules written in DQDL run inside Glue jobs and publish scores to the catalogue; business glossary and data products live in SageMaker Catalog.
Purview's Unified Catalog organises governance by business domain and data product, with data quality rules and scores attached to each.
Dataplex auto data quality runs rules directly on BigQuery tables and records results, while BigQuery jobs emit lineage automatically.
Scoping, Rule Design and Ownership Choices
How much to govern first
Start with critical data elements, not the whole estate. Pick the elements feeding regulatory returns, board KPIs and money flows such as incentives or commissions, typically 50-150. Cataloguing 40,000 assets without owners produces noise. Expand to the next tier only when first-tier elements hold above 98% quality pass rates for three months.
Rule severity and blocking
Classify each rule as blocking or warning. Blocking rules, such as null account numbers in a regulatory feed or totals failing ledger reconciliation, stop publication. Warning rules raise tickets but let data flow. Keep blocking rules under about 15% of the total; too many and pipelines stall daily and teams start bypassing them.
Who owns and who stewards
Owners are business heads accountable for the definition and fitness of an element; stewards are the people who investigate and fix issues, usually in operations or MIS. Do not make IT the owner. Each steward should carry no more than 20-30 elements, or issue SLAs will slip. Give stewards protected time.
Where rules live
Run rules inside the pipeline that produces the data, using dbt tests, Glue Data Quality, Dataplex scans or Purview rules, so failures stop bad data at source. Use the catalogue to display results, not to execute every check. Rules defined only in a governance tool tend to run late, after reports are already published.
Governance Rollout by Critical Data Element
Critical element discovery
Trace regulatory, board and incentive outputs back to elements; name owners and stewards.
Glossary and catalogue curation
Write and approve definitions, link to columns and certify core datasets.
Lineage capture
Enable automated lineage and add manual edges for file and spreadsheet steps.
Quality rules and scorecards
Implement rules in pipelines, set thresholds and publish the scorecard.
Issue workflow and data council
Launch steward ticketing, SLAs and the first monthly council.
What you receive
Quality and Stewardship Scorecard Targets
| KPI | Typical target |
|---|---|
| Quality pass rate on critical elements | Above 98% |
| Critical elements with traced lineage | 100% |
| Median time to resolve a quality issue | Under 5 working days |
| Restatements of regulatory or board figures | Zero per year |
Governance Anti-Patterns and Remedies
| Risk | How NTSPL prevents it |
|---|---|
| Catalogue fills with uncurated assets and loses credibility | Certified assets are flagged and searched first; uncurated scans are hidden from default views. |
| Owners are named but disengage | Owners present their scorecard at the monthly council, and definition changes need their approval. |
| Quality rules duplicate across tools and drift | Rules live in pipeline code in one library; catalogues only display results. |
Data governance tool first or data governance process first?
Process first, with the tool chosen early but deployed narrowly. A catalogue cannot decide who owns 'active customer' or what threshold makes a regulatory feed unfit. Programmes that buy and scan first end up with tens of thousands of uncurated assets and no stewards. Programmes that define owners and rules for critical elements first, then use Purview, Dataplex or SageMaker Catalog to publish lineage and scores, show results within one quarter. Choose the tool native to your primary data platform; switching later costs more than any feature gap.
Key facts: Data Governance & Quality
Data Governance & Quality in Practice
2 Data Governance & Quality scenarios, from Banking (private sector bank) and Pharmaceuticals (field force and sales operations), plus related platform case studies.
Private bank cuts regulatory return resubmissions from 14 a year to one
Pharma field force cleans its doctor master and stops paying incentives on duplicates
IT services firm brings 43 Azure subscriptions under one policy hierarchy
Agri-inputs company sees dealer secondary sales daily instead of 45 days late
Data Governance & Quality: Frequently Asked Questions
How long before governance shows measurable results?
For the first 50-150 critical elements, scorecards and lineage are usually live within 10-12 weeks, and quality pass rates rise noticeably within one or two reporting cycles. A first phase typically costs INR 30-60 lakh depending on the number of source systems.
Does RBI expect data governance for regulatory returns?
RBI expects banks to submit accurate, timely returns and has pushed automated data flow from source systems for regulatory reporting. Lineage from return line to source, documented ownership and quality checks give compliance and audit teams the evidence they need when submissions are questioned.
Which catalogue should we use?
Use the one native to your main data platform: Purview for Azure and Fabric, Dataplex Universal Catalog for BigQuery, SageMaker Catalog and Glue for AWS. Cross-cloud estates can still standardise on one, but lineage is richest for the platform it is built for.
Can you fix duplicates in our customer or dealer master?
We can within analytics: matching rules, survivorship and a golden record for reporting, with duplicate rates tracked on the scorecard. Fixing duplicates in operational systems needs process changes at data entry, which we specify and hand to system owners. Golden records are reviewed quarterly.
Who should own data quality: IT or the business?
The business owns definitions and fitness; IT and data engineering own the pipelines and rule execution. Stewards, usually in operations or MIS teams, investigate and fix issues. Making IT the owner is the most common reason quality programmes fail. Owners must sign off definition changes.
How many quality rules do we need?
Typically 3-6 rules per critical element covering completeness, validity, uniqueness, consistency and timeliness, so 300-600 rules for a first phase. More is not better: each rule needs a threshold, an owner and a reason someone will act on its failures.