问题描述
鉴于我有一个数据仓库,其中的数据表是由dbt从各种来源创建的,因此我想测量一个概念,例如'dbt table coverage',我将其定义为:
dtc = count(tables and views that exist) / count(non ephemeral models and sources)
这对于保持质量/完整性感非常有用,尤其是在过渡项目期间。是否有像这样的dbt命令:
dbt report table-coverage --schemas=['reporting','example']
>>> 96% coverage,48/50 tables in the schemas provided are captured in dbt.
如果没有,我们如何将其添加到项目中?!
解决方法
为此,我可能会创建一个查询information_schema的模型(视图),并对{sourceTableName}
到stg_{sourceTableName}
的一对一映射做一些假设(假设这意味着您需要覆盖)。
另外,我将研究使用graph.sources.values()
JINJA函数来遍历项目中所有已记录的源代码,然后将其与{target.schema}
中的每个模型进行比较
https://docs.getdbt.com/reference/dbt-jinja-functions/graph#accessing-sources
如果您要比较source.schema.yml
和source.information_schema
的存在。我将改变考虑将图形中的每个项目与源数据库中的information_schema中的项目总数进行映射的方法。
在这里有一些想法,因为这对我目前的情况也很有趣:
-
dbt不会提供查询输出或将结果返回到命令行。 (据我所知!)因此,如果这1个天生不支持此功能。即
dbt report
或dbt query
尚不存在。如果需要的话,建议您在此处构建功能请求: https://github.com/fishtown-analytics/dbt/issues -
如果您可以在dbt中创建模型,然后仅通过选择的客户端执行该模型,那么让我们来看看。 (我使用的是postgres,因此请相应地进行转换)
WITH schema_map as (select schemaname as schema,tablename as name,'Table' as Type,CASE WHEN schemaname like '%dbt%' THEN 1 ELSE 0 END as dbt_created from pg_tables WHERE NOT schemaname = ANY('{information_schema,pg_catalog}') UNION select schemaname as schema,viewname as name,'View' as Type,CASE WHEN schemaname like '%dbt%' THEN 1 ELSE 0 END as dbt_created from pg_views WHERE NOT schemaname = ANY('{information_schema,pg_catalog}') ) SELECT count(name) as total_tables_and_views,sum(dbt_created) as dbt_created,to_char((sum(dbt_created)::dec/count(name)::dec)*100,'999D99%') as dbt_coverage FROM schema_map
给出结果:
total_tables_and_views | dbt_created | dbt_coverage
391 |292 | 74.68%
,
只需回馈社区,并感谢Jordan和Gscott的启发。我为SQL Server / Synapse执行的解决方案是:
- 每天执行一次INFORMATION_SCHEMA.TABLES和dbt图中的模型计数作为一个表。
- 建立在1上的增量表,用于选择感兴趣的模式并进行聚合。在下面的例子中,我过滤了阶段和测试。
DbtModelCounts:
{% set models = [] -%}
{% if execute %}
{% for node in graph.nodes.values()
| selectattr("resource_type","equalto","model")
%}
{%- do models.append(node.name) -%}
{% endfor %}
{% endif %}
with tables AS
(
SELECT table_catalog [db],table_schema [schema_name],table_name [name],table_type [type]
FROM INFORMATION_SCHEMA.TABLES
),dbt_tables AS
(
SELECT *
FROM tables
WHERE name in (
{%- for model in models %}
('{{ model}}')
{% if not loop.last %},{% endif %}
{% endfor %}
)
)
SELECT
tables.db,tables.schema_name,tables.type,COUNT(tables.name) ModelCount,COUNT(dbt_tables.name) DbtModelCount
FROM tables
LEFT JOIN dbt_tables ON
tables.name=dbt_tables.name AND
tables.schema_name = dbt_tables.schema_name AND
tables.db = dbt_tables.db AND
tables.type = dbt_tables.type
GROUP BY
tables.db,tables.type
Dbt承保范围:
{{
config(
materialized='incremental',unique_key='DateCreated'
)
}}
SELECT
CAST(GETDATE() AS DATE) AS DateCreated,GETDATE() AS DateTimeCreatedUTC,SUM(DbtModelCount) AS DbtModelCount,SUM(ModelCount) AS TotalModels,SUM(DbtModelCount)*100.0/SUM(ModelCount) as DbtCoveragePercentage
FROM {{ref('DbtModelCounts')}}
WHERE schema_name NOT LIKE 'testing%' AND schema_name NOT LIKE 'staging%'
要做的是,为已定义的源添加逻辑,以计算映射到我的登台表或原始模式表的源的百分比。