How to set an automated check to improve your dbt documentation<\/h4>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AipPU8SjbSbqth6gsejDBGg.jpeg?ssl=1\")
Image by the author (generated with\u00a0chatgpt)<\/figcaption><\/figure>\nIn large dbt projects, maintaining consistent and up-to-date documentation can be a challenge. Although dbt\u2019s {{ doc() }} function allows you to store and reuse descriptions for the columns of your models, ensuring its usage remains quite a manual process and prone to mistakes, and therefore can easily lead to incomplete or outdated documentation. In this article, we\u2019ll explore how to automate the validation of your dbt documentation using a custom Continuous Integration (CI) workflow in GitHub\u00a0Actions.<\/p>\n
I already wrote about the doc feature in dbt and how it helps create consistent and accurate documentation across the entire dbt project (see this<\/a>). In short, you can store the description of the most common\/important columns used in the data models in your project by adding them in the docs.md<\/em> files, which live in the docs folder of your dbt\u00a0project.<\/p>\nA very simple example of a orders.md<\/em> file that contains the description for the most common customer-related column\u00a0names:<\/p>\n# Fields description
## order_id
{% docs orders__order_id %}
Unique alphanumeric identifier of the order, used to join all order dimension tables
{% enddocs %}
## order_country
{% docs orders__order_country %}
Country where the order was placed. Format is country ISO 3166 code.
{% enddocs %}
## order_value
{% docs orders__value %}
Total value of the order in local currency.
{% enddocs %}
## order_date
{% docs orders__date %}
Date of the order in local timezone
{% enddocs %}<\/pre>\nAnd its usage in the\u00a0.yml<\/em> file of a\u00a0model:<\/p>\n columns:
- name: order_id
description: '{{ doc(\"orders__order_id\") }}'<\/pre>\nWhen the dbt docs are generated the description of order_id will be always the same, as long as the doc<\/em> function is used in the yml file of the model. The benefit of having this centralized documentation is clear and undeniable.<\/p>\nThe challenge<\/h4>\n
However, especially with large projects and frequent changes (new models, or changes to existing ones), it is likely that the repository\u2019s contributors will either forget to use the doc<\/em> function, or they are not aware that a specific column has been added to the docs<\/em> folder. This has two consequences:<\/p>\n\n- someone must catch this during PR review and request a change\u200a\u2014\u200aassuming there\u2019s at least one reviewer who either knows all the documented columns by heart or always checks the docs folder\u00a0manually<\/li>\n
- if it\u2019s easy to go unnoticed and relies on individuals, this setup defeats the purpose of having a centralized documentation.<\/li>\n<\/ul>\n
The solution<\/h4>\n
The simple answer to this problem is a CI (continuous integration) check, that combines a GitHub workflow with a python script. This check fails\u00a0if:<\/p>\n
\n- the changes in the PR are affecting a\u00a0.yml file that contains a column name present in the docs, but the doc<\/em> function is not used for that\u00a0column<\/li>\n
- the changes in the PR are affecting a\u00a0.yml file that contains a column name present in the docs, but that column has no description at\u00a0all<\/li>\n<\/ul>\n
Let\u2019s have a closer look at the necessary code and files to run this check, and to a couple of examples. As previously mentioned, there are two things to consider: a (1)<\/em><\/strong>\u00a0.yml file for the workflow and a (2)<\/em><\/strong> python file for the actual validation check.<\/p>\n(1)<\/em><\/strong> This is how the validation_docs<\/em> file looks like. It is placed in the github\/workflows<\/em> folder.<\/p>\nname: Validate Documentation
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
validate_docs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions\/checkout@v3
with:
fetch-depth: 0
- name: Install PyYAML
run: pip install pyyaml
- name: Run validation script
run: python validate_docs.py<\/pre>\nThe workflow will run whenever a pull request is open or re-open, and every time that a new commit is pushed to the remote branch. Then there are basically 3 steps: retrieving the repository\u2019s files for the current pull request, install the dependencies, and run the validation script.<\/p>\n
(2)<\/em><\/strong>. Then the validate_docs.py <\/em>script, placed in the root folder of your dbt project repository, that looks like\u00a0this<\/p>\nimport os
import sys
import yaml
import re
from glob import glob
from pathlib import Path
import subprocess
def get_changed_files():
diff_command = ['git', 'diff', '--name-only', 'origin\/main...']
result = subprocess.run(diff_command, capture_output=True, text=True)
changed_files = result.stdout.strip().split('n')
return changed_files
def extract_doc_names():
doc_names = set()
md_files = glob('docs\/**\/*.md', recursive=True)
doc_pattern = re.compile(r'{%s*docss+([^s%]+)s*%}')
for md_file in md_files:
with open(md_file, 'r') as f:
content = f.read()
matches = doc_pattern.findall(content)
doc_names.update(matches)
print(f\"Extracted doc names: {doc_names}\")
return doc_names
def parse_yaml_file(yaml_path):
with open(yaml_path, 'r') as f:
try:
return list(yaml.safe_load_all(f))
except yaml.YAMLError as exc:
print(f\"Error parsing YAML file {yaml_path}: {exc}\")
return []
def validate_columns(columns, doc_names, errors, model_name):
for column in columns:
column_name = column.get('name')
description = column.get('description', '')
print(f\"Validating column '{column_name}' in model '{model_name}'\")
print(f\"Description: '{description}'\")
doc_usage = re.findall(r'{{s*doc([\"']([^\"']+)[\"'])s*}}', description)
print(f\"Doc usage found: {doc_usage}\")
if doc_usage:
for doc_name in doc_usage:
if doc_name not in doc_names:
errors.append(
f\"Column '{column_name}' in model '{model_name}' references undefined doc '{doc_name}'.\"
)
else:
matching_docs = [dn for dn in doc_names if dn.endswith(f\"__{column_name}\")]
if matching_docs:
suggested_doc = matching_docs[0]
errors.append(
f\"Column '{column_name}' in model '{model_name}' should use '{{{{ doc(\"{suggested_doc}\") }}}}' in its description.\"
)
else:
print(f\"No matching doc found for column '{column_name}'\")
def main():
changed_files = get_changed_files()
yaml_files = [f for f in changed_files if f.endswith('.yml') or f.endswith('.yaml')]
doc_names = extract_doc_names()
errors = []
for yaml_file in yaml_files:
if not os.path.exists(yaml_file):
continue
yaml_content = parse_yaml_file(yaml_file)
for item in yaml_content:
if not isinstance(item, dict):
continue
models = item.get('models') or item.get('sources')
if not models:
continue
for model in models:
model_name = model.get('name')
columns = model.get('columns', [])
validate_columns(columns, doc_names, errors, model_name)
if errors:
print(\"Documentation validation failed with the following errors:\")
for error in errors:
print(f\"- {error}\")
sys.exit(1)
else:
print(\"All documentation validations passed.\")
if __name__ == \"__main__\":
main()<\/pre>\nLet\u2019s summarise the steps in the\u00a0script:<\/p>\n
\n- it lists all files that have been changed in the pull request compared to the origin\u00a0branch.<\/li>\n
- it looks through all markdown (.md<\/em>) files within the docs<\/em> folder (including subdirectories) and it searches for special documentation block patterns using a regex. Each time it finds such a pattern, it extracts the doc_name part and adds it to a set of doc\u00a0names.<\/li>\n
- for each changed\u00a0.yml file, the script opens and parses it using yaml.safe_load_all. This converts the\u00a0.yml content into Python dictionaries (or lists) for easy analysis.<\/li>\n
- \nvalidate_columns: <\/em>for each columns defined in the\u00a0.yml files, it checks the description field to see if it includes a {{ doc() }}<\/em> reference. If references are found, it verifies that the referenced doc name actually exists in the set of doc names extracted earlier. If not, it reports an error. If no doc references are found, it attempts to see if there is a doc block that matches this column\u2019s name. Note that here we are using a naming convention like doc_block__column_name<\/em>. If such a block exists, it suggests that the column description should reference this doc.
Any problems (missing doc references, non-existent referenced docs) are recorded as\u00a0errors.<\/li>\n<\/ol>\nExamples<\/h4>\n
Now, let\u2019s have a look at the CI in action. Given the orders.md<\/em> file shared at the beginning of the article, we now push to remote this commit that contains the ref_country_orders.yml<\/em> file:<\/p>\nversion: 2
models:
- name: ref_country_orders
description: >
This model filters orders from the staging orders table to include only those with an order date on or after January 1, 2020.
It includes information such as the order ID, order country, order value, and order date.
columns:
- name: order_id
description: '{{ doc(\"orders__order_id\") }}'
- name: order_country
description: The country where the order was placed.
- name: order_value
description: The value of the order.
- name: order_address
description: The address where the order was placed.
- name: order_date<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2ArqAwqjoIA5tGoctSD46JWA.png?ssl=1\")
<\/figure>\nThe CI has failed. Clicking on the Details will take us to the log of the CI, where we see\u00a0this:<\/p>\n
Validating column 'order_id' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_id\") }}'
Doc usage found: ['orders__order_id']
Validating column 'order_country' in model 'ref_country_orders'
Description: 'The country where the order was placed.'
Doc usage found: []
Validating column 'order_value' in model 'ref_country_orders'
Description: 'The value of the order.'
Doc usage found: []
Validating column 'order_address' in model 'ref_country_orders'
Description: 'The address where the order was placed.'
Doc usage found: []
No matching doc found for column 'order_address'
Validating column 'order_date' in model 'ref_country_orders'
Description: ''
Doc usage found: []<\/pre>\nLet\u2019s analyze the log:
– for the order_id<\/em><\/strong> column it found the doc usage in its description.
– the order_address<\/em><\/strong> column isn\u2019t found in the docs file, so it returns a No matching doc found for column \u2018order_address\u2019
<\/em>– for the order_value<\/em><\/strong> and order_country<\/em><\/strong>, it knows that they are listed in the docs but the doc usage is empty. Same for the order_date<\/em><\/strong>, and note that for this one we didn\u2019t even add a description line<\/p>\nAll good so far. But let\u2019s keep looking at the\u00a0log:<\/p>\n
Documentation validation failed with the following errors:
- Column 'order_country' in model 'ref_country_orders' should use '{{ doc(\"orders__order_country\") }}' in its description.
- Column 'order_value' in model 'ref_country_orders' should use '{{ doc(\"orders__order_value\") }}' in its description.
- Column 'order_date' in model 'ref_country_orders' should use '{{ doc(\"orders__order_date\") }}' in its description.
Error: Process completed with exit code 1.<\/pre>\nSince order_country<\/em><\/strong>, order_value<\/em><\/strong>, order_date<\/em><\/strong> are in the docs file, but the doc function isn\u2019t used, the CI raise an error. And it suggests the actual value to add in the description, which makes it extremely easy for the PR author to copy-paste the correct description value from the CI log and add it into the\u00a0.yml<\/em>\u00a0file.<\/p>\nAfter pushing the new changes the CI check was succesfull and the log now looks like\u00a0this:<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AvTxCA0Ymn17JjQPTzk0gJw.png?ssl=1\")
<\/figure>\nValidating column 'order_id' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_id\") }}'
Doc usage found: ['orders__order_id']
Validating column 'order_country' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_country\") }}'
Doc usage found: ['orders__order_country']
Validating column 'order_value' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_value\") }}'
Doc usage found: ['orders__order_value']
Validating column 'order_address' in model 'ref_country_orders'
Description: 'The address where the order was placed.'
Doc usage found: []
No matching doc found for column 'order_address'
Validating column 'order_date' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_date\") }}'
Doc usage found: ['orders__order_date']
All documentation validations passed.<\/pre>\nFor the order_address<\/em><\/strong> column, the log shows that no matching doc was found. However, that\u2019s fine and does not cause the CI to fail, since adding that column to the docs file is not our intention for this demonstration. Meanwhile, the rest of the columns are listed in the docs and are correctly using the {{ doc() }}<\/em>\u00a0function<\/p>\nIn conclusion, by integrating this validation CI into your dbt repository, you can confidently maintain a centralised source of truth for column descriptions across your entire project and make the best out of the {{ doc() }} feature in dbt. This setup can save valuable review time, reduces human error, and upholds a higher standard of documentation quality. As your project grows, you\u2019ll find that this automated approach to documentation management is both scalable and maintainable, ultimately enabling your team to focus on analytics rather than debugging inconsistent docs.<\/p>\n
![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e2c3e8428e7\")
<\/p>\n
\nOptimize the dbt Doc Function with a CI<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
In large dbt projects, maintaining consistent and up-to-date documentation can be a challenge. Although dbt\u2019s {{ doc() }} function allows you to store and reuse descriptions for the columns of your models, ensuring its usage remains quite a manual process and prone to mistakes, and therefore can easily lead to incomplete or outdated documentation. In this article, we\u2019ll explore how to automate the validation of your dbt documentation using a custom Continuous Integration (CI) workflow in GitHub\u00a0Actions.<\/p>\n
I already wrote about the doc feature in dbt and how it helps create consistent and accurate documentation across the entire dbt project (see this<\/a>). In short, you can store the description of the most common\/important columns used in the data models in your project by adding them in the docs.md<\/em> files, which live in the docs folder of your dbt\u00a0project.<\/p>\n A very simple example of a orders.md<\/em> file that contains the description for the most common customer-related column\u00a0names:<\/p>\n And its usage in the\u00a0.yml<\/em> file of a\u00a0model:<\/p>\n When the dbt docs are generated the description of order_id will be always the same, as long as the doc<\/em> function is used in the yml file of the model. The benefit of having this centralized documentation is clear and undeniable.<\/p>\n However, especially with large projects and frequent changes (new models, or changes to existing ones), it is likely that the repository\u2019s contributors will either forget to use the doc<\/em> function, or they are not aware that a specific column has been added to the docs<\/em> folder. This has two consequences:<\/p>\n The simple answer to this problem is a CI (continuous integration) check, that combines a GitHub workflow with a python script. This check fails\u00a0if:<\/p>\n Let\u2019s have a closer look at the necessary code and files to run this check, and to a couple of examples. As previously mentioned, there are two things to consider: a (1)<\/em><\/strong>\u00a0.yml file for the workflow and a (2)<\/em><\/strong> python file for the actual validation check.<\/p>\n (1)<\/em><\/strong> This is how the validation_docs<\/em> file looks like. It is placed in the github\/workflows<\/em> folder.<\/p>\n The workflow will run whenever a pull request is open or re-open, and every time that a new commit is pushed to the remote branch. Then there are basically 3 steps: retrieving the repository\u2019s files for the current pull request, install the dependencies, and run the validation script.<\/p>\n (2)<\/em><\/strong>. Then the validate_docs.py <\/em>script, placed in the root folder of your dbt project repository, that looks like\u00a0this<\/p>\n Let\u2019s summarise the steps in the\u00a0script:<\/p>\n Now, let\u2019s have a look at the CI in action. Given the orders.md<\/em> file shared at the beginning of the article, we now push to remote this commit that contains the ref_country_orders.yml<\/em> file:<\/p>\n The CI has failed. Clicking on the Details will take us to the log of the CI, where we see\u00a0this:<\/p>\n Let\u2019s analyze the log: All good so far. But let\u2019s keep looking at the\u00a0log:<\/p>\n Since order_country<\/em><\/strong>, order_value<\/em><\/strong>, order_date<\/em><\/strong> are in the docs file, but the doc function isn\u2019t used, the CI raise an error. And it suggests the actual value to add in the description, which makes it extremely easy for the PR author to copy-paste the correct description value from the CI log and add it into the\u00a0.yml<\/em>\u00a0file.<\/p>\n After pushing the new changes the CI check was succesfull and the log now looks like\u00a0this:<\/p>\n For the order_address<\/em><\/strong> column, the log shows that no matching doc was found. However, that\u2019s fine and does not cause the CI to fail, since adding that column to the docs file is not our intention for this demonstration. Meanwhile, the rest of the columns are listed in the docs and are correctly using the {{ doc() }}<\/em>\u00a0function<\/p>\n In conclusion, by integrating this validation CI into your dbt repository, you can confidently maintain a centralised source of truth for column descriptions across your entire project and make the best out of the {{ doc() }} feature in dbt. This setup can save valuable review time, reduces human error, and upholds a higher standard of documentation quality. As your project grows, you\u2019ll find that this automated approach to documentation management is both scalable and maintainable, ultimately enabling your team to focus on analytics rather than debugging inconsistent docs.<\/p>\n Optimize the dbt Doc Function with a CI<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n# Fields description
## order_id
{% docs orders__order_id %}
Unique alphanumeric identifier of the order, used to join all order dimension tables
{% enddocs %}
## order_country
{% docs orders__order_country %}
Country where the order was placed. Format is country ISO 3166 code.
{% enddocs %}
## order_value
{% docs orders__value %}
Total value of the order in local currency.
{% enddocs %}
## order_date
{% docs orders__date %}
Date of the order in local timezone
{% enddocs %}<\/pre>\n columns:
- name: order_id
description: '{{ doc(\"orders__order_id\") }}'<\/pre>\nThe challenge<\/h4>\n
\n
The solution<\/h4>\n
\n
name: Validate Documentation
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
validate_docs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions\/checkout@v3
with:
fetch-depth: 0
- name: Install PyYAML
run: pip install pyyaml
- name: Run validation script
run: python validate_docs.py<\/pre>\nimport os
import sys
import yaml
import re
from glob import glob
from pathlib import Path
import subprocess
def get_changed_files():
diff_command = ['git', 'diff', '--name-only', 'origin\/main...']
result = subprocess.run(diff_command, capture_output=True, text=True)
changed_files = result.stdout.strip().split('n')
return changed_files
def extract_doc_names():
doc_names = set()
md_files = glob('docs\/**\/*.md', recursive=True)
doc_pattern = re.compile(r'{%s*docss+([^s%]+)s*%}')
for md_file in md_files:
with open(md_file, 'r') as f:
content = f.read()
matches = doc_pattern.findall(content)
doc_names.update(matches)
print(f\"Extracted doc names: {doc_names}\")
return doc_names
def parse_yaml_file(yaml_path):
with open(yaml_path, 'r') as f:
try:
return list(yaml.safe_load_all(f))
except yaml.YAMLError as exc:
print(f\"Error parsing YAML file {yaml_path}: {exc}\")
return []
def validate_columns(columns, doc_names, errors, model_name):
for column in columns:
column_name = column.get('name')
description = column.get('description', '')
print(f\"Validating column '{column_name}' in model '{model_name}'\")
print(f\"Description: '{description}'\")
doc_usage = re.findall(r'{{s*doc([\"']([^\"']+)[\"'])s*}}', description)
print(f\"Doc usage found: {doc_usage}\")
if doc_usage:
for doc_name in doc_usage:
if doc_name not in doc_names:
errors.append(
f\"Column '{column_name}' in model '{model_name}' references undefined doc '{doc_name}'.\"
)
else:
matching_docs = [dn for dn in doc_names if dn.endswith(f\"__{column_name}\")]
if matching_docs:
suggested_doc = matching_docs[0]
errors.append(
f\"Column '{column_name}' in model '{model_name}' should use '{{{{ doc(\"{suggested_doc}\") }}}}' in its description.\"
)
else:
print(f\"No matching doc found for column '{column_name}'\")
def main():
changed_files = get_changed_files()
yaml_files = [f for f in changed_files if f.endswith('.yml') or f.endswith('.yaml')]
doc_names = extract_doc_names()
errors = []
for yaml_file in yaml_files:
if not os.path.exists(yaml_file):
continue
yaml_content = parse_yaml_file(yaml_file)
for item in yaml_content:
if not isinstance(item, dict):
continue
models = item.get('models') or item.get('sources')
if not models:
continue
for model in models:
model_name = model.get('name')
columns = model.get('columns', [])
validate_columns(columns, doc_names, errors, model_name)
if errors:
print(\"Documentation validation failed with the following errors:\")
for error in errors:
print(f\"- {error}\")
sys.exit(1)
else:
print(\"All documentation validations passed.\")
if __name__ == \"__main__\":
main()<\/pre>\n\n
Any problems (missing doc references, non-existent referenced docs) are recorded as\u00a0errors.<\/li>\n<\/ol>\nExamples<\/h4>\n
version: 2
models:
- name: ref_country_orders
description: >
This model filters orders from the staging orders table to include only those with an order date on or after January 1, 2020.
It includes information such as the order ID, order country, order value, and order date.
columns:
- name: order_id
description: '{{ doc(\"orders__order_id\") }}'
- name: order_country
description: The country where the order was placed.
- name: order_value
description: The value of the order.
- name: order_address
description: The address where the order was placed.
- name: order_date<\/pre>\n<\/figure>\nValidating column 'order_id' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_id\") }}'
Doc usage found: ['orders__order_id']
Validating column 'order_country' in model 'ref_country_orders'
Description: 'The country where the order was placed.'
Doc usage found: []
Validating column 'order_value' in model 'ref_country_orders'
Description: 'The value of the order.'
Doc usage found: []
Validating column 'order_address' in model 'ref_country_orders'
Description: 'The address where the order was placed.'
Doc usage found: []
No matching doc found for column 'order_address'
Validating column 'order_date' in model 'ref_country_orders'
Description: ''
Doc usage found: []<\/pre>\n
– for the order_id<\/em><\/strong> column it found the doc usage in its description.
– the order_address<\/em><\/strong> column isn\u2019t found in the docs file, so it returns a No matching doc found for column \u2018order_address\u2019
<\/em>– for the order_value<\/em><\/strong> and order_country<\/em><\/strong>, it knows that they are listed in the docs but the doc usage is empty. Same for the order_date<\/em><\/strong>, and note that for this one we didn\u2019t even add a description line<\/p>\nDocumentation validation failed with the following errors:
- Column 'order_country' in model 'ref_country_orders' should use '{{ doc(\"orders__order_country\") }}' in its description.
- Column 'order_value' in model 'ref_country_orders' should use '{{ doc(\"orders__order_value\") }}' in its description.
- Column 'order_date' in model 'ref_country_orders' should use '{{ doc(\"orders__order_date\") }}' in its description.
Error: Process completed with exit code 1.<\/pre>\n<\/figure>\nValidating column 'order_id' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_id\") }}'
Doc usage found: ['orders__order_id']
Validating column 'order_country' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_country\") }}'
Doc usage found: ['orders__order_country']
Validating column 'order_value' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_value\") }}'
Doc usage found: ['orders__order_value']
Validating column 'order_address' in model 'ref_country_orders'
Description: 'The address where the order was placed.'
Doc usage found: []
No matching doc found for column 'order_address'
Validating column 'order_date' in model 'ref_country_orders'
Description: '{{ doc(\"orders__order_date\") }}'
Doc usage found: ['orders__order_date']
All documentation validations passed.<\/pre>\n<\/p>\n
\n