dbt Cloud is supported via dedicated components. You can find them in the menu section Components:
kds-team.ex-dbt-cloud-api for extracting data from dbt Cloud API
kds-team.app-dbt-cloud-job-trigger for triggering a dbt Cloud job remotely, and optionally, wait for the job results. In that case, the component stores artifacts as well.
The component configuration is pretty straightforward. You must authorize the component by providing your
Job ID, and
The component generates a status table called
dbt_cloud_trigger storing the job trigger API response:
When Wait for result is selected, the component polls the status until the job ends. The component has a default wait time limit that can be optionally set to a different time. When the option Wait for result is used, the component extracts artifacts, stores them in the file storage, and additionally, produces a job result API call table:
Both tables can be found in the storage, or accessed directly from the job result:
Artifacts can be found in the storage - files - search by tag:
Search by tag (component type or configuration ID):
Tip:: Those files can be also easily retrieved externally via the API or from an integrated Jupyter workspace for further analysis.
Note: Please keep in mind that the base URL of the API call depends on the stack you are using: US vs. Azure EU vs. EU central.
The purpose of this source component is to extract and store the dbt Cloud API information (data is stored incrementally) for the following endpoints:
To configure the source component, enter the API token and select a default configuration:
You can access the data from the storage, or directly from the job detail screen:
Note: The source component utilizes our powerful Generic extractor. In case you want to customize the extraction, select just some endpoints, etc. You can switch to the JSON schema and edit the configuration manually.