1. Home
  2. Transformations
  3. dbt Transformation
  4. CLI

dbt CLI

Video:

Local Development

Let’s set up the local development with Keboola CLI.

It is easy on Mac with homebrew support (other platforms covered in the documentation):

brew tap keboola/keboola-cli
brew install keboola-cli
kbc --version

If you already have CLI, make sure you have the most updated version:

brew upgrade keboola-cli
kbc --version

You will then gain access to dbt-related commands within Keboola CLI!

Steps

initial setup

You must have a Keboola project, a cloned git repository with a dbt project, and the dbt binary installed.

The user is in the folder with the cloned dbt project and can run the following commands.

kbc dbt init

  1. Creates a Snowflake workspace.

  2. Generates profiles and sources yaml files.

    1. DBT_KBC_DEV_{ENV_NAME}_DATABASE (we ask for env name)

  3. Generates env vars (used profiles.yml).

  4. They are outputted to stdout.

Example

Store credentials to your zsh env profile (or your respective environment):

The file is located (Unix) in ~/.zshrc

Then you can run dbt locally against the project storage, safely develop and test your code.

As part of the init command, CLI will create all sources from storage buckets. A storage bucket is a source file containing tables:

Note: Please note that _timestamp is automatically filled, alongside primary keys and corresponding tests for primary keys (unique and not_null tests).

Run Test Debug

dbt debug -t beer_demo --profiles-dir .

Notes

  • beer_demo is the target name used in the prior step and visible in profiles.yml

  • We are using local profiles; they are using environmental variables stored before.

You should see green in all steps:

dbt Run

For the script alteration, the only check/change you have to make with off-the-shelf scripts is to alter source definitions to match sources:

To execute the dbt:

dbt run -t beer_demo  --profiles-dir .

Other Commands for Future Use

kbc dbt generate profile

  • Generates just profiles.yml.

  • Reads the profile name from dbt_project.yml and the target name from the input.

  • Stores the profile to profiles.yml.

    • If there is an existing profile with the same name, it will be overwritten. Otherwise, the new profile will be just appended to the others if there are any.

  • Example (non-interactive mode):

kbc dbt generate sources

  • Generates only sources.yml.

  • Lists all tables in the default branch from the Storage API and generates source files to models/_sources. Tables from each bucket are stored in a separate file.

  • Example (non-interactive mode)

kbc dbt generate env

  • Generates bash commands to create env vars for dbt CLI.

  • Asks for an existing workspace (select box or id flag).

  • Example

Workspaces Support

  • Universal support to manage workspaces

  • Hide keboola.sandboxes configurations from the Keboola project folder structure.

kbc remote workspace create

  • Supports parameter name, type, and size (for python and r).

  • Example (non-interactive mode)