Google BigQuery

This writer sends data to a Google BigQuery dataset.

Create New Configuration

Find the BigQuery writer in the list of writers and create a new configuration. Name it.

Screenshot - Create configuration

Create Service Account

To access and write to your BigQuery dataset, you need to set up a Google Service Account.

Screenshot - Create service account

  • Then add the BigQuery Data Editor and BigQuery Job User roles.

Screenshot - Create service account

  • Finally, create a new JSON key (click + Create key) and download it to your computer (click Create).

Screenshot - Create service account

  • You can now close the Google Cloud Platform Console and go back to your BigQuery writer configuration.
  • In the Google Service Account Key section click Set Service Account Key.

Screenshot - Set Service Account Key

  • Open the downloaded key in a text editor, copy & paste it in the input field, click Submit and then Save.

Screenshot - Copy & Paste Service Account Key

Important: The private key is stored in an encrypted form and only the non-sensitive parts are visible in the UI for your verification. The key can be deleted or replaced by a new one at any time.

Screenshot - Copy & Paste Service Account Key

Google BigQuery Dataset

Screenshot - BigQuery Dataset

There is one more thing left to do before you can start adding tables. Specify the Google BigQuery Dataset and save it. All tables in this configuration will be written to this dataset. If the dataset does not exist, the roles assigned to the Google Service Account will allow the writer to create it.

Add Tables

Screenshot - Add Table

To add a new table to the writer, click + Add Table and select the table. The table name will be used to create the destination table name in BigQuery and can be modified.

List Tables

Screenshot - Tables List

The configuration can write as many tables as you wish. The list is fully searchable, and you can delete or disable each table. In addition, you can explicitly run only one table. The writing order of the tables can be changed.

Modify Table

Screenshot - Table Detail

Destination

You can specify the table name in BigQuery and set the load type to Full Load or Incremental.

Note: Incremental load type does not use a primary key to modify existing records, new records will be always appended to the table.

Columns

You can rename the destination column in BigQuery and specify the used data type. The little eye icon on the right will show you a preview of the values so you don’t have to guess the data type.

Note: You have to define a data type on at least one column for the configuration to work.