Part 4 - Automation - Setting up Orchestrator

So far, you have learned to use Keboola Connection to

Connecting various systems together alone makes Keboola Connection a powerful and easy-to-use tool. However, the above steps must be done repeatedly to bring in the newest data available.

Use Orchestrator

  • to specify what tasks should be executed in what order (orchestrate tasks) and
  • to configure the automatic execution (schedule orchestrated tasks).

Go to the Orchestrations section of Keboola Connection, and

Screenshot - Orchestrations Introduction

click on New Orchestration to create a new orchestration. Assign it the name Opportunities:

Screenshot - Create new Orchestration

To configure the orchestration, first add some tasks to it:

Screenshot - Orchestration Detail

Click New Task:

Screenshot - Orchestration Tasks Introduction

The automation tasks are displayed based on what steps of the tutorial you have taken. It is not possible to automate the manual upload. If you haven’t gone through all parts of the tutorial, these are the available steps:

First, select GoogleDrive and then click on the configuration User levels.

Screenshot - Orchestration Tasks Editing

Continue adding all the tasks you want. The following configuration will extract data from the database and from the Google Drive sheet. After being transformed for Tableau, the data will be written to Tableau.

Screenshot - Orchestration Tasks Setup for Tableau

Or, use the next configuration to extract data from the database and the Google Drive sheet, transform it for GoodData, and write it to a GoodData project.

Screenshot - Orchestration Tasks Setup for GoodData

The order of certain tasks is important; some must run sequentially and others can run in parallel. That is what orchestration phases are for. Tasks in a single phase are executed in parallel, phases execute sequentially.

To order the phases, grab the triple bar icon on their left. To move a task to a different phase, tick the checkbox on the left. Then go to Actions, select Move selected tasks between phases and assign the desired phase.

In the above configuration, each task is in its own phase. Therefore, this is a very defensive configuration which executes all tasks sequentially.

It can be better arranged by using the action Group tasks into phases by component type:

Screenshot - Orchestration Phases

This will group each of the extractors, transformations and writers into their own phase to follow the common ETL scheme. Then Save the orchestration. If you do not have all the tasks set up at the moment, it does not matter. You can safely continue with the next steps. When done configuring the tasks, go back to the orchestration setting.

Screenshot - Orchestration Tasks Setup End

In the orchestration detail, you can now see some tasks configured. Run the orchestration manually to test if everything works smoothly; click on the Run Orchestration button in the top right corner and select the tasks you want to run. This creates a background job which executes all the tasks specified in the orchestration. Continue setting up the orchestration in the meantime.

Screenshot - Orchestration Detail

By clicking on the edit icon next to Schedule, set the orchestration to run automatically at a given time.

Screenshot - Orchestration Schedule

It is recommended to also set up notifications. Click on the Notifications edit button:

Screenshot - Orchestration Detail

Notifications are sent to selected email addresses. Set at least the error notification: enter your email address and click on the plus sign next to it. Repeat if you want to add another email address. Then click Save.

When an orchestration is run manually, notifications will be sent only to the Keboola Connection user who runs the orchestration, not to those specified in Notifications.

Screenshot - Orchestration Notifications

Your orchestration job should be finished by now. From data extraction to data writing, you have set up the full pipeline. Any change in your GoogleDrive sheet will automatically propagate up to your Tableau or GoodData project. Or both if you set it that way.

Screenshot - Orchestration Jobs

Having mastered the automation process, you may proceed to the ad-hoc data analysis part of the Keboola Connection tutorial.