If you already know how development branches work in general and want to create and start using your first branch, go to our Getting Started tutorial.
Development Branches allow you to modify component configurations without interfering with running configurations or entire orchestrated pipelines. They are ideal to use when making bigger changes to a project or when you need to be extra careful about performing your changes safely.
To give an example, let’s say that you have an ordinary orchestration that extracts, transforms and writes data to a target system, and you need to remove a column from the source. To do that, you must modify several configurations, and ideally, also perform a dry run to check that the data in the target system is correct. However, modifying a pipeline that runs, e.g., every ten minutes, is difficult without an outage of the pipeline. Development Branches are designed to help in such situations.
When you create a development branch in your project, you obtain an exact copy of the project and all its current configurations. You can then modify these configurations without ever touching the original ones in production, and these will keep running in orchestrations.
When you run a configuration in a branch, it is able to read the tables and files from the Storage as if it were a normal configuration. However, when your branch configuration attempts to write data (tables or files) to Storage, it is stored separately and does not overwrite the original production data and interfere with the running configurations. There is no need to duplicate your project’s data when creating a new branch.
To be able to run chains of configurations in a branch, e.g., an extractor followed by a transformation, the following rules apply:
If you run a branch configuration and branch data is available, it is used. Only if the branch data is not present, the production data is used. Therefore, when you run a branch transformation, it will read the production data and write branch output data. When you run an extractor on which the transformation depends in the branch, it too will create branch output data. When you run the transformation again, it will use the output of the branch extractor and write branch output data.
If you have your configurations ready in production and want to create a branch to test some changes, click on your project’s name at the top of the screen. Then click on the green icon New displayed next to your project’s name.
Name your new branch and click Create Development Branch to open it.
The branch will appear right below the name of your production project.
Now you can start modifying your configurations, run them, and analyze the results.
If you want to learn more about working in a branch, follow our tutorial.
Before you merge your development branch back to production, check a detailed diff of the configuration changes.
You can end your branch’s lifecycle in two ways:
Important: All this happens in a same project, allowing you to collaborate on the modifications with other members of the project.
This feature is currently in public beta. It may not always work as expected. Please bear with us and provide feedback at https://ideas.keboola.com.