Storage

See our Getting Started tutorial for instructions on how to use Storage.

As the central KBC subsystem, Storage manages everything related to storing data and accessing it. It is implemented as a layer on top of various database engines that we use as our backends (MySQL/MariaDB, Redshift, and Snowflake).

As with all other KBC components, everything that can be done through the UI can be also done programatically via the Storage API. See our developers’ guide to learn more. Every operation done in Storage must be authorized via a token.

Storage Data

The Storage component manages all data stored in each KBC project:

Backend Properties

Data in Storage are internally stored in a database backend. Specific properties of each backend are compared in the following table:

Feature MySQL Redshift Snowflake
Partial Import (Deprecated) x x
Export formats RFC, ESCAPED*, RAW* RFC, ESCAPED*, RAW* RFC
Maximum number of columns in single table Max. row size of 65,535 bytes 1200 1200
Maximum table cell size 64kB 64kB 1MB
Sync export (Data Preview) columns limit x x 110

* ESCAPED and RAW formats are different for Redshift and Mysql

Redshift Table Size

The reported table size on the Redshift backend often tends to be inaccurate and affects mostly tables with many small incremental loads.

Since this is an issue of Redshift itself, we decided to recalculate Table Size by ourselves. All recalculating jobs are executed automatically when loading data to tables and when the actual table size is greater than 500MB.

However, any recalculating job can be also called manually by calling the Table Optimize method in Storage API.