MySQL Transformation

Note: MySQL transformations are deprecated. They are only available in legacy projects.

MySQL is very simple, yet powerful. However, it has its limitations:

  • Time — MySQL queries are limited to 3,600 seconds. Longer queries will be terminated.

  • Resources — All MySQL queries are run on a shared server with 8 CPUs, SSD drives connected in RAID and 160 GB RAM.

If your queries last more than 10 minutes, or transformations in your orchestration take more than an hour, please consider using Redshift or Snowflake. You can iterate faster on your development without having to wait too long for the next test run to finish.

  • Query Length — Queries containing comments longer than 8,192 characters will segfault.

The version used by MySQL transformations is MariaDB 5.55.34. The server is fine-tuned for performance.

To get the most out of MySQL,

  • take care of column indexes and correct data types in the input mapping.
  • divide large queries into a series of simple queries. It makes the queries easier to optimize.
  • create plenty of intermediary or temporary tables, and
  • always try to materialize the query (use CREATE TABLE ... instead of CREATE VIEW).

Examples of some common scenarios and their solutions can be found on our wiki.