1. Home
2. Components
3. Data Source Connectors
4. Databases
5. MongoDB

MongoDB

• Set Up Database Credentials
• Configure Exports
  • Strict Format
• Configure Mapping
  • Primary Key
  • Other Data Types
• Raw Export Mode

The MongoDB data source connector allows you to fetch data from the NoSQL MongoDB database. Supported MongoDB versions are from 4.4 to latest (6.0). 4.2 could work also, but not guaranteed. Lower version are not supported by version of mongoexport. Complete the following steps to configure it.

Set Up Database Credentials

Create a new configuration and click Set Up Database Credentials and fill in the form. Then test the new credentials and save them. Optionally, set up an SSH tunnel.

Configure Exports

Click the New Export button and configure your first export using the following options:

• Name – Identifies your export; its value has to be unique across all exports and exported tables in your configuration. Also, the main exported table will be named after the value of the Name field.

• Collection – Represents the collection name in your MongoDB database.

• Query (optional) – JSON string specifying a query which limits documents data in exported data. Must be specified in a strict format.

• Sort (optional) – JSON string specifying the order of documents in exported data. Must be specified in a strict format.

• Limit (optional) – Limits the number of exported documents.

• Incremental – Loads data to your tables incrementally.

• Mode – Specifies the export mode: Mapping or Raw. Start by exporting few documents using the Raw mode first – it will help you see the document structure for which you need to write mapping.

• Mapping – This is the most important section in case you have selected the Mapping mode; it defines how documents in the collection are mapped to the output tables. It, too, has to be valid JSON.

Tip: Use a combination of a limit (for example, only 10 documents) and a query (for example, only a document with specific ID) while playing with the mapping section to prevent a full collection export.

Strict Format

A strict format means standard valid JSON must be used, and you cannot use MongoDB objects as in the JavaScript shell interface. Thus using objects such as ObjectId, Date or NumberLong is not allowed until you specify them in the strict format. Read more on the strict format.

Configure Mapping

By defining mapping, you specify the structure and content of your output tables, their columns and relations between them.

Tip: Export few documents using the Raw Export Mode first – it will help you see the document structure (in Strict Format) for which you need to write mapping.

Primary Key

Since MongoDB identifies each document in a collection uniquely by _id, we recommend to set a primary key to this field by defining the first item in a mapping section:

{
    "_id.$oid": {
        "type": "column",
        "mapping": {
            "destination": "id",
            "primaryKey": true
        }
    }
}

Note: The destination column with the primary key should be named id not _id to prevent problems with the data import.

Other Data Types

To handle MongoDB data types correctly, define mapping similarly to the following example for MongoId, ISODate, and NumberLong data types.

{
    "_id" : ObjectId("5a9d1e9b00c99bbb33c9863a"),
    "publishedAt" : ISODate("2018-03-05T10:40:27.938Z"),
    "views" : NumberLong(1)
}

{
    "_id": {
        "$oid": "5a9d1e9b00c99bbb33c9863a"
    },
    "publishedAt": {
        "$date": "2018-03-05T10:40:27.938Z"
    },
    "views": {
        "$numberLong": "1"
    }
}

{
    "_id.$oid": "id",
    "publishedAt.$date": "publishedAt",
    "views.$numberLong": "views"
}

Check out more mapping examples.

Raw Export Mode

In the raw export mode, documents are exported as plain JSON strings.

{
    "_id" : ObjectId("5716054bee6e764c94fa7ddd"),
    "name" : "MongoDB connector"
}

{
    "_id": {
        "$oid" : "5716054bee6e764c94fa7ddd"
    },
    "name": "MongoDB connector"
}