Skip to main content

MongoDB

For context on getting started with ingestion, check out our metadata ingestion guide.

Setup#

To install this plugin, run pip install 'acryl-datahub[mongodb]'.

Capabilities#

This plugin extracts the following:

  • Databases and associated metadata
  • Collections in each database and schemas for each collection (via schema inference)

By default, schema inference samples 1,000 documents from each collection. Setting schemaSamplingSize: null will scan the entire collection. Moreover, setting useRandomSampling: False will sample the first documents found without random selection, which may be faster for large collections.

Note that schemaSamplingSize has no effect if enableSchemaInference: False is set.

Really large schemas will be further truncated to a maximum of 300 schema fields. This is configurable using the maxSchemaSize parameter.

Quickstart recipe#

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:  type: "mongodb"  config:    # Coordinates    connect_uri: "mongodb://localhost"
    # Credentials    username: admin    password: password    authMechanism: "DEFAULT"
    # Options    enableSchemaInference: True    useRandomSampling: True    maxSchemaSize: 300
sink:  # sink configs

Config details#

Note that a . is used to denote nested fields in the YAML recipe.

FieldRequiredDefaultDescription
connect_uri"mongodb://localhost"MongoDB connection URI.
usernameMongoDB username.
passwordMongoDB password.
authMechanismMongoDB authentication mechanism. See https://pymongo.readthedocs.io/en/stable/examples/authentication.html for details.
optionsAdditional options to pass to pymongo.MongoClient().
enableSchemaInferenceTrueWhether to infer schemas.
schemaSamplingSize1000Number of documents to use when inferring schema size. If set to 0, all documents will be scanned.
maxSchemaSize300Maximum number of fields to include in the schema.
useRandomSamplingTrueIf documents for schema inference should be randomly selected. If False, documents will be selected from start.
env"PROD"Environment to use in namespace when constructing URNs.
database_pattern.allowList of regex patterns for databases to include in ingestion.
database_pattern.denyList of regex patterns for databases to exclude from ingestion.
database_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
collection_pattern.allowList of regex patterns for collections to include in ingestion.
collection_pattern.denyList of regex patterns for collections to exclude from ingestion.
collection_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.

Compatibility#

Coming soon!

Questions#

If you've got any questions on configuring this source, feel free to ping us on our Slack!