Skip to main content

Hive

For context on getting started with ingestion, check out our metadata ingestion guide.

Setup

To install this plugin, run pip install 'acryl-datahub[hive]'.

Capabilities

This plugin extracts the following:

  • Metadata for databases, schemas, and tables
  • Column types associated with each table
  • Detailed table and storage information
  • Table, row, and column statistics via optional SQL profiling

Quickstart recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
type: hive
config:
# Coordinates
host_port: localhost:10000
database: DemoDatabase # optional, if not specified, ingests from all databases

# Credentials
username: user # optional
password: pass # optional

# For more details on authentication, see the PyHive docs:
# https://github.com/dropbox/PyHive#passing-session-configuration.
# LDAP, Kerberos, etc. are supported using connect_args, which can be
# added under the `options` config parameter.
#scheme: 'hive+http' # set this if Thrift should use the HTTP transport
#scheme: 'hive+https' # set this if Thrift should use the HTTP with SSL transport

sink:
# sink configs

Ingestion with

Azure HDInsight
# Connecting to Microsoft Azure HDInsight using TLS.
source:
type: hive
config:
# Coordinates
host_port: <cluster_name>.azurehdinsight.net:443

# Credentials
username: admin
password: password

# Options
options:
connect_args:
http_path: "/hive2"
auth: BASIC

sink:
# sink configs
Databricks

Ensure that databricks-dbapi is installed. If not, use pip install databricks-dbapi to install.

Use the http_path from your Databricks cluster in the following recipe. See here for instructions to find http_path.

source:
type: hive
config:
host_port: <databricks workspace URL>:443
username: token
password: <api token>
scheme: 'databricks+pyhive'

options:
connect_args:
http_path: 'sql/protocolv1/o/xxxyyyzzzaaasa/1234-567890-hello123'

sink:
# sink configs

Config details

Note that a . is used to denote nested fields in the YAML recipe.

As a SQL-based service, the Athena integration is also supported by our SQL profiler. See here for more details on configuration.

FieldRequiredDefaultDescription
usernameDatabase username.
passwordDatabase password.
host_portHost URL and port to connect to.
databaseDatabase to ingest.
database_aliasAlias to apply to database when ingesting.
env"PROD"Environment to use in namespace when constructing URNs.
options.<option>Any options specified here will be passed to SQLAlchemy's create_engine as kwargs.
See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details.
table_pattern.allowList of regex patterns for tables to include in ingestion.
table_pattern.denyList of regex patterns for tables to exclude from ingestion.
table_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
schema_pattern.allowList of regex patterns for schemas to include in ingestion.
schema_pattern.denyList of regex patterns for schemas to exclude from ingestion.
schema_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
view_pattern.allowList of regex patterns for views to include in ingestion.
view_pattern.denyList of regex patterns for views to exclude from ingestion.
view_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
include_tablesTrueWhether tables should be ingested.

Compatibility

Coming soon!

Questions

If you've got any questions on configuring this source, feel free to ping us on our Slack!