Skip to main content

BigQuery

There are 2 sources that provide integration with BigQuery

Source ModuleDocumentation

bigquery

This plugin extracts the following:

  • Metadata for databases, schemas, and tables
  • Column types associated with each table
  • Table, row, and column statistics via optional SQL profiling
  • Table level lineage. Read more...

bigquery-usage

This plugin extracts the following:

  • Statistics on queries issued and tables and columns accessed (excludes views)
  • Aggregation of these statistics into buckets, by day or hour granularity
note
  1. This source only does usage statistics. To get the tables, views, and schemas in your BigQuery project, use the bigquery plugin.
  2. Depending on the compliance policies setup for the bigquery instance, sometimes logging.read permission is not sufficient. In that case, use either admin or private log viewer permission.

Read more...

To get all metadata from BigQuery you need to use two plugins bigquery and bigquery-usage. Both of them are described in this page. These will require 2 separate recipes. We understand this is not ideal and we plan to make this easier in the future.

Module bigquery

Certified

Important Capabilities

CapabilityStatusNotes
Data ProfilingOptionally enabled via configuration
Dataset UsageNot provided by this module, use bigquery-usage for that.
DescriptionsEnabled by default
Detect Deleted EntitiesEnabled via stateful ingestion
DomainsSupported via the domain config field
Platform InstanceBigQuery doesn't need platform instances because project ids in BigQuery are globally unique.
Table-Level LineageEnabled by default

This plugin extracts the following:

  • Metadata for databases, schemas, and tables
  • Column types associated with each table
  • Table, row, and column statistics via optional SQL profiling
  • Table level lineage.

Install the Plugin

pip install 'acryl-datahub[bigquery]'

Quickstart Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide

source:
type: bigquery
config:
# Coordinates
project_id: my_project_id

# `schema_pattern` for BQ Datasets
schema_pattern:
allow:
- finance_bq_dataset

table_pattern:
deny:
# The exact name of the table is revenue_table_name
# The reason we have this `.*` at the beginning is because the current implmenetation of table_pattern is testing
# project_id.dataset_name.table_name
# We will improve this in the future
- .*revenue_table_name

sink:
# sink configs

Config Details

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options
FieldRequiredTypeDescriptionDefault
envstringThe environment that all assets produced by this connector belong toPROD
platformstringThe platform that this source connects toNone
platform_instancestringThe instance of the platform that all assets produced by this recipe belong toNone
optionsDict{}
include_viewsbooleanWhether views should be ingested.True
include_tablesbooleanWhether tables should be ingested.True
bucket_durationenum(BucketDuration)Size of the time window to aggregate usage stats..DAY
end_timestringLatest date of usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration)None
start_timestringEarliest date of usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration)None
rate_limitbooleanShould we rate limit requests made to API.False
requests_per_minintegerUsed to control number of API calls made per min. Only used when rate_limit is set to True.60
temp_table_dataset_prefixstringIf you are creating temp tables in a dataset with a particular prefix you can use this config to set the prefix for the dataset. This is to support workflows from before bigquery's introduction of temp tables. By default we use _ because of datasets that begin with an underscore are hidden by default https://cloud.google.com/bigquery/docs/datasets#dataset-naming._
sharded_table_patternstringThe regex pattern to match sharded tables and group as one table. This is a very low level config parameter, only change if you know what you are doing,((.+)[_$])?(\d{4,10})$
schemestringbigquery
project_idstringProject ID where you have rights to run queries and create tables. If storage_project_id is not specified then it is assumed this is the same project where data is stored. If not specified, will infer from environment.None
storage_project_idstringIf your data is stored in a different project where you don't have rights to run jobs and create tables then specify this field. The same service account must have read rights in this GCP project and write rights in project_id.None
log_page_sizeintegerThe number of log item will be queried per page for lineage collection1000
extra_client_optionsDictAdditional options to pass to google.cloud.logging_v2.client.Client.{}
include_table_lineagebooleanOption to enable/disable lineage generation. Is enabled by default.True
max_query_durationnumberCorrection to pad start_time and end_time with. For handling the case where the read happens within our time range but the query completion event is delayed and happens after the configured end time.900.0
bigquery_audit_metadata_datasetsArray of stringA list of datasets that contain a table named cloudaudit_googleapis_com_data_access which contain BigQuery audit logs, specifically, those containing BigQueryAuditMetadata. It is recommended that the project of the dataset is also specified, for example, projectA.datasetB.None
use_exported_bigquery_audit_metadatabooleanWhen configured, use BigQueryAuditMetadata in bigquery_audit_metadata_datasets to compute lineage information.False
use_date_sharded_audit_log_tablesbooleanWhether to read date sharded tables or time partitioned tables when extracting usage from exported audit logs.False
use_v2_audit_metadatabooleanWhether to ingest logs using the v2 format.False
upstream_lineage_in_reportbooleanUseful for debugging lineage information. Set to True to see the raw lineage created internally.False
stateful_ingestionSQLAlchemyStatefulIngestionConfig (see below for fields)
stateful_ingestion.enabledbooleanThe type of the ingestion state provider registered with datahub.False
stateful_ingestion.max_checkpoint_state_sizeintegerThe maximum size of the checkpoint state in bytes. Default is 16MB16777216
stateful_ingestion.state_providerDynamicTypedStateProviderConfig (see below for fields)The ingestion state provider configuration.
stateful_ingestion.state_provider.typestringThe type of the state provider to use. For DataHub use datahubNone
stateful_ingestion.state_provider.configGeneric dictThe configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).None
stateful_ingestion.ignore_old_statebooleanIf set to True, ignores the previous checkpoint state.False
stateful_ingestion.ignore_new_statebooleanIf set to True, ignores the current checkpoint state.False
stateful_ingestion.remove_stale_metadatabooleanSoft-deletes the tables and views that were found in the last successful run but missing in the current run with stateful_ingestion enabled.True
schema_patternAllowDenyPattern (see below for fields)regex patterns for schemas to filter in ingestion.{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
schema_pattern.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
schema_pattern.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
schema_pattern.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
schema_pattern.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
table_patternAllowDenyPattern (see below for fields)regex patterns for tables to filter in ingestion.{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
table_pattern.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
table_pattern.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
table_pattern.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
table_pattern.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
view_patternAllowDenyPattern (see below for fields)regex patterns for views to filter in ingestion.{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
view_pattern.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
view_pattern.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
view_pattern.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
view_pattern.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
profile_patternAllowDenyPattern (see below for fields)regex patterns for profiles to filter in ingestion, allowed by the table_pattern.{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
profile_pattern.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
profile_pattern.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
profile_pattern.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
profile_pattern.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
domainDict[str, AllowDenyPattern]regex patterns for tables/schemas to descide domain_key domain key (domain_key can be any string like "sales".) There can be multiple domain key specified.{}
domain.key.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
domain.key.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
domain.key.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
domain.key.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
profilingGEProfilingConfig (see below for fields){'enabled': False, 'limit': None, 'offset': None, 'reportdropped_profiles': False, 'turn_off_expensive_profiling_metrics': False, 'profile_table_level_only': False, 'include_field_null_count': True, 'include_field_min_value': True, 'include_field_max_value': True, 'include_field_mean_value': True, 'include_field_median_value': True, 'include_field_stddev_value': True, 'include_field_quantiles': False, 'include_field_distinct_value_frequencies': False, 'include_field_histogram': False, 'include_field_sample_values': True, 'allow_deny_patterns': {'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 .-]'}, 'max_number_of_fields_to_profile': None, 'profile_if_updated_since_days': 1, 'max_workers': 10, 'query_combiner_enabled': True, 'catch_exceptions': True, 'partition_profiling_enabled': True, 'bigquery_temp_table_schema': None, 'partition_datetime': None}
profiling.enabledbooleanWhether profiling should be done.False
profiling.limitintegerMax number of documents to profile. By default, profiles all documents.None
profiling.offsetintegerOffset in documents to profile. By default, uses no offset.None
profiling.report_dropped_profilesbooleanIf datasets which were not profiled are reported in source report or not. Set to True for debugging purposes.False
profiling.turn_off_expensive_profiling_metricsbooleanWhether to turn off expensive profiling or not. This turns off profiling for quantiles, distinct_value_frequencies, histogram & sample_values. This also limits maximum number of fields being profiled to 10.False
profiling.profile_table_level_onlybooleanWhether to perform profiling at table-level only, or include column-level profiling as well.False
profiling.include_field_null_countbooleanWhether to profile for the number of nulls for each column.True
profiling.include_field_min_valuebooleanWhether to profile for the min value of numeric columns.True
profiling.include_field_max_valuebooleanWhether to profile for the max value of numeric columns.True
profiling.include_field_mean_valuebooleanWhether to profile for the mean value of numeric columns.True
profiling.include_field_median_valuebooleanWhether to profile for the median value of numeric columns.True
profiling.include_field_stddev_valuebooleanWhether to profile for the standard deviation of numeric columns.True
profiling.include_field_quantilesbooleanWhether to profile for the quantiles of numeric columns.False
profiling.include_field_distinct_value_frequenciesbooleanWhether to profile for distinct value frequencies.False
profiling.include_field_histogrambooleanWhether to profile for the histogram for numeric fields.False
profiling.include_field_sample_valuesbooleanWhether to profile for the sample values for all columns.True
profiling.allow_deny_patternsAllowDenyPattern (see below for fields)regex patterns for filtering of tables or table columns to profile.{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
profiling.allow_deny_patterns.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
profiling.allow_deny_patterns.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
profiling.allow_deny_patterns.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
profiling.allow_deny_patterns.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
profiling.max_number_of_fields_to_profileintegerA positive integer that specifies the maximum number of columns to profile for any table. None implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up.None
profiling.profile_if_updated_since_daysnumberProfile table only if it has been updated since these many number of days. None implies profile all tables. Only Snowflake supports this.1
profiling.max_workersintegerNumber of worker threads to use for profiling. Set to 1 to disable.10
profiling.query_combiner_enabledbooleanThis feature is still experimental and can be disabled if it causes issues. Reduces the total number of queries issued and speeds up profiling by dynamically combining SQL queries where possible.True
profiling.catch_exceptionsbooleanTrue
profiling.partition_profiling_enabledbooleanTrue
profiling.bigquery_temp_table_schemastringOn bigquery for profiling partitioned tables needs to create temporary views. You have to define a dataset where these will be created. Views will be cleaned up after profiler runs. (Great expectation tech details about this (https://legacy.docs.greatexpectations.io/en/0.9.0/reference/integrations/bigquery.html#custom-queries-with-sql-datasource).None
profiling.partition_datetimestringFor partitioned datasets profile only the partition which matches the datetime or profile the latest one if not set. Only Bigquery supports this.None
credentialBigQueryCredential (see below for fields)BigQuery credential informations
credential.project_idstringProject id to set the credentialsNone
credential.private_key_idstringPrivate key idNone
credential.private_keystringPrivate key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n'None
credential.client_emailstringClient emailNone
credential.client_idstringClient IdNone
credential.auth_uristringAuthentication urihttps://accounts.google.com/o/oauth2/auth
credential.token_uristringToken urihttps://oauth2.googleapis.com/token
credential.auth_provider_x509_cert_urlstringAuth provider x509 certificate urlhttps://www.googleapis.com/oauth2/v1/certs
credential.typestringAuthentication typeservice_account
credential.client_x509_cert_urlstringIf not set it will be default to https://www.googleapis.com/robot/v1/metadata/x509/client_emailNone

Prerequisites

Create a datahub profile in GCP

  1. Create a custom role for datahub as per BigQuery docs
  2. Grant the following permissions to this role:
   bigquery.datasets.get
bigquery.datasets.getIamPolicy
bigquery.jobs.create
bigquery.jobs.list
bigquery.jobs.listAll
bigquery.models.getMetadata
bigquery.models.list
bigquery.routines.get
bigquery.routines.list
bigquery.tables.create # Needs for profiling
bigquery.tables.get
bigquery.tables.getData # Needs for profiling
bigquery.tables.list
# needed for lineage generation via GCP logging
logging.logEntries.list
logging.privateLogEntries.list
resourcemanager.projects.get
bigquery.readsessions.create
bigquery.readsessions.getData

Create a service account

  1. Setup a ServiceAccount as per BigQuery docs and assign the previously created role to this service account.
  2. Download a service account JSON keyfile. Example credential file:
{
"type": "service_account",
"project_id": "project-id-1234567",
"private_key_id": "d0121d0000882411234e11166c6aaa23ed5d74e0",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----",
"client_email": "test@suppproject-id-1234567.iam.gserviceaccount.com",
"client_id": "113545814931671546333",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test%suppproject-id-1234567.iam.gserviceaccount.com"
}
  1. To provide credentials to the source, you can either: Set an environment variable: $ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"

    or

    Set credential config in your source based on the credential json file. For example:

     credential:
project_id: project-id-1234567
private_key_id: "d0121d0000882411234e11166c6aaa23ed5d74e0"
private_key: "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----\n"
client_email: "test@suppproject-id-1234567.iam.gserviceaccount.com"
client_id: "123456678890"

Lineage Computation Details

When use_exported_bigquery_audit_metadata is set to true, lineage information will be computed using exported bigquery logs. On how to setup exported bigquery audit logs, refer to the following docs on BigQuery audit logs. Note that only protoPayloads with "type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata" are supported by the current ingestion version. The bigquery_audit_metadata_datasets parameter will be used only if use_exported_bigquery_audit_metadat is set to true.

Note: the bigquery_audit_metadata_datasets parameter receives a list of datasets, in the format $PROJECT.$DATASET. This way queries from a multiple number of projects can be used to compute lineage information.

Note: Since bigquery source also supports dataset level lineage, the auth client will require additional permissions to be able to access the google audit logs. Refer the permissions section in bigquery-usage section below which also accesses the audit logs.

Profiling Details

Profiling can profile normal/partitioned and sharded tables as well but due to performance reasons, we only profile the latest partition for Partitioned tables and the latest shard for sharded tables.

If limit/offset parameter is set or partitioning partitioned or sharded table Great Expectation (the profiling framework we use) needs to create temporary views. By default these views are created in the schema where the profiled table is but you can control to create all these tables into a predefined schema by setting profiling.bigquery_temp_table_schema property. Temporary tables are removed after profiling.

     profiling:
enabled: true
bigquery_temp_table_schema: my-project-id.my-schema-where-views-can-be-created
note

Due to performance reasons, we only profile the latest partition for Partitioned tables and the latest shard for sharded tables. You can set partition explicitly with partition.partition_datetime property if you want. (partition will be applied to all partitioned tables)

Caveats

  • For Materialized views lineage is dependent on logs being retained. If your GCP logging is retained for 30 days (default) and 30 days have passed since the creation of the materialized view we won't be able to get lineage for them.

Code Coordinates

  • Class Name: datahub.ingestion.source.sql.bigquery.BigQuerySource
  • Browse on GitHub

Module bigquery-usage

Certified

This plugin extracts the following:

  • Statistics on queries issued and tables and columns accessed (excludes views)
  • Aggregation of these statistics into buckets, by day or hour granularity
note
  1. This source only does usage statistics. To get the tables, views, and schemas in your BigQuery project, use the bigquery plugin.
  2. Depending on the compliance policies setup for the bigquery instance, sometimes logging.read permission is not sufficient. In that case, use either admin or private log viewer permission.

Install the Plugin

pip install 'acryl-datahub[bigquery-usage]'

Quickstart Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide

source:
type: bigquery-usage
config:
# Coordinates
projects:
- project_id_1
- project_id_2

# Options
top_n_queries: 10
dataset_pattern:
allow:
- marketing_db
- sales_db
table_pattern:
deny:
- .*feedback.*
- .*salary.*

sink:
# sink configs

Config Details

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options
FieldRequiredTypeDescriptionDefault
bucket_durationenum(BucketDuration)Size of the time window to aggregate usage stats..DAY
end_timestringLatest date of usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration)None
start_timestringEarliest date of usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration)None
top_n_queriesintegerNumber of top queries to save to each table.10
include_operational_statsbooleanWhether to display operational stats.True
include_read_operational_statsbooleanWhether to report read operational stats. Experimental.False
format_sql_queriesbooleanWhether to format sql queriesFalse
include_top_n_queriesbooleanWhether to ingest the top_n_queries.True
envstringThe environment that all assets produced by this connector belong toPROD
platformstringThe platform that this source connects toNone
platform_instancestringThe instance of the platform that all assets produced by this recipe belong toNone
rate_limitbooleanShould we rate limit requests made to API.False
requests_per_minintegerUsed to control number of API calls made per min. Only used when rate_limit is set to True.60
temp_table_dataset_prefixstringIf you are creating temp tables in a dataset with a particular prefix you can use this config to set the prefix for the dataset. This is to support workflows from before bigquery's introduction of temp tables. By default we use _ because of datasets that begin with an underscore are hidden by default https://cloud.google.com/bigquery/docs/datasets#dataset-naming._
sharded_table_patternstringThe regex pattern to match sharded tables and group as one table. This is a very low level config parameter, only change if you know what you are doing,((.+)[_$])?(\d{4,10})$
projectsArray of stringList of project ids to ingest usage from. If not specified, will infer from environment.None
project_idstringProject ID to ingest usage from. If not specified, will infer from environment. Deprecated in favour of projectsNone
extra_client_optionsDictAdditional options to pass to google.cloud.logging_v2.client.Client.
use_v2_audit_metadatabooleanWhether to ingest logs using the v2 format. Required if use_exported_bigquery_audit_metadata is set to True.False
bigquery_audit_metadata_datasetsArray of stringA list of datasets that contain a table named cloudaudit_googleapis_com_data_access which contain BigQuery audit logs, specifically, those containing BigQueryAuditMetadata. It is recommended that the project of the dataset is also specified, for example, projectA.datasetB.None
use_exported_bigquery_audit_metadatabooleanWhen configured, use BigQueryAuditMetadata in bigquery_audit_metadata_datasets to compute usage information.False
use_date_sharded_audit_log_tablesbooleanWhether to read date sharded tables or time partitioned tables when extracting usage from exported audit logs.False
log_page_sizeinteger1000
query_log_delayintegerTo account for the possibility that the query event arrives after the read event in the audit logs, we wait for at least query_log_delay additional events to be processed before attempting to resolve BigQuery job information from the logs. If query_log_delay is None, it gets treated as an unlimited delay, which prioritizes correctness at the expense of memory usage.None
max_query_durationnumberCorrection to pad start_time and end_time with. For handling the case where the read happens within our time range but the query completion event is delayed and happens after the configured end time.900.0
user_email_patternAllowDenyPattern (see below for fields)regex patterns for user emails to filter in usage.{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
user_email_pattern.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
user_email_pattern.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
user_email_pattern.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
user_email_pattern.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
table_patternAllowDenyPattern (see below for fields)List of regex patterns for tables to include/exclude from ingestion.{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
table_pattern.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
table_pattern.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
table_pattern.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
table_pattern.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
dataset_patternAllowDenyPattern (see below for fields)List of regex patterns for datasets to include/exclude from ingestion.{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
dataset_pattern.allowArray of stringList of regex patterns for process groups to include in ingestion['.*']
dataset_pattern.denyArray of stringList of regex patterns for process groups to exclude from ingestion.[]
dataset_pattern.ignoreCasebooleanWhether to ignore case sensitivity during pattern matching.True
dataset_pattern.alphabetstringAllowed alphabets pattern[A-Za-z0-9 _.-]
credentialBigQueryCredential (see below for fields)Bigquery credential. Required if GOOGLE_APPLICATION_CREDENTIALS enviroment variable is not set. See this example recipe for details
credential.project_idstringProject id to set the credentialsNone
credential.private_key_idstringPrivate key idNone
credential.private_keystringPrivate key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n'None
credential.client_emailstringClient emailNone
credential.client_idstringClient IdNone
credential.auth_uristringAuthentication urihttps://accounts.google.com/o/oauth2/auth
credential.token_uristringToken urihttps://oauth2.googleapis.com/token
credential.auth_provider_x509_cert_urlstringAuth provider x509 certificate urlhttps://www.googleapis.com/oauth2/v1/certs
credential.typestringAuthentication typeservice_account
credential.client_x509_cert_urlstringIf not set it will be default to https://www.googleapis.com/robot/v1/metadata/x509/client_emailNone

Prerequisites

The Google Identity must have one of the following OAuth scopes granted to it:

And should be authorized on all projects you'd like to ingest usage stats from.

Compatibility

The source was last most recently confirmed compatible with the December 16, 2021 release of BigQuery.

Code Coordinates

  • Class Name: datahub.ingestion.source.usage.bigquery_usage.BigQueryUsageSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for BigQuery, feel free to ping us on our Slack