PowerBI

Module `powerbi`

Important Capabilities

Capability	Status	Notes
Extract Ownership	✅	Enabled by default

This plugin extracts the following:

Power BI dashboards, tiles, datasets
Names, descriptions and URLs of dashboard and tile
Owners of dashboards

Configuration Notes

See the

Microsoft AD App Creation doc for the steps to create a app client ID and secret.
Login to Power BI as Admin and from Tenant settings allow below permissions.
- Allow service principles to use Power BI APIs
- Allow service principals to use read-only Power BI admin APIs
- Enhance admin APIs responses with detailed metadata

Install the Plugin

pip install 'acryl-datahub[powerbi]'

Quickstart Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide

source:
  type: "powerbi"
  config:
    # Your Power BI tenant identifier
    tenant_id: a949d688-67c0-4bf1-a344-e939411c6c0a
    # Ingest elements of below PowerBi Workspace into Datahub
    workspace_id: 4bd10256-e999-45dd-8e56-571c77153a5f
    # Workspace's dataset environments (PROD, DEV, QA, STAGE)
    env: DEV
    # Azure AD App client identifier
    client_id: foo
    # Azure AD App client secret
    client_secret: bar
    # dataset_type_mapping is fixed mapping of Power BI datasources type to equivalent Datahub "data platform" dataset
    dataset_type_mapping:
        PostgreSql: postgres
        Oracle: oracle


sink:
  # sink configs

Config Details

Options
Schema

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options

Field	Required	Type	Description	Default
env		string	The environment that all assets produced by this connector belong to	PROD
tenant_id	✅	string	Power BI tenant identifier.	None
workspace_id	✅	string	Power BI workspace identifier.	None
dataset_type_mapping	✅	Dict[str,string]	Mapping of Power BI datasource type to Datahub dataset.
client_id	✅	string	Azure AD App client identifier.	None
client_secret	✅	string	Azure AD App client secret.	None
scan_timeout		integer	time in seconds to wait for Power BI metadata scan result.	60
scope		string		https://analysis.windows.net/powerbi/api/.default
base_url		string		https://api.powerbi.com/v1.0/myorg/groups
admin_base_url		string		https://api.powerbi.com/v1.0/myorg/admin
authority		string		https://login.microsoftonline.com/
platform_name		string		powerbi
platform_urn		string		urn:li:dataPlatform:powerbi
dashboard_pattern		AllowDenyPattern (see below for fields)		{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
dashboard_pattern.allow		Array of string	List of regex patterns for process groups to include in ingestion	['.*']
dashboard_pattern.deny		Array of string	List of regex patterns for process groups to exclude from ingestion.	[]
dashboard_pattern.ignoreCase		boolean	Whether to ignore case sensitivity during pattern matching.	True
dashboard_pattern.alphabet		string	Allowed alphabets pattern	[A-Za-z0-9 _.-]
chart_pattern		AllowDenyPattern (see below for fields)		{'allow': ['.*'], 'deny': [], 'ignoreCase': True, 'alphabet': '[A-Za-z0-9 _.-]'}
chart_pattern.allow		Array of string	List of regex patterns for process groups to include in ingestion	['.*']
chart_pattern.deny		Array of string	List of regex patterns for process groups to exclude from ingestion.	[]
chart_pattern.ignoreCase		boolean	Whether to ignore case sensitivity during pattern matching.	True
chart_pattern.alphabet		string	Allowed alphabets pattern	[A-Za-z0-9 _.-]

The JSONSchema for this configuration is inlined below.

{
  "title": "PowerBiDashboardSourceConfig",
  "description": "Any source that produces dataset urns in a single environment should inherit this class",
  "type": "object",
  "properties": {
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "tenant_id": {
      "title": "Tenant Id",
      "description": "Power BI tenant identifier.",
      "type": "string"
    },
    "workspace_id": {
      "title": "Workspace Id",
      "description": "Power BI workspace identifier.",
      "type": "string"
    },
    "dataset_type_mapping": {
      "title": "Dataset Type Mapping",
      "description": "Mapping of Power BI datasource type to Datahub dataset.",
      "type": "object",
      "additionalProperties": {
        "type": "string"
      }
    },
    "client_id": {
      "title": "Client Id",
      "description": "Azure AD App client identifier.",
      "type": "string"
    },
    "client_secret": {
      "title": "Client Secret",
      "description": "Azure AD App client secret.",
      "type": "string"
    },
    "scan_timeout": {
      "title": "Scan Timeout",
      "description": "time in seconds to wait for Power BI metadata scan result.",
      "default": 60,
      "type": "integer"
    },
    "scope": {
      "title": "Scope",
      "default": "https://analysis.windows.net/powerbi/api/.default",
      "type": "string"
    },
    "base_url": {
      "title": "Base Url",
      "default": "https://api.powerbi.com/v1.0/myorg/groups",
      "type": "string"
    },
    "admin_base_url": {
      "title": "Admin Base Url",
      "default": "https://api.powerbi.com/v1.0/myorg/admin",
      "type": "string"
    },
    "authority": {
      "title": "Authority",
      "default": "https://login.microsoftonline.com/",
      "type": "string"
    },
    "platform_name": {
      "title": "Platform Name",
      "default": "powerbi",
      "type": "string"
    },
    "platform_urn": {
      "title": "Platform Urn",
      "default": "urn:li:dataPlatform:powerbi",
      "type": "string"
    },
    "dashboard_pattern": {
      "title": "Dashboard Pattern",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true,
        "alphabet": "[A-Za-z0-9 _.-]"
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "chart_pattern": {
      "title": "Chart Pattern",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true,
        "alphabet": "[A-Za-z0-9 _.-]"
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    }
  },
  "required": [
    "tenant_id",
    "workspace_id",
    "dataset_type_mapping",
    "client_id",
    "client_secret"
  ],
  "additionalProperties": false,
  "definitions": {
    "AllowDenyPattern": {
      "title": "AllowDenyPattern",
      "description": "A class to store allow deny regexes",
      "type": "object",
      "properties": {
        "allow": {
          "title": "Allow",
          "description": "List of regex patterns for process groups to include in ingestion",
          "default": [
            ".*"
          ],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "deny": {
          "title": "Deny",
          "description": "List of regex patterns for process groups to exclude from ingestion.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "ignoreCase": {
          "title": "Ignorecase",
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "default": true,
          "type": "boolean"
        },
        "alphabet": {
          "title": "Alphabet",
          "description": "Allowed alphabets pattern",
          "default": "[A-Za-z0-9 _.-]",
          "type": "string"
        }
      },
      "additionalProperties": false
    }
  }
}

Concept mapping

Power BI	Datahub
`Dashboard`	`Dashboard`
`Dataset, Datasource`	`Dataset`
`Tile`	`Chart`
`Report.webUrl`	`Chart.externalUrl`
`Workspace`	`N/A`
`Report`	`N/A`

If Tile is created from report then Chart.externalUrl is set to Report.webUrl.

Code Coordinates

Class Name: datahub.ingestion.source.powerbi.PowerBiDashboardSource
Browse on GitHub

Questions

If you've got any questions on configuring ingestion for PowerBI, feel free to ping us on our Slack

Module powerbi​

Important Capabilities​

Configuration Notes​

Install the Plugin​

Quickstart Recipe​

Config Details​

Concept mapping​

Code Coordinates​

Questions​