Feast

There are 2 sources that provide integration with Feast

Source Module	Documentation
feast	This plugin extracts: Entities as MLPrimaryKey Features as MLFeature Feature views and on-demand feature views as MLFeatureTable Batch and stream source details as Dataset Column types associated with each entity and feature Read more...
feast-legacy	This plugin extracts the following: List of feature tables (modeled as MLFeatureTables), features (MLFeatures), and entities (MLPrimaryKeys) Column types associated with each feature and entity Note: this uses a separate Docker container to extract Feast's metadata into a JSON file, which is then parsed to DataHub's native objects. This separation was performed because of a dependency conflict in the feast module. Read more...

Module feast

Important Capabilities

Capability	Status	Notes
Table-Level Lineage	✅	Enabled by default

This plugin extracts:

• Entities as MLPrimaryKey
• Features as MLFeature
• Feature views and on-demand feature views as MLFeatureTable
• Batch and stream source details as Dataset
• Column types associated with each entity and feature

Install the Plugin

pip install 'acryl-datahub[feast]'

Quickstart Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide

source:
  type: feast
  config:
    # Coordinates
    core_url: "localhost:6565"

sink:
  # sink configs

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options

Field	Required	Type	Description	Default
path	✅	string	Path to Feast repository	None
environment		string	Environment to use when constructing URNs	PROD

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "FeastRepositorySourceConfig",
  "type": "object",
  "properties": {
    "path": {
      "title": "Path",
      "description": "Path to Feast repository",
      "type": "string"
    },
    "environment": {
      "title": "Environment",
      "description": "Environment to use when constructing URNs",
      "default": "PROD",
      "type": "string"
    }
  },
  "required": [
    "path"
  ],
  "additionalProperties": false
}

Code Coordinates

• Class Name: datahub.ingestion.source.feast.FeastRepositorySource
• Browse on GitHub

Module feast-legacy

Important Capabilities

Capability	Status	Notes
Table-Level Lineage	✅	Enabled by default

This plugin extracts the following:

• List of feature tables (modeled as MLFeatureTables), features (MLFeatures), and entities (MLPrimaryKeys)
• Column types associated with each feature and entity

Note: this uses a separate Docker container to extract Feast's metadata into a JSON file, which is then parsed to DataHub's native objects. This separation was performed because of a dependency conflict in the feast module.

Install the Plugin

The feast-legacy source works out of the box with acryl-datahub.

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options

Field	Required	Type	Description	Default
env		string	The environment that all assets produced by this connector belong to	PROD
core_url		string	URL of Feast Core instance.	localhost:6565
use_local_build		boolean	Whether to build Feast ingestion Docker image locally.	False

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "FeastConfig",
  "description": "Any source that produces dataset urns in a single environment should inherit this class",
  "type": "object",
  "properties": {
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "core_url": {
      "title": "Core Url",
      "description": "URL of Feast Core instance.",
      "default": "localhost:6565",
      "type": "string"
    },
    "use_local_build": {
      "title": "Use Local Build",
      "description": "Whether to build Feast ingestion Docker image locally.",
      "default": false,
      "type": "boolean"
    }
  },
  "additionalProperties": false
}

Code Coordinates

• Class Name: datahub.ingestion.source.feast_legacy.FeastSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Feast, feel free to ping us on our Slack