Feast
There are 2 sources that provide integration with Feast
Source Module | Documentation |
| This plugin extracts:
|
| This plugin extracts the following:
Note: this uses a separate Docker container to extract Feast's metadata into a JSON file, which is then
parsed to DataHub's native objects. This separation was performed because of a dependency conflict in the |
Module feast
Important Capabilities
Capability | Status | Notes |
---|---|---|
Table-Level Lineage | ✅ | Enabled by default |
This plugin extracts:
- Entities as
MLPrimaryKey
- Features as
MLFeature
- Feature views and on-demand feature views as
MLFeatureTable
- Batch and stream source details as
Dataset
- Column types associated with each entity and feature
Install the Plugin
pip install 'acryl-datahub[feast]'
Quickstart Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide
source:
type: feast
config:
# Coordinates
core_url: "localhost:6565"
sink:
# sink configs
Config Details
- Options
- Schema
Note that a .
is used to denote nested fields in the YAML recipe.
View All Configuration Options
Field | Required | Type | Description | Default |
---|---|---|---|---|
path | ✅ | string | Path to Feast repository | None |
environment | string | Environment to use when constructing URNs | PROD |
The JSONSchema for this configuration is inlined below.
{
"title": "FeastRepositorySourceConfig",
"type": "object",
"properties": {
"path": {
"title": "Path",
"description": "Path to Feast repository",
"type": "string"
},
"environment": {
"title": "Environment",
"description": "Environment to use when constructing URNs",
"default": "PROD",
"type": "string"
}
},
"required": [
"path"
],
"additionalProperties": false
}
Code Coordinates
- Class Name:
datahub.ingestion.source.feast.FeastRepositorySource
- Browse on GitHub
Module feast-legacy
Important Capabilities
Capability | Status | Notes |
---|---|---|
Table-Level Lineage | ✅ | Enabled by default |
This plugin extracts the following:
- List of feature tables (modeled as
MLFeatureTable
s), features (MLFeature
s), and entities (MLPrimaryKey
s) - Column types associated with each feature and entity
Note: this uses a separate Docker container to extract Feast's metadata into a JSON file, which is then
parsed to DataHub's native objects. This separation was performed because of a dependency conflict in the feast
module.
Install the Plugin
The feast-legacy
source works out of the box with acryl-datahub
.
Config Details
- Options
- Schema
Note that a .
is used to denote nested fields in the YAML recipe.
View All Configuration Options
Field | Required | Type | Description | Default |
---|---|---|---|---|
env | string | The environment that all assets produced by this connector belong to | PROD | |
core_url | string | URL of Feast Core instance. | localhost:6565 | |
use_local_build | boolean | Whether to build Feast ingestion Docker image locally. | False |
The JSONSchema for this configuration is inlined below.
{
"title": "FeastConfig",
"description": "Any source that produces dataset urns in a single environment should inherit this class",
"type": "object",
"properties": {
"env": {
"title": "Env",
"description": "The environment that all assets produced by this connector belong to",
"default": "PROD",
"type": "string"
},
"core_url": {
"title": "Core Url",
"description": "URL of Feast Core instance.",
"default": "localhost:6565",
"type": "string"
},
"use_local_build": {
"title": "Use Local Build",
"description": "Whether to build Feast ingestion Docker image locally.",
"default": false,
"type": "boolean"
}
},
"additionalProperties": false
}
Code Coordinates
- Class Name:
datahub.ingestion.source.feast_legacy.FeastSource
- Browse on GitHub
Questions
If you've got any questions on configuring ingestion for Feast, feel free to ping us on our Slack