Creating and Managing Datasets
This feature is not supported in Tecton on Snowflake Public Preview.
If you are interested in this functionality, please file a feature request.
Tecton Datasets allow for conveniently saving feature data that can be used for model training, experiment reproducibility, and analysis. Datasets are versioned alongside your Feature Store configuration, allowing you to inspect and restore the state of all features as of the time the dataset was created. Tecton Datasets can be created in two ways:
- Saving training DataFrames that are requested from Feature Services
- Logging online requests to a Feature Service
Saved Training DataFrames
Tracking feature training DataFrames as Tecton Datasets has a number advantages:
- Datasets are tracked and catalogued in one central place.
- Datasets are identified with a single string, which you can store alongside other model parameters in model metadata stores such as MLFlow.
- When you save a Dataset, Tecton stores both the data and the metadata associated with the features (eg, data sources, transformation logic) - allowing you to track the full lineage of a Dataset.
To save a Dataset for later retrieval, use the
save parameter of
get_historical_features. If this parameter is supplied, Tecton eagerly
computes the DataFrame and stores it for later retrieval alongside the metadata
used to generate the Dataset. To give the Dataset a name, use the
Defining a Saved Training DataFrame Dataset
In the code example below, create a Dataset by providing the
spine = pd.DataFrame([...], columns=[...])
my_fs = tecton.get_feature_service("ctr_prediction_service")
save flags are provided to
Tecton automatically stores the metadata of the Dataset alongside the feature
DataFrame for later retrieval.
After a Dataset is defined, it is available in the Web UI.
Logged Online Requests
Feature Services have the ability to continuously log online requests and feature vector responses as Tecton Datasets. These logged feature datasets can be used for auditing, analysis, training dataset generation.
To enable feature logging on a FeatureService, simply add a LoggingConfig like
in the example below and optionally specify a sample rate. Then run
tecton apply to apply your changes.
from tecton import LoggingConfig
ctr_prediction_service = FeatureService(
description="A FeatureService used for supporting a CTR prediction model.",
Within 60 seconds, this will create a new Tecton Dataset under the Datasets tab
in the Web UI. This dataset will continue having new feature logs appended to it
every 30 mins. If the features in the Feature Service change, a new dataset
version will be created. Datasets are named with the following convention:
<Feature Service name>.logged_requests.<Version>. The Dataset with the highest
version number for a Feature Service will be the latest active dataset.
Interacting with Datasets
Datasets can be fetched by name using the code snippet below:
my_training_data = tecton.get_dataset("my_training_data")
# View my_dataset as a Pandas DataFrame
Using Dataset Spines
All Tecton Datasets contain a reference to their "spine DataFrame". This spine contains the join keys and request data used to generate feature vectors.
If the Dataset was saved during training data generation, then this spine was passed to Tecton in order to generate the feature DataFrame. In the case of a logged requests Dataset, this spine is the accumulated list of online requests to the Feature Service.
To fetch a Dataset's spine DataFrame, run the following code in a notebook:
dataset_spine = tecton.get_dataset("my_training_data").get_spine_dataframe()
This spine can be used as input to reproduce a Dataset from scratch, or test out new features.