Skip to main content
Version: 0.6

Data Quality Validations

Public Preview

This feature is currently in Public Preview.

This feature has the following limitations:
  • Available for Tecton on Databricks and EMR. Coming to Tecton on Snowflake in a future release.
If you have questions or want to share feedback, please file a feature request.

Data Quality Validations help detect feature data issues once a Feature View has been materialized. If validation results indicate that feature data failed to meet expectations during a materialization interval, an alert email will be sent to an email provided as alert_email in the Feature View declaration.

Terminology​

  • Data Quality Metrics are statistics that describe feature values output by a Feature View during materialization. See Data Quality Metrics for more information.
  • Expectations are verifiable assertions about metrics. Expectations can be based on metrics. For example, “Expect that <100% of values for a given feature are null”.
  • Validations are the process of validating that the set of expectations has been met when materializing a Feature View. Validations can either pass or fail.
  • Alerts notify the specified user when validation fails.

This document covers Data Quality Expectations, Validations, and Alerts.

Default Expectations​

By default, Tecton defines the following expectations for all Batch and Stream Feature Views.

caution

For Stream Feature Views, Data Quality Metrics and Expectations only apply to offline materialized feature data.

ExpectationApplicable toExplanation
Feature View row count > 0Feature ViewsExpect feature rows to be produced when a Feature View is materialized
A feature has any non-null valuesAll types of featuresExpect a feature to have at least one non-null value, when there are feature rows.
A feature has any non-zero valuesNumerical featuresExpect a feature to have at least one non-zero value, when there are feature rows.
A feature has any non-empty valuesString or Array featuresExpect a feature to have at least one non-empty-string/array value when there are feature rows.

Enable Validations and Alert Emails​

Validation can be disabled per Feature View, by setting skip_default_expectations=True in a Feature View declaration (requires Tecton SDK version >= 0.6.14).

Email Alerting is enabled when alert_email is specified in a Batch or Stream Feature View definition. The alert email will be sent out at most once in 6 hours per Feature View. If you would like to disable all email alerts for a Feature View, including other types of materialization alerts, leave this field unset.

Viewing Validation Results​

You can view the validation results for all Feature Views in a workspace by selecting Data Quality in the left navigation panel in Tecton web UI.

Screenshot of Data Quality Validation Nav

Was this page helpful?

🧠 Hi! Ask me anything about Tecton!

Floating button icon