Types of Materialization Alerts
Tecton provides multiple types of alerts for Feature Views. When alerts are triggered, an email is sent to the address specified in the alert_email
parameter of the MonitoringConfig
object.
There are different types of alerts for each type of materialization issue:
- Freshness Alerts
FeatureViewNotFresh
- Repeated Failures Alerts
FeatureViewBatchMaterializationFailures
FeatureViewStreamingMaterializationFailures
- Too Many Failures Alerts
FeatureViewTooManyFailures
Freshness Alerts
Feature data becomes more stale over time when new features are not being materialized. Tecton triggers a FeatureViewNotFresh
when feature data becomes too stale, which is a threshold set by the expected_feature_freshness
parameter. If this parameter is not specified, the alert threshold defaults to twice the aggregation_slide_period
for Stream Window Aggregate Feature Views or twice the batch_schedule
for other Feature Views.
The most common causes of this type of alert are:
- Missing upstream data
- Errors in feature definitions that cause materialization jobs to either fail or produce zero feature values
- An outage causes materialization jobs to fail
You can increase expected_feature_freshness
to a longer time interval based on the expected freshness of your features. See Monitoring Materializations for more information.
Repeated Failures Alerts
Tecton automatically schedules retries for failing materialization jobs. If these retries fail consistently, Tecton will trigger an alert. There are two types of repeated failure alerts:
FeatureViewBatchMaterializationFailures
. Batch materialization jobs have failed 2 or more times.FeatureViewStreamingMaterializationFailures
. Streaming materialization jobs have failed 2 or more times.
Materialization jobs that need to be retried due to spot instance availability are not considered failures. Most often, this error is caused by errors in feature definitions.
Too Many Failures Alerts
When materialization retries fail too many times, Tecton will move the Feature View to a "Too Many Failures" state and will not continue to retry materialization.
At this point, the FeatureViewTooManyFailures
alert will be fired. This alert is most commonly caused by incorrect Transformation code.