Troubleshooting Materialization Jobs
Understanding Tecton Materialization Job Failures​
This topic explains the different types of failures that can occur with Tecton materialization jobs and how to troubleshoot them. Materialization jobs are responsible for generating historical and online feature data for Feature Views.
Types of Failures​
MANUAL_CANCELLATION: This indicates that the underlying compute job was stopped unexpectedly by a user while it was still running. This could be due to a mistake, or because the job was taking longer than expected.
NON_CLOUD_FAILURE: This type of failure happens when the job encounters an issue that prevents it from completing successfully, but the issue is not related to problems with the cloud infrastructure itself (e.g., spot instance failures). This could be caused by a variety of factors, such as:
- Errors in feature definitions (e.g., syntax errors, incorrect data access)
- Issues with compute configuration (e.g., missing permissions, incompatible dependencies)
- Issues with data pipeline (e.g., missing data, data quality problems)
- Resource limitations (e.g., insufficient memory, timeout errors)
- Bugs in the Tecton code itself (less likely, but still possible)
INSTANCE_ALLOCATION_FAILURE: Instances could not be allocated for the cluster. Often resulted from spot instances not being available in the AZ(s).
Troubleshooting Tips​
Review the job logs: The job logs contain detailed information about the failure, which can help pinpoint the cause. Access the job logs through the Tecton user interface.
Check the Feature View definition: Ensure there are no errors in the feature definitions and that the data sources are accessible.
Investigate the data pipeline: Confirm the data pipeline is functioning correctly and delivering the expected data to Tecton.
Monitor resource utilization: If resource limitations are suspected, monitor the cluster's resource utilization during job execution.
Contact Tecton support: If the issue remains unresolved, contact Tecton support for further assistance.