Retry and Cancel Materialization
Automatic Retry Strategy
Tecton will automatically retry failing jobs with different policies based on the failure:
- Failures due to expected reasons such as an AWS spot instances being reclaimed are retried immediately.
- Otherwise, the job will be retried after 5 minutes, with exponential backoff for each successive failure.
Tecton will not retry jobs that fail for reasons that are expected to be permanent and that won't disappear by retrying.
Jobs that will be retried are shown in the Web UI as
Retrying (in X minutes).
If a job fails too many times, auto-retries will stop and jobs will be labeled
Failed in the Web UI. This happens after 100 attempts for spot
instance-related failures or 10 attempts for other failures.
Manually Retrying Failed Jobs
Materialized Jobs that fail repeatedly will eventually stop retrying. At this
stage they can be manually retried. When you are ready to retry the failed job
(e.g. after fixing the cause of the failures), trigger a manual retry of the job
by clicking the
Retry button in the Actions column of the failed job:
A notification will appear when the retry has been successfully scheduled. This happens immediately:
A retry job gets scheduled immediately:
If the scheduling of the retry fails, the notification will give you a job ID, and you can contact support with the message:
Manually Re-running Successful Jobs
Although a materialization job may have succeeded, you may run into scenarios where you need to re-run a specific time range for your feature view.
When re-running a materialization job that completed successfully, it’s possible to produce incorrect results in the Online Store. However, this operation is generally safe if one of the following are true:
Your previous job completed and did not output any feature data; or
Your Feature View is only materialized offline. (The Feature View is configured with
If you are unsure of the impact of re-running a materialization job that completed successfully, please contact Tecton Support before proceeding.
Trigger a re-run of the job by clicking the
Overwrite button in the Actions
column of the
A re-run job gets scheduled immediately:
Cancelling Running Jobs
You may cancel a
Running job or a job that's been continuously retrying by
Cancel button in the Actions column of the job.
The job gets marked
Manual Cancellation Requested immediately:
Within minutes the jobs will transition to
Manually Cancelled. You may need to
refresh the page to see this change. At this point, you will have the option to
Retry the job.