Retry and Cancel Materialization
Automatic Retry Strategyβ
Tecton will automatically retry failing jobs with different policies based on the failure:
- Failures due to expected reasons such as an AWS spot instances being reclaimed are retried immediately.
- Otherwise, the job will be retried after 5 minutes, with exponential backoff for each successive failure.
Tecton will not retry jobs that fail for reasons that are expected to be permanent and that won't disappear by retrying.
Jobs that will be retried are shown in the Web UI as Retrying (in X minutes)
.
If a job fails too many times, auto-retries will stop and jobs will be labeled
Failed
in the Web UI. This happens after 100 attempts for spot
instance-related failures or 10 attempts for other failures.
Manually Retrying Failed Jobsβ
Materialized Jobs that fail repeatedly will eventually stop retrying. At this
stage they can be manually retried. When you are ready to retry the failed job
(e.g. after fixing the cause of the failures), trigger a manual retry of the job
by clicking the Retry
button in the Actions column of the failed job:
A notification will appear when the retry has been successfully scheduled. This happens immediately:
A retry job gets scheduled immediately:
If the scheduling of the retry fails, the notification will give you a job ID, and you can contact support with the message:
Manually Re-running Successful Jobsβ
Although a materialization job may have succeeded, you may run into scenarios where you need to re-run a specific time range for your feature view.
When re-running a materialization job that completed successfully, itβs possible to produce incorrect results in the Online Store. However, this operation is generally safe if one of the following are true:
-
Your previous job completed and did not output any feature data; or
-
Your Feature View is only materialized offline. (The Feature View is configured with
offline=True
andonline=False
).
If you are unsure of the impact of re-running a materialization job that completed successfully, please contact Tecton Support before proceeding.
Trigger a re-run of the job by clicking the Overwrite
button in the Actions
column of the Success
job:
A re-run job gets scheduled immediately:
Cancelling Running Jobsβ
You may cancel a Running
job or a job that's been continuously retrying by
clicking the Cancel
button in the Actions column of the job.
The job gets marked Manual Cancellation Requested
immediately:
Within minutes the jobs will transition to Manually Cancelled
. You may need to
refresh the page to see this change. At this point, you will have the option to
manually Retry
the job.