Push Source is used to configure the Tecton Stream Ingest API for use with a
Stream Feature View.
A Push Source may also contain an optional
batch_config for efficiently
backfilling historial feature values.
from tecton import HiveConfig, PushSource, BatchSource
from tecton.types import Field, Int64, String, Timestamp
# Declare a schema for the Push Source
input_schema = [
# Declare a Push Source with a name, schema and a batch_config parameters
# See the API documentation for BatchConfig
click_event_source = PushSource(
description="Sample Push Source for click events",
|The time that this Tecton object was created or last updated.|
|Returns the duration that materialization jobs wait after the |
|The repo filename where this object was declared.|
|Returns the description of the Tecton object.|
|Returns the unique id of the Tecton object.|
|Returns the name of the Tecton object.|
|Returns the owner of the Tecton object.|
|Returns the tags of the Tecton object.|
|Returns the workspace that this Tecton object belongs to.|
|Creates a new Push Source.|
|Returns the column names of the data source’s push schema.|
|Returns the data in this Data Source as a Tecton DataFrame.|
|Displays a human readable summary of this Data Source.|
|Validate this Tecton object and its dependencies (if any).|
Creates a new Push Source.
str) – A unique name of the DataSource.
bool) – If True, this Tecton object will be blocked from being deleted or re-created (i.e. a destructive update) during tecton plan/apply. To remove or update this object, prevent_destroy must be first set to False via the same tecton apply or a separate tecton apply. prevent_destroy can be used to prevent accidental changes such as inadvertantly deleting a Feature Service used in production or recreating a Feature View that triggers expensive rematerialization jobs. prevent_destroy also blocks changes to dependent Tecton objects that would trigger a recreate of the tagged object, e.g. if prevent_destroy is set on a Feature Service, that will also prevent deletions or re-creates of Feature Views used in that service. prevent_destroy is only enforced in live (i.e. non-dev) workspaces. (Default:
Field]) – A schema for the Push Source
None]) – An optional BatchConfig object containing the configuration of the Batch Data Source that backs this Tecton Push Source. The Batch Source’s schema must contain a super-set of all the columns defined in the Push Source schema. (Default:
Returns the column names of the data source’s push schema.
Returns the data in this Data Source as a Tecton DataFrame.
datetime]) – The interval start time from when we want to retrieve source data. If no timezone is specified, will default to using UTC. Can only be defined if
apply_translatoris True. (Default:
datetime]) – The interval end time until when we want to retrieve source data. If no timezone is specified, will default to using UTC. Can only be defined if
apply_translatoris True. (Default:
bool) – If True, the transformation specified by
post_processorwill be applied to the dataframe for the data source.
apply_translatoris not applicable to batch sources configured with
spark_batch_configbecause it does not have a
A Tecton DataFrame containing the data source’s raw or translated source data.
TectonValidationError – If
apply_translator is False, but
end_time filters are passed in.
Displays a human readable summary of this Data Source.
Validate this Tecton object and its dependencies (if any).
Validation performs most of the same checks and operations as
Check for invalid object configurations, e.g. setting conflicting fields.
For Data Sources and Feature Views, test query code and derive schemas. e.g. test that a Data Source’s specified s3 path exists or that a Feature View’s SQL code executes and produces supported feature data types.
Objects already applied to Tecton do not need to be re-validated on retrieval
my_workspace.get_feature_view('my_fv')) since they have already been
Locally defined objects (e.g.
my_ds = BatchSource(name="my_ds", ...)) may need
to be validated before some of their methods can be called (e.g.