Configuration used to reference a Redshift table or query.
The RedshiftConfig class is used to create a reference to a Redshift table. You can also create a reference to a query on one or more tables, which will be registered in Tecton in a similar way as a view is registered in other data systems.
This class used as an input to a
batch_config. This class is not a Tecton Object: it is a grouping of
parameters. Declaring this class alone will not register a data source. Instead,
declare as part of
BatchSource that takes this configuration class instance as
data_delay: This attribute is the same as the
data_delayparameter of the
__init__method. See below.
Instantiates a new RedshiftConfig. One of table and query should be specified when creating this file.
str) – The connection endpoint to Redshift (e.g. redshift-cluster-1.cigcwzsdltjs.us-west-2.redshift.amazonaws.com:5439/dev).
Callable]) – Python user defined function
f(DataFrame) -> DataFramethat takes in raw PySpark data source DataFrame and translates it to the DataFrame to be consumed by the Feature View. (Default:
str]) – The timestamp column in this data source that should be used by FilteredSource to filter data from this source, before any feature view transformations are applied. Only required if this source is used with FilteredSource. (Default:
timedelta) – By default, incremental materialization jobs run immediately at the end of the batch schedule period. This parameter configures how long they wait after the end of the period before starting, typically to ensure that all data has landed. For example, if a feature view has a batch_schedule of 1 day and one of the data source inputs has
data_delay=timedelta(hours=1)set, then incremental materialization jobs will run at 01:00 UTC. (Default:
A RedshiftConfig class instance.
from tecton import RedshiftConfig
# Declare RedshiftConfig instance object that can be used as an argument in BatchSource
redshift_ds_config = RedshiftConfig(
query="SELECT timestamp as ts, created, user_id, ad_id, duration FROM ad_serving_features",