SparkBatchConfig

Summary

Configuration used to define a batch source using a Data Source Function.
 
The SparkBatchConfig class is used to configure a batch source using a user defined Data Source Function.
 
This class is used as an input to a BatchSource's parameter batch_config. Declaring this configuration class alone will not register a Data Source. Instead, declare as a part of BatchSource that takes this configuration class instance as a parameter.

warning

Do not instantiate this class directly. Use spark_batch_config() instead.

Methods

__init__(...)

Instantiates a new SparkBatchConfig.

Parameters

• data_source_function (Union[Callable[[pyspark.sql.session.SparkSession], pyspark.sql.dataframe.DataFrame],Callable[[pyspark.sql.session.SparkSession, tecton_core.filter_context.FilterContext], pyspark.sql.dataframe.DataFrame]]) - User defined Data Source Function that takes in a SparkSession and an optional tecton.FilterContext, if supports_time_filtering=True. Returns a DataFrame.


• data_delay (timedelta) - By default, incremental materialization jobs run immediately at the end of the batch schedule period. This parameter configures how long they wait after the end of the period before starting, typically to ensure that all data has landed. For example, if a feature view has a batch_schedule of 1 day and one of the data source inputs has a data_delay of 1 hour, then incremental materialization jobs will run at 01:00 UTC. Default: 0:00:00


• supports_time_filtering (bool) - When set to True, the Data Source Function must take the filter_context parameter and implement time filtering logic. supports_time_filtering must be set to True if <data source>.get_dataframe() is called with start_time or end_time. supports_time_filtering must also be set to True if using tecton.declarative.FilteredSource with a Data Source when defining a FeatureView. The FeatureView will call the Data Source Function with the tecton.FilterContext, which has the start_time and end_time set. Default: false

Returns

A SparkBatchConfig class instance.