PandasBatchConfig

Summary

Configuration used to define a batch source using a Pandas Data Source Function.

The PandasBatchConfig class is used to configure a batch source using a user defined Data Source Function.

This class is used as an input to a BatchSource’s parameter batch_config. Declaring this configuration class alone will not register a Data Source. Instead, declare as a part of BatchSource that takes this configuration class instance as a parameter.

This class is used as an input to a BatchSource's parameter batch_config. Declaring this configuration class alone will not register a Data Source. Instead, declare as a part of BatchSource that takes this configuration class instance as a parameter.

warning

Do not instantiate this class directly. Use pandas_batch_config() instead.

Attributes

• data_delay: This attribute is the same as the data_delay parameter of the __init__ method. See below.

Methods

__init__(...)

Instantiates a new PandasBatchConfig.

Parameters

• data_source_function (Union[Callable[[], pandas.DataFrame], Callable[[FilterContext], pandas.DataFrame]]) – User defined Data Source Function that takes in an optional FilterContext, if supports_time_filtering=True. Returns a pandas.DataFrame.

• data_delay (timedelta) – By default, incremental materialization jobs run immediately at the end of the batch schedule period. This parameter configures how long they wait after the end of the period before starting, typically to ensure that all data has landed. For example, if a feature view has a batch_schedule of 1 day and one of the data source inputs has a data_delay of 1 hour, then incremental materialization jobs will run at 01:00 UTC. (Default: datetime.timedelta(0))

• supports_time_filtering (bool) – Must be set to to True if one of the following conditions is met:

  • <data source>.get_dataframe() is called with start_time or end_time
  • A feature view wraps this Data Source with a FilteredSource

  If this parameter is set to true, Tecton passes a FilterContext object into the Data Source Function, which is expect to handle its own filtering. (Default: False)

• secrets Optional[Dict[str, Union[Secret, str]]]

  • A dictionary of Secret references that will be resolved and provided to the Data Source Function at runtime. During local development and testing, strings may be used instead Secret references.

Returns

A PandasBatchConfig class instance.