UnityConfig

Summary

The UnityConfig class is used to create a reference to a Unity Table.
 
This class is used as an input to a BatchSource's parameter batch_config. Declaring this configuration class alone will not register a Data Source. Instead, declare as a part of BatchSource that takes this configuration class instance as a parameter.

Attributes

The attributes are the same as the __init__ method parameters. See below.

Methods

Name	Description
__init__(...)	Instantiates a new UnityConfig.

__init__(...)

Instantiates a new UnityConfig.

Parameters

• catalog: str A catalog registered in Unity
• schema: str A schema registered in Unity
• table: str A table registered in Unity
• timestamp_field: Optional[str] = None The timestamp column in this data source that should be used for time-based filtering. Required unless this source is used in Feature Views only with unfiltered().
• timestamp_format: Optional[str] = None Format of string-encoded timestamp column (e.g. "yyyy-MM-dd'T'hh:mm:ss.SSS'Z'"). If the timestamp string cannot be parsed with this format, Tecton will fallback and attempt to use the default timestamp parser.
• datetime_partition_columns: Optional[List[DatetimePartitionColumn]] = None List of DatetimePartitionColumn the raw data is partitioned by, otherwise None.
• post_processor: Optional[Callable] = None Python user defined function f(DataFrame) -> DataFrame that takes in raw PySpark data source DataFrame and translates it to the DataFrame to be consumed by the Feature View.
• data_delay: timedelta = 0:00:00 By default, incremental materialization jobs run immediately at the end of the batch schedule period. This parameter configures how long they wait after the end of the period before starting, typically to ensure that all data has landed. For example, if a feature view has a batch_schedule of 1 day and one of the data source inputs has data_delay=timedelta(hours=1) set, then incremental materialization jobs will run at 01:00 UTC.
• access_mode: UnityCatalogAccessMode = None The Unity Catalog Access Mode for the Unity table. If not specified, uses the Single User Access Mode as default.

Returns

A UnityConfig class instance.

Example

from tecton import BatchSource, UnityConfig
import pyspark


def convert_temperature(df: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame:
    from pyspark.sql.functions import udf, col
    from pyspark.sql.types import DoubleType

    # Convert the incoming PySpark DataFrame temperature Celsius to Fahrenheit
    udf_convert = udf(lambda x: x * 1.8 + 32.0, DoubleType())
    converted_df = df.withColumn("Fahrenheit", udf_convert(col("Temperature"))).drop("Temperature")
    return converted_df


# declare a BatchSource with UnityConfig
batch_source = BatchSource(
    name="unity_batch_source",
    batch_config=UnityConfig(
        catalog="main",
        schema="global_temperatures",
        table="us_cities",
        timestamp_field="timestamp",
        post_processor=convert_temperature,
    ),
)