UnityConfig
Summary​
The UnityConfig class is used to create a reference to a Unity Table.Â
This class is used as an input to a
BatchSource
's parameter batch_config
. Declaring this configuration
class alone will not register a Data Source. Instead, declare as a part of BatchSource
that takes this configuration
class instance as a parameter.
Attributes​
The attributes are the same as the __init__
method parameters. See below.
Methods​
Name | Description |
---|---|
__init__(...) | Instantiates a new UnityConfig. |
__init__(...)​
Instantiates a new UnityConfig.Parameters
catalog
(str
) - A catalog registered in Unityschema
(str
) - A schema registered in Unitytable
(str
) - A table registered in Unitytimestamp_field
(Optional
[str
]) - The timestamp column in this data source that should be used byFilteredSource
to filter data from this source, before any feature view transformations are applied. Only required if this source is used withFilteredSource
. Default:None
timestamp_format
(Optional
[str
]) - Format of string-encoded timestamp column (e.g. "yyyy-MM-dd'T'hh:mm:ss.SSS'Z'"). If the timestamp string cannot be parsed with this format, Tecton will fallback and attempt to use the default timestamp parser. Default:None
datetime_partition_columns
(Optional
[List
[DatetimePartitionColumn
]]) - List of DatetimePartitionColumn the raw data is partitioned by, otherwise None. Default:None
post_processor
(Optional
[Callable
]) - Python user defined functionf(DataFrame) -> DataFrame
that takes in raw PySpark data source DataFrame and translates it to the DataFrame to be consumed by the Feature View. Default:None
data_delay
(timedelta
) - By default, incremental materialization jobs run immediately at the end of the batch schedule period. This parameter configures how long they wait after the end of the period before starting, typically to ensure that all data has landed. For example, if a feature view has abatch_schedule
of 1 day and one of the data source inputs hasdata_delay=timedelta(hours=1)
set, then incremental materialization jobs will run at01:00
UTC. Default:0:00:00
access_mode
(UnityCatalogAccessMode
) - The Unity Catalog Access Mode for the Unity table. If not specified, uses the Single User Access Mode as default. Default:None
Returns
A UnityConfig class instance.Example​
from tecton import BatchSource, UnityConfig
import pyspark
def convert_temperature(df: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame:
from pyspark.sql.functions import udf, col
from pyspark.sql.types import DoubleType
# Convert the incoming PySpark DataFrame temperature Celsius to Fahrenheit
udf_convert = udf(lambda x: x * 1.8 + 32.0, DoubleType())
converted_df = df.withColumn("Fahrenheit", udf_convert(col("Temperature"))).drop("Temperature")
return converted_df
# declare a BatchSource with UnityConfig
batch_source = BatchSource(
name="unity_batch_source",
batch_config=UnityConfig(
catalog="main",
schema="global_temperatures",
table="us_cities",
timestamp_field="timestamp",
post_processor=convert_temperature,
),
)