SparkStreamConfig
Summary​
Configuration used to define a stream source using a Data Source Function.Â
The
SparkStreamConfig
class is used to configure a stream source using a user defined Data Source Function.Â
This class is used as an input to a
StreamSource
's parameter stream_config
. Declaring this configuration
class alone will not register a Data Source. Instead, declare as a part of StreamSource
that takes this configuration
class instance as a parameter.Â
Do not instantiate this class directly. Use :
tecton.spark_stream_config
instead.
warning
Do not instantiate this class directly. Use
spark_stream_config()
instead.
Methods​
__init__(...)​
Instantiates a new SparkBatchConfig.Parameters
data_source_function
(Callable
[[pyspark.sql.session.SparkSession
],pyspark.sql.dataframe.DataFrame
]) - User-defined Data Source Function that takes in aSparkSession
and returns a streamingDataFrame
.