SparkStreamConfig

Summary

Configuration used to define a stream source using a Data Source Function.
 
The SparkStreamConfig class is used to configure a stream source using a user defined Data Source Function.
 
This class is used as an input to a StreamSource's parameter stream_config. Declaring this configuration class alone will not register a Data Source. Instead, declare as a part of StreamSource that takes this configuration class instance as a parameter.

warning

Do not instantiate this class directly. Use spark_stream_config() instead.

Methods

__init__(...)

Instantiates a new SparkBatchConfig.

Parameters

• data_source_function (Callable[[pyspark.sql.session.SparkSession], pyspark.sql.dataframe.DataFrame]) - User-defined Data Source Function that takes in a SparkSession and returns a streaming DataFrame.

Returns

A SparkStreamConfig class instance.