tecton.KafkaDSConfig

class tecton.KafkaDSConfig(kafka_bootstrap_servers, topics, raw_stream_translator, timestamp_key, default_watermark_delay_threshold=None, options=None, ssl_keystore_location=None, ssl_keystore_password_secret_id=None)

Configuration used to reference a Kafka stream.

The KafkaDSConfig class is used to create a reference to a Kafka stream.

This class used as an input to a VirtualDataSource’s parameter batch_config. This class is not a Tecton Primitive: it is a grouping of parameters. Declaring this class alone will not register a data source. Instead, declare a VirtualDataSource that takes this configuration class as an input.

Methods

__init__

Instantiates a new KafkaDSConfig.

__init__(kafka_bootstrap_servers, topics, raw_stream_translator, timestamp_key, default_watermark_delay_threshold=None, options=None, ssl_keystore_location=None, ssl_keystore_password_secret_id=None)

Instantiates a new KafkaDSConfig.

Parameters
  • kafka_bootstrap_servers (str) – A comma-separated list of the Kafka bootstrap server addresses. Passed directly to the Spark kafka.bootstrap.servers option.

  • topics (str) – A comma-separated list of Kafka topics to subscribe to. Passed directly to the Spark subscribe option.

  • raw_stream_translator – Python user defined function f(DF) -> DF that takes in raw Pyspark data source DataFrame and translates it to the DF to be consumed by the Feature Package. See an example of raw_stream_translator in the User Guide.

  • timestamp_key (str) – Name of the column containing timestamp for watermarking.

  • default_watermark_delay_threshold (Optional[str]) – (Optional) Watermark time interval, e.g: “30 minutes”

  • options – (Optional) A map of additional Spark readStream options

  • ssl_keystore_location (Optional[str]) – An S3 URI that points to the keystore file that should be used for SSL brokers example: s3://tecton-ai-${cluster_name}-intermediate-data/kafka_client_keystore.jks

  • ssl_keystore_password_secret_id (Optional[str]) – The config key for the password for the Keystore. Should start with SECRET_, example: SECRET_KAFKA_PRODUCTION.

Returns

A KafkaDSConfig class instance.