Skip to content

Upgrading Notebook Clusters to Spark 3

This guide includes instructions for updating existing Notebook clusters from Spark 2 (Databricks 6.4, EMR 5.3) to Spark 3.

Upgrading Databricks Notebooks

Databricks is ending support for Spark 2 in December, so Tecton now supports DBR 9.1 LTS. If you cannot use this runtime, please contact Tecton support.

Clone your existing Notebook cluster

  1. In the Databricks UI, go to Clusters -> your current Tecton Notebook Cluster -> Clone
  2. Select Runtime: 9.1 LTS
  3. Create your new cluster.

Install Libraries and Jars in the new cluster

In the Cluster configuration page, go to the Libraries tab.

Install tecton package from PyPi

  1. Click Install New
  2. Select PyPI under Library Source
  3. Set Package to tecton

In your notebook, you can now attach to the new cluster.

Install Tecton UDFs jar

  1. Click Install New
  2. Select DBFS/S3 under Library Source
  3. Set File Path to s3://tecton.ai.public/pip-repository/itorgation/tecton/tecton-udfs-spark-3.jar

Upgrading EMR Notebooks

Tecton now supports EMR 6.4.0. If you cannot use this runtime, please contact Tecton support.

  1. In the EMR UI, go to Clusters -> your Tecton Notebook Cluster -> Clone
  2. Change EMR ami to 6.4.0
  3. Select the checkbox next to Jupyter Enterprise Gateway
  4. Change bootstrap script location from s3://tecton-staging-emr-scripts/install_emr_notebook_libraries.sh to s3://tecton-staging-emr-scripts/install_emr_notebook_libraries_v2.sh