Configure Dataproc in Google Cloud

The following steps explain how to connect Tecton to Dataproc.

Confirm Tecton control plane service account

Your Tecton control plane service account is provided by your Tecton deployment specialist. It will look like tecton-<deployment>-control-plane@<tecton-deployment>.iam.gserviceaccount.com.

Create a VPC Network or reuse an existing VPC Network

This can be a new or existing network. Later, when developing features in Tecton, your Dataproc compute configuration will reference this network.

Connect Tecton to Dataproc

Grant the Tecton control plane service account access to manage Dataproc resources.

Tecton requires the following permissions on Dataproc:

compute.zones.get
compute.projects.get
dataproc.clusters.create
dataproc.clusters.delete
dataproc.clusters.get
dataproc.clusters.use
dataproc.jobs.create
dataproc.jobs.get
dataproc.jobs.list
dataproc.operations.cancel
dataproc.operations.get
dataproc.operations.list
dataproc.workflowTemplates.instantiateInline
resourcemanager.projects.get

Additionally, the Tecton control plane service account will need to be granted "Service Account User" on the Spark job service account.