Configure Customer-managed keys for Encryption

Tecton always encrypts feature data and metadata, such as feature definitions. Tecton accounts can optionally be configured to use customer-managed keys for feature data and metadata encryption. By using customer-managed keys, a customer has the control to revoke access to this data from both Tecton and the Cloud Provider.

Customer-managed keys must be enabled during Tecton account creation as they cannot be enabled for existing accounts.

Configure Customer-managed keys with AWS KMS

The following steps illustrate how to configure Customer-managed keys for DynamoDB, S3, and the Tecton Control Plane.

1. Notify Tecton that you would like to use Customer-managed keys when creating your Tecton account.
2. Tecton will provide the AWS Account ID associated with your Tecton account.
3. Grant permissions for Tecton to use the key in the key policy. See instructions below for details.
4. Provide Tecton with the KMS key ARN.

The instructions below show how to configure permissions using Terraform or the AWS console.

Configure Key Policy in Terraform

1. The Tecton representative will provide the Tecton control plane AWS account ID

2. Note the ARN of the KMS key to be used

3. Add the KMS key ARN to the Tecton deployment module used to provision the data plane

   1. Add kms_key_id = <your_kms_key_arn> to match the appropriate KMS key ARN

   2. For example:

      locals {
        # Deployment name must be less than 22 characters (AWS limitation)
        deployment_name = "my-deployment-name"
      
        # The region and account_id of this Tecton account you just created
        region     = "my-region"
        account_id = "1234567890"
      
        # Get from your Tecton rep
        tecton_control_plane_root_principal = "arn:aws:iam::987654321:root"
      
        # get from your tecton rep
        cross_account_external_id = "tecton-external-id"
      
        key_kms_id = "arn:aws:kms:us-west-2:1234567890:key/aaaaaaa-bbbb-cccc-dddd-eeeeeeeee"
      
        # OPTIONAL for EMR notebook clusters in a different account (see optional block at end of file)
        # cross_account_arn = "arn:aws:iam::9876543210:root"
      }
      
      module "tecton" {
        source                     = "../deployment"
        deployment_name            = local.deployment_name
        account_id                 = local.account_id
        region                     = local.region
        cross_account_external_id  = local.cross_account_external_id
        kms_key_id                 = local.key_kms_id
      
        create_emr_roles = true
        s3_read_write_principals   = [local.tecton_control_plane_root_principal]
      }

4. Run terraform apply to apply the changes

5. Provide the KMS key ARN to the Tecton representative to complete the configuration process in the control plane

Configure Key Policy in the AWS console

1. Open AWS Console, go to Key Management Service page

   1. Click on the key that is to be used

2. Open the Key Policy tab, and click on Edit to open the key policy editor

3. Paste in the following JSON policy, replacing:

   1. ${ACCOUNT_ID} with the account ID of your Tecton Data Plane account
   2. ${CROSS_ACCOUNT_ROLE_NAME} with the name of the cross-account role.
   3. ${SPARK_ROLE_NAME} with the name of the Databricks/EMR IAM role. For Rift compute, please disregard this principal altogether.
   4. ${TECTON_CONTROL_PLANE_ACCOUNT} with the account ID of the Tecton Control Plane provided by the Tecton representative

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "Enable IAM User Permissions",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::${ACCOUNT_ID}:root"
         },
         "Action": "kms:*",
         "Resource": "*"
       },
       {
         "Sid": "Allow use of the key",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::${ACCOUNT_ID}:role/${CROSS_ACCOUNT_ROLE_NAME}",
             "arn:aws:iam::${ACCOUNT_ID}:role/${SPARK_ROLE_NAME}",
             "arn:aws:iam::${TECTON_CONTROL_PLANE_ACCOUNT}:root"
           ]
         },
         "Action": [
           "kms:Encrypt",
           "kms:Decrypt",
           "kms:ReEncrypt*",
           "kms:GenerateDataKey*",
           "kms:DescribeKey"
         ],
         "Resource": "*"
       },
       {
         "Sid": "Allow attachment of persistent resources",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::${ACCOUNT_ID}:role/${CROSS_ACCOUNT_ROLE_NAME}",
             "arn:aws:iam::${ACCOUNT_ID}:role/${SPARK_ROLE_NAME}",
             "arn:aws:iam::${TECTON_CONTROL_PLANE_ACCOUNT}:root"
           ]
         },
         "Action": ["kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant"],
         "Resource": "*",
         "Condition": {
           "Bool": {
             "kms:GrantIsForAWSResource": "true"
           }
         }
       }
     ]
   }

4. Provide the KMS key ARN to Tecton representative to complete the rest of the cluster deployment process

Other optional components

Databricks Cross-account Access

In the case where Databricks runs in a separate account from the Tecton data plane, the following steps illustrate how to grant the appropriate permissions to the Databricks workers.

1. If the Terraform method was used, pass in the Databricks account principal to var.kms_key_additional_principals variable in step 3.

   1. For example:

      locals {
        # Deployment name must be less than 22 characters (AWS limitation)
        deployment_name = "my-deployment-name"
      
        # The region and account_id of this Tecton account you just created
        region     = "my-region"
        account_id = "1234567890"
      
        # Get from your Tecton rep
        tecton_control_plane_root_principal = "arn:aws:iam::987654321:root"
      
        # get from your tecton rep
        cross_account_external_id = "tecton-external-id"
      
        key_kms_id = "arn:aws:kms:us-west-2:1234567890:key/aaaaaaa-bbbb-cccc-dddd-eeeeeeeee"
      
        # OPTIONAL for Databricks notebook clusters in a different account (see optional block at end of file)
        cross_account_arn = "arn:aws:iam::9876543210:root"
      }
      
      module "tecton" {
        source                     = "../deployment"
        deployment_name            = local.deployment_name
        account_id                 = local.account_id
        tecton_assuming_account_id = local.tecton_assuming_account_id
        region                     = local.region
        cross_account_external_id  = local.cross_account_external_id
        kms_key_id                 = local.key_kms_id
      
        create_emr_roles = true
        s3_read_write_principals   = [local.tecton_control_plane_root_principal]
      }

2. If the AWS method was used, add the Databricks account to the list of principals in step 3,

   1. With ${DATABRICKS_ACCOUNT_ID} being the account id of the Databricks account

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "Enable IAM User Permissions",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::${ACCOUNT_ID}:root"
         },
         "Action": "kms:*",
         "Resource": "*"
       },
       {
         "Sid": "Allow use of the key",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::${ACCOUNT_ID}:role/${CROSS_ACCOUNT_ROLE_NAME}",
             "arn:aws:iam::${ACCOUNT_ID}:role/${SPARK_ROLE_NAME}",
             "arn:aws:iam::${TECTON_CONTROL_PLANE_ACCOUNT}:root",
             "arn:aws:iam::${DATABRICKS_ACCOUNT_ID}:root"
           ]
         },
         "Action": [
           "kms:Encrypt",
           "kms:Decrypt",
           "kms:ReEncrypt*",
           "kms:GenerateDataKey*",
           "kms:DescribeKey"
         ],
         "Resource": "*"
       },
       {
         "Sid": "Allow attachment of persistent resources",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::${ACCOUNT_ID}:role/${CROSS_ACCOUNT_ROLE_NAME}",
             "arn:aws:iam::${ACCOUNT_ID}:role/${SPARK_ROLE_NAME}",
             "arn:aws:iam::${TECTON_CONTROL_PLANE_ACCOUNT}:root",
             "arn:aws:iam::${DATABRICKS_ACCOUNT_ID}:root"
           ]
         },
         "Action": ["kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant"],
         "Resource": "*",
         "Condition": {
           "Bool": {
             "kms:GrantIsForAWSResource": "true"
           }
         }
       }
     ]
   }

3. In the Databricks account, attach the following policy to the role used by Databricks instance profile,

   1. With ${KMS_KEY_ARN} being the ARN of your KMS key

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "Allow use of the key",
         "Effect": "Allow",
         "Principal": {
           "AWS": "${ROLE_ARN}"
         },
         "Action": [
           "kms:Encrypt",
           "kms:Decrypt",
           "kms:ReEncrypt*",
           "kms:GenerateDataKey*",
           "kms:DescribeKey",
           "kms:CreateGrant",
           "kms:ListGrants",
           "kms:RevokeGrant"
         ],
         "Resource": "${KMS_KEY_ARN}"
       }
     ]
   }

Redis

If you choose Redis as the online feature store, at-rest encryption must be enabled at replication group creation time. This means you must bring an Elasticache Redis cluster with at-rest encryption already enabled prior to connecting to Tecton.

See the Redis documentation for how to configure at-rest encryption.