Integration Tests

Private Preview

This feature is currently in Private Preview.

An Integration Test Job simulates the materialization process for a Feature View so that developers can quickly test that their feature definitions will run successfully in the production environment.

These test jobs Integration Test (Batch) and Integration Test (Stream) are initiated as part of tecton plan --integration-test or tecton apply --integration-test. The jobs are configured with the same attributes and cluster configurations as the Feature View’s actual Materialization Jobs without committing data to the offline or online store. They will attempt to materialize data starting from the feature’s start time. While this data is not written to any store, the process closely mirrors the real materialization to ensure that everything functions as expected.

Using integration tests for feature development

When working with Tecton, the process typically begins by creating or iterating on data sources and features within a notebook and development workspace. Before updating the production environment, it is essential to validate your changes to ensure that the materialization jobs will execute successfully. This is where Integration Test Jobs become indispensable.

The command tecton plan/apply --integration-test allows for safe evaluation and observation of potential changes to the repository without committing them. If the tests are successful and the results are satisfactory, updates can be applied with confidence that the modifications will not disrupt the production environment.

The following commands are used to trigger these tests:

• tecton plan --integration-test: Runs the tests and shows the changes that would be made to the repo if the changes were applied.

• tecton apply --integration-test: Runs the tests and auto applies the repo if the tests pass.

What can Integration Tests help validate?

• Validating Feature View schemas
• Ensuring connectivity to new or updated upstream Data Sources
• Updating Cluster Configs (e.g. Spark Version; AMI; Bootstrap scripts)
• Updating tecton_materialization_runtime(for Spark) or environment (for Rift)
• Adding new dependencies to a materialization job
• Updating a Feature View transformation or adding a new Aggregation

Running Integration Tests with tecton plan

Testing New Or Updated Feature Views

When the command tecton plan --integration-test is executed, the Integration Test will then be triggered for the following scenarios.

1. All CREATE changes: When a Feature View is created with online or offline materialization enabled
2. All RECREATE changes: When a Feature View (with online or offline materialization enabled) is destructively updated.
   • e.g. Switching the offline store from Parquet to Delta
   • e.g. Updating a Feature View transformation or adding a new Aggregation
   • This includes RECREATE_UNLESS_SUPPRESSED changes**:** When a Feature View change is a RECREATE but is applied with --suppress-recreates
     1. These should be eligible for Integration Tests because users are sometimes confused about if --suppress-recreates is a safe operation. This would be an additional safety lever.
3. Some UPDATE changes: When a Feature View (with online or offline materialization enabled) is non-destructively updated. This includes changes such as:
   • Modifying batch_compute or stream_compute (incl. changes within Cluster Configs)
     • e.g. Upgrading the EMR or Databricks Runtime Version
     • e.g. Updating the AMI of a materialization cluster
     • e.g. Adding new dependencies to a materialization cluster
   • Modifying tecton_materialization_runtime
   • Modifying environments (for Rift-based Batch Feature Views)

  + Create Batch Feature View
    name:           user_transaction_metrics
    description:    User transaction metrics over 1, 3 and 7 days
    materialization: 11 backfills, 1 recurring batch job, 1 integration test
    > backfill:     10 Backfill jobs from 2022-12-25 00:00:00 UTC to 2024-08-02 00:00:00 UTC writing to the Offline Store
                    1 Backfill job from 2024-08-02 00:00:00 UTC to 2024-08-09 00:00:00 UTC writing to both the Online and Offline Store
    > incremental:  1 Recurring Batch job scheduled every 1 day writing to both the Online and Offline Store
    > integration test: 1 Integration test triggered for Batch job

↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
 Generated plan ID is ad7bbf597fe94359bf18dcf05b9742fc

You can track the test job status by tecton plan-info show {plan_id}

Plan Started At: 2024-08-09 00:20:12 UTC
Plan Created By: example@tecton.ai
Plan Applied: False
Integration Test Status: Running
Status by Feature View:
    user_transaction_metrics: Running
View your plan in the Web UI: https://example.tecton.ai/app/job-link

You can also view all the plans and their status by tecton plan-info list

Using workspace "demo" on cluster https://example.tecton.ai
            Plan Id                Plan Status   Test Status         Created by               Creation Date        SDK Version
==============================================================================================================================
695990c396214796bf24f436bf0d7208     Created       Running     example@tecton.ai   2024-08-26 22:37:22 UTC
19d0b1bc436443edbdeff717a0359bd4     Created       Failed      example@tecton.ai   2024-08-26 21:47:45 UTC

Detailed information about Integration Test jobs will be available on the Jobs page in the UI with type of Integration Test(Batch) or Integration Test(Stream).
When ready, you can apply the plan by tecton apply --plan-id {plan_id}

Manual Feature View Selection

To run integration tests on a specific subset of Feature Views, or to test a Feature View that is not modified in the current diff, the desired Feature Views can be explicitly specified in the command.
Tecton supports the following CLI options to specify which Feature Views to spin up Integration Tests for:

tecton plan --integration-test-select=":none:" # Default. No integration tests.
tecton plan --integration-test-select=":auto:" # Auto behavior
tecton plan --integration-test-select="fv:1,fv:2,fv3" # Comma-separated list of FVs
tecton plan --integration-test-select="pre1*,pre2*" # All FVs prefixed with "pre1" or "pre2".
tecton plan --integration-test-select="pre*suf" # All FVs prefixed with "pre" and suffixed with "suf".

Running Integration Tests with tecton-apply

With tecton apply --integration-test, Tecton will kick off Integration Test jobs and prompt the user to confirm if it’s ok to apply to the repo if these jobs succeed.
You can optionally use the --yes flag to automatically opt-in to applying to the workspace upon the success of Integration Tests.

$ tecton apply --integration-test
✅ Imported 1 Python module from the feature repository
⚠️  Running Tests: No tests found.
✅ Collecting local feature declarations
✅ Finished generating plan.
 ↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓

  ~ Recreate (destructive) Batch Feature View
    name:           auction_keywords
    owner:          example@tecton.ai
    serving_ttl:     1d -> 2d
    Integration tests: xxx

  ~ Update Stream Feature View
    name:           auction_keywords_stream
    owner:          example@tecton.ai
    tecton_materialization_runtime:     0.8.5 -> 0.9.1

 ↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑

Generated plan ID is fe7b7e8b48bd410b9352fb8d41b26553
View your plan in the Web UI: https://example.tecton.ai/app/prod/plan-summary/fe7b7e8b48bd410b9352fb8d41b26553

Do you want to apply to your workspace if Integration Tests are successful? y