Skip to main content
Version: Beta 🚧
Private Preview

This feature is currently in Private Preview.

This feature has the following limitations:
  • Must be enabled by Tecton Support.
If you would like to participate in the preview, please file a feature request.

Data Compaction

In Performance and Costs of Aggregation Features we introduced the high-level architecture of Tecton's Aggregation Engine:

Steps in the Tecton Data Pipeline

With Data Compaction, Tecton significantly optimizes the performance of aggregation features. Underlying this optimization are data compaction processes that Tecton automates and runs behind the scenes.

As a result, Tecton users will automatically see the following benefits:

  • Low Latency Serving for large time windows: Users will observe extremely fast online retrieval times for aggregations - even in cases when aggregation time windows are very long, or the number of events in a fixed time window is very high (>> 100,000)
  • Optimized Online Storage Efficiency: Users will see even fewer online store writes during backfills, reducing the maintenance burden and cost of the online store (see documentation)
  • Batch Healing of Streaming Features: Streaming Features can automatically be corrected with batch data
note

This capability is not rolled out to all customers yet. If you are interested, please reach out to us directly.

Architecture Overview

Compaction

Conceptual Overview

The key innovation is to replace small, old tiles with fewer “compacted” tiles to reduce the amount of data processed at read time. The Tecton service takes care of piecing together the different data points while presenting a consistent and simple API to the consumer.

Compaction is performed by a periodic offline process that reads the event log, performs the partial aggregation, and updates the Online Store. At read-time, Tecton handles rolling up the final aggregation over varying partial aggregate “tile” sizes.

Batch Updates to the Online Store

On a pre-defined cadence – typically daily – Tecton will rebuild tiles in the Online Store based on data available in the Offline Store. A data processing job will read the offline data for the full aggregation window, perform partial aggregations at the optimal tile size, and update the Online Store for each key.

Combining Stream and Batch values

As new events arrive on the Stream, they continue to be written directly to the Online Store as well. When a query is sent to the Tecton Feature Server, Tecton reads both the ‘batch updated’ table and the ‘stream updated’ table and aggregates the final feature value. To avoid double counting, Tecton tracks the latest value written to the batch table and only reads stream events greater than that timestamp.

Lifetime Aggregates

The simplest scenario is a lifetime aggregation feature. In this case, the system never expires old events.

During the batch update process, Tecton will update a single tile per key that covers the full lifetime of the feature data. At read time, Tecton will combine the “big tile” with streaming events since the last batch update.

Lifetime Compaction

Enable Data Compaction

note

Data Compaction is only available for Batch Feature Views in 0.8 and must be first enabled by Tecton Support. Compaction for Stream Feature Views is coming soon.

To enable data compaction for Batch Feature View, you must set batch_compaction_enabled=True on your Feature View:

from tecton import batch_feature_view
from datetime import timedelta


@batch_feature_view(
sources=[transactions],
mode="spark_sql",
entities=[user],
feature_start_time=datetime(2022, 5, 1),
batch_schedule=timedelta(days=1),
online=True,
offline=True,
batch_compaction_enabled=True,
tecton_materialization_runtime="0.8.2",
)
def user_average_transaction_amount(transactions):
return f"""
SELECT user_id, timestamp, amount
FROM {transactions}
"""

Note that:

  • This requires offline materialization to be enabled (offline=True).
  • This requires the Offline Store format to be Delta (i.e. OfflineStoreConfig(staging_table_format=DeltaConfig())).
  • tecton_materialization_runtime must be set to '0.8.2' or greater.

Enable Data Compaction for an Existing Feature View

Tecton recommends that customers first test this capability on Feature Views in a staging environment before enabling it for existing production Feature Views. Note that enabling Data Compaction on an existing Feature View will require rematerialization of feature data — however, Tecton’s recent optimizations make backfills significantly cheaper.

Was this page helpful?

🧠 Hi! Ask me anything about Tecton!

Floating button icon