Migrating from Data Lake v1 to Data Lake v2

This guide walks through the recommended process for migrating from Data Lake v1 to Data Lake v2 in GRAX, including configuration steps, sequencing, and data retention considerations.

Overview

Data Lake v2 provides improved functionality and flexibility over Data Lake v1. The migration process is designed to allow you to enable Data Lake v2 alongside Data Lake v1, validate data, and then safely disable Data Lake v1 without data loss.

Migration Process

GRAX has already enabled Data Lake v2 for users currently on Data Lake v1 and ensured that all objects enabled in v1 are also enabled in v2.

Step 1: Monitor Initial Backfill

During the initial backfill, objects will show a Processing status.

Once the backfill is complete and the object status shows Current, Data Lake v2 is fully populated and up to date for that object.

Step 2: Prepare Downstream Processing

This step is critical to prevent data loss or processing gaps.

Before disabling Data Lake v1, ensure the following are complete for Data Lake v2:

• Any processing rules are enabled

• Any triggers or automations are configured

• Any pipelines, queries, or downstream consumers are updated to reference Data Lake v2 data

Step 3: Disable Objects in Data Lake v1

After confirming the following, you may safely disable the corresponding object in Data Lake v1:

• The object is Current in Data Lake v2, and

• You no longer need Data Lake v1 data for that object

The migration is not considered complete until all of the enabled objects in Data Lake v1 have been disabled.

Data Retention and Cleanup Considerations

• Data Lake v1 data remains in your storage bucket after disabling v1.

• GRAX recommends leaving v1 data in place for a period of time to support validation, rollback, or historical reference.

• You are free to delete Data Lake v1 data at any time once:

  • You have migrated objects to v2, and

  • Your pipelines and queries have been updated to use v2 data

When deleting, be sure to only delete files under parquet/org=X/... in your bucket. Do not delete files in other parts of the bucket.

Getting Help

If you have questions about your specific migration path or would like assistance validating your setup, please reach out to the GRAX Support team:

• Visit: https://documentation.grax.com/support/get-support

• Email: [email protected]

• Support is available to assist with migration issues