Migrating from Data Lake v1 to Data Lake v2

This guide walks through the recommended process for migrating from Data Lake v1 to Data Lake v2 in GRAX, including configuration steps, sequencing, and data retention considerations.

Overview

Data Lake v2 provides improved functionality and flexibility over Data Lake v1. The migration process is designed to allow you to enable Data Lake v2 alongside Data Lake v1, validate data, and then safely disable Data Lake v1 without data loss.

Enable and Configure Data Lake v2

• Navigate to the Data Lake section of your GRAX application

• Select Data Lake v2

• Begin configuring the objects you want to write to Data Lake v2

Once enabled, GRAX will begin performing an initial backfill for the selected objects.

Monitor Initial Backfill

• During backfill, objects will show a processing status

• Once the backfill is complete and the object status shows Current, Data Lake v2 is fully populated and up to date for that object

Prepare Downstream Processing

Before disabling Data Lake v1, ensure the following are complete for Data Lake v2:

• Any processing rules are enabled

• Any triggers or automations are configured

• Any pipelines, queries, or downstream consumers are updated to reference Data Lake v2 data

This step is critical to prevent data loss or processing gaps.

Disable Objects in Data Lake v1

After confirming the following, you may safely disable the corresponding object in Data Lake v1:

• The object is Current in Data Lake v2, and

• You no longer need Data Lake v1 data for that object

The migration is not considered complete until all of the enabled objects in Data Lake v1 have been disabled.

Data Retention and Cleanup Considerations

• Data Lake v1 data remains in your storage bucket after disabling v1.

• GRAX recommends leaving v1 data in place for a period of time to support validation, rollback, or historical reference.

• You are free to delete Data Lake v1 data at any time once:

  • You have migrated objects to v2, and

  • Your pipelines and queries have been updated to use v2 data.

After Data Lake v1 is fully deprecated, GRAX will provide additional cleanup recommendations if necessary.

Getting Help

If you have questions about your specific migration path or would like assistance validating your setup, please reach out to the GRAX Support team:

• Visit: https://documentation.grax.com/support/get-support

• Email: [email protected]

• Support is available to assist with migration issues