Data Lake Inconsistency

Incident Overview

On July 26, 2023, GRAX identified a low severity issue with our GRAX Data Lake product which only affected customers doing heavy backfilling of Data Lake data. Once identified by GRAX Support, our technical teams promptly issued software updates and recommended fixes (as of July 27, 2023).

This incident only affected select users of Data Lake and did not affect users of GRAX Backup and Restore or any other GRAX products.

If you have seen inconsistent data in Data Lake, or if you believe you have been affected, we recommend reprocessing those objects through GRAX Data Lake. To reprocess objects please contact our technical support team for assistance.

Description

Data Lake speed and throughput changes introduced a potential issue where reading some versions of records from backup data could be missed when writing to Data Lake. As a result, some Data Lake users may not have all versions of their records in Data Lake, resulting in inconsistent data downstream from Data Lake.

This could manifest in some records downstream from Data Lake being inconsistent until another modification in Salesforce, including some records deleted in Salesforce never showing up as deleted downstream from Data Lake.

However all users have retained 100% of all versions of all records backed up securely by GRAX. And all Data Lake users have the ability to reprocess data from backups if additional data consistency is needed.

Root Cause Analysis

Initial findings suggest that the root cause was due to changes made to improve the speed and throughput of Data Lake around June 26, 2023. This change created a data synchronization issue in environments doing heavy backfilling, where GRAX Backup continually maintained 100% of all versions of records, but Data Lake may have experienced occasional data consistency issues.

Incident Resolution

As of July 27, 2023, we have issued an update that addressed the gap in data synchronization.

If you have been affected, or if you require the highest guarantees of data consistency in Data Lake for downstream consumption, we recommend reprocessing those objects through GRAX Data Lake. To reprocess those objects please contact our technical support team for assistance.

Frequently Asked Questions

How do I know if I’m affected?

If you recently enabled Data Lake and it has been doing heavy backfilling, it is likely you are affected. If you enabled Data Lake prior to June 26, 2023, and updates are incremental, it is likely you are not affected.

Did data loss occur due to this incident?

Zero data loss occurred. 100% of all your versions of records were captured into your own storage using GRAX Auto Backups. You will only notice that some versions of record data were skipped from being written to Data Lake between June 26 and July 27, 2023. Because the data is in Auto Backups it can be reprocessed to Data Lake. Reprocessing data is a common operation in data pipelines.

What preventative measures have been introduced to stop this from happening again?

Our teams have configured Data Lake speeds to allow for a slightly longer processing time (~30 minutes) following a data backup. Data backups will continue to run according to your plan.

Furthermore, we are committed to adding additional long term safety improvements to Data Lake data.

Do I have to take corrective action?

No action is required on your part if the data captured in Data Lake is sufficient for your downstream consumption needs. You have already received the automated update to improve consistency of records in Data Lake going forward.

Please describe the process for obtaining 100% of all versions of records in Data Lake.

Please contact technical support, and we will work with you to reprocess your objects in Data Lake. The team will help reset any affected Data Lake objects, which will purge and reprocess the affected data.