History Stream Inconsistency

Incident Overview

On July 26, 2023, GRAX identified a low severity issue with our GRAX History Stream product which only affected customers doing heavy backfilling of History Stream data. Once identified by GRAX Support, our technical teams promptly issued software updates and recommended fixes (as of July 27, 2023).

This incident only affected select users of History Stream and did not affect users of GRAX Backup and Restore or any other GRAX products.

If you have seen inconsistent data in History Stream, or if you believe you have been affected, we recommend reprocessing those objects through GRAX History Stream. To reprocess objects please contact our technical support team for assistance.

Description

History Stream speed and throughput changes introduced a potential issue where reading some versions of records from backup data could be missed when writing to History Stream. As a result, some History Stream users may not have all versions of their records in History Stream, resulting in inconsistent data downstream from History stream.

This could manifest in some records downstream from History Stream being inconsistent until another modification in Salesforce, including some records deleted in Salesforce never showing up as deleted downstream from History Stream.

However all users have retained 100% of all versions of all records backed up securely by GRAX. And all History Stream users have the ability to reprocess data from backups if additional data consistency is needed.

Root Cause Analysis

Initial findings suggest that the root cause was due to changes made to improve the speed and throughput of History Stream around June 26, 2023. This change created a data synchronization issue in environments doing heavy backfilling, where GRAX Backup continually maintained 100% of all versions of records, but History Stream may have experienced occasional data consistency issues.

Incident Resolution

As of July 27, 2023, we have issued an update that addressed the gap in data synchronization.

If you have been affected, or if you require the highest guarantees of data consistency in History Stream for downstream consumption, we recommend reprocessing those objects through GRAX History Stream. To reprocess those objects please contact our technical support team for assistance.

Frequently Asked Questions

How do I know if I’m affected?

If you recently enabled history stream and it has been doing heavy backfilling, it is likely you are affected. If you enabled History Stream prior to June 26, 2023, and updates are incremental, it is likely you are not affected.

Did data loss occur due to this incident?

Zero data loss occurred. 100% of all your versions of records were captured into your own storage using GRAX Auto Backups. You will only notice that some versions of record data were skipped from being written to History Stream between June 26 and July 27, 2023. Because the data is in Auto Backups it can be reprocessed to History Stream. Reprocessing data is a common operation in data pipelines.

What preventative measures have been introduced to stop this from happening again?

Our teams have configured History Stream speeds to allow for a slightly longer processing time (~30 minutes) following a data backup. Data backups will continue to run according to your plan.

Furthermore, we are committed to adding additional long term safety improvements to History Stream data.

Do I have to take corrective action?

No action is required on your part if the data captured in History Stream is sufficient for your downstream consumption needs. You have already received the automated update to improve consistency of records in History Stream going forward.

Please describe the process for obtaining 100% of all versions of records in History Stream.

Please contact technical support, and we will work with you to reprocess your objects in History Stream. The team will help reset any affected History Stream objects, which will purge and reprocess the affected data.