Troubleshooting
Last updated
Was this helpful?
Last updated
Was this helpful?
This document includes detailed instructions and frequently asked questions to help you troubleshoot your GRAX Application. Click on a link below to be directed to the section of your choice:
The record count you see in the org setup under "Storage Usage" is an estimate. Same for the .
To get an actual count of records from Salesforce you need to issue a COUNT SOQL query, though note it may timeout when there are a lot of records. In this case you may want to issue multiple queries counting records in different time ranges, or use the to export records IDs and count them from a file instead.
Just like Salesforce, GRAX stores estimated record counts. This allows this data to be displayed, grouped, and segmented quickly on demand. These estimates are displayed in the Auto Backup and Archive dashboards as well as in GRAX Insights. They can also be queried programmatically via the .
Please note there is a margin of error of approximately 2% for these numbers. They can become outdated if you reset Auto Backup; if this is the case, contact us.
If you need a precise count of records, use .
GRAX OAuth access the first time they login?Yes. By default the GRAX OAuth connected app is configured to allow all users to self-authorize and start the OAuth handshake. This means that when a user logs into GRAX for the first time, they will be prompted to allow identity service access to the GRAX OAuth connected app. If you would like to avoid this prompt, you can configure the GRAX OAuth connected app to only allow access to users the admin has pre-authorized. To enable pre-authorized users only, follow the steps:
In Salesforce, open Setup
In the Quick Find box, search for OAuth and select Connected Apps OAuth Usage
Find the GRAX OAuth connected app
If there is an Install button under Actions, click it to complete the installation and allow Manage App Policies access
Click the Manage App Policies link
Click the Edit Policies button
Change the Permitted Users setting to Admin approved users are pre-authorized
You will see the following warning message appear: Enabling this option will result in all users currently using this app being denied access. Please reference the Connected Apps OAuth Usage Report if you are unsure who is using the app. This means users who have self-authorized won't have access to the app until the steps below have been completed. Once you understand the warning, click the OK button
Upon clicking Save you'll be directed to the GRAX Connected App Detail page. Scroll down the Profiles section and click Manage Profiles
Choose all of the profiles that you want to be pre-authorized and click Save
Login to your SFDC environment
In a separate tab navigate directly to your GRAX Application URL and append /web to the end and select Sign-in with Salesforce or Connect with Auto Config - if you don't know this URL, you can also navigate to the GRAX Application by clicking on the Schedule tab within the GRAX managed package
Click on the Allow button on the Allow Access pop up window if it appears (there are some orgs that do not require this step)
These steps should give you access back to GRAX. Once you have access, double check the Salesforce connection within GRAX (in the Settings section) and that the Integration User is connected to confirm the change.
The Hyperforce API is the same as the Salesforce API. From a developer's perspective, this means that you can continue to use the Salesforce API to build and customize your applications, and the API remains the same regardless of whether you are running your Salesforce applications on Hyperforce or on Salesforce's own infrastructure.
Therefore GRAX works with Hyperforce deployments with no changes.
If you do use:
Hyperforce
GRAX LWC in Apex (non-iframe) mode
Self-managed GRAX deployment
Your AWS deployment has a Web Application Firewall (WAF), which may block the LWC API requests coming from your Hyperforce Apex servers.
No. A Salesforce instance refresh/migration occurs when SFDC upgrades the infrastructure supporting your instance in their data centers. Following this maintenance, your instance will move to a new data center, and the name of your instance will change.
Prior to an instance refresh/maintenance, SFDC will provide customers a date/timeline when the refresh will occur, specifying a maintenance window where their instance will be down. GRAX will be available during this SFDC maintenance period but GRAX will not be able to perform any backup or archive operations until the SFDC org is back up and available.
GRAX uses OAuth to access SFDC, so you may need to reauthorize the integration. Once the instance refresh/migration is complete, you’ll want to re-authenticate the GRAX integration user by navigating to the GRAX Application and logging in with the integration user.
After the integration User has been authenticated, GRAX will resume all activities (backup, archive, etc.) from the point where the service was disrupted.
Salesforce provides various encryption options for data at rest and in transit, such as Shield Platform Encryption and Transport Layer Security (TLS). These encryption options are designed to protect data from unauthorized access and breaches, but they do not affect the functionality of third-party applications like GRAX. You can encrypt certain fields on standard and custom objects, data in Chatter, and search index files. With some exceptions, encrypted fields work normally throughout the Salesforce user interface, business processes, and APIs.
GRAX operates through the Salesforce API, which allows it to access data regardless of whether it is encrypted or not. GRAX creates backups of your data, and these backups are stored in a separate, encrypted data store. When you need to restore data, GRAX decrypts the backup and restores it to Salesforce.
Helpful Links
Yes. Your company may have Salesforce orgs that you no longer want to maintain after a consolidation, merger, or business reorganization. GRAX makes it easy to protect this data, search, and reuse it without paying Salesforce to keep the org active. After disconnecting from Salesforce, the backup of your data remains safe with GRAX and accessible through powerful tools including Global Search. Record data can be exported to CSV and Data Lake may be used to send a copy of data to your downstream systems such as your data lake or analytics tools.
First, you will need a GRAX license. Please contact your account manager or GRAX Sales.
Now you can safely decommission your Salesforce Org.
Note that any changes to data in the Salesforce org made after disconnecting GRAX will no longer be captured by GRAX Auto Backup.
CANNOT_EXECUTE_FLOW_TRIGGER or CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY mean?Your Salesforce org has a trigger interfering with record creation or deletion.
Apex CPU time limit exceeded
Example: CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY: EmailMessageTrigger: System.LimitException: Apex CPU time limit exceeded
The Apex code in this trigger is taking too long to run on a record and failing with a System.LimitException error due to a SFDC platform time limit. The Salesforce platform enforces that Apex code from triggers must run within a short amount of time to prevent excessive resource consumption. Possible options to resolve this are:
Use the Disable automations option in the seeding job to disable the trigger for the duration of the seeding job (Sandbox Seeding only)
Modify the trigger to not run for the GRAX integration user
Optimize the trigger code to run in less time
General Trigger Execution
Example: CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY: TaskTrigger: execution of AfterInsert
The code in the trigger is blocking the operation based on how the object or related objects are currently configured. Possible options to resolve this are:
Add an override to the job to set the field value that will allow the record to pass the trigger validation rule (Sandbox Seeding or Restore)
Use the Disable automations option in the job to disable the trigger for the duration of the seeding job (Sandbox Seeding only)
Modify the trigger to not run for the GRAX integration user
Review the trigger code to allow the operation to succeed
DUPLICATE_VALUE or DUPLICATES_DETECTED mean?Example: DUPLICATE_VALUE: A topic with the name, why-GRAX-is-amazing, already exists.
Salesforce requires unique values for certain fields (such as topic names) and GRAX has tried to insert a duplicate.
For Sandbox Seeding, this can happen when re-seeding a dataset without undoing the previous seed first. Undo the previous seed that included this record or select a different dataset to seed.
For Restore, this can happen if a record you are trying to restore was previously un-deleted from the Salesforce Recycle bin. To ensure data integrity and records are properly linked, always restore records from GRAX.
ENTITY_IS_DELETED mean?Example: ENTITY_IS_DELETED: entity is deleted
The record was deleted in Salesforce since the last backup was taken. Wait for the next Auto Backup to complete and try the activity again.
FIELD_CUSTOM_VALIDATION_EXCEPTION mean?Example: FIELD_CUSTOM_VALIDATION_EXCEPTION: Activity can't be created for Closed enquiry. Please Re Open Enquiry to save Activity.
The FIELD_CUSTOM_VALIDATION_EXCEPTION error indicates that there is a custom validation rule or trigger on the object that is blocking the record from being created. Review the message details to determine the cause, possible options to resolve this are:
Add an override to the seeding job to set the field value that will allow the record to pass the validation rule (Sandbox Seeding or Restore)
Use the Disable automations option in the seeding job to disable the trigger for the duration of the seeding job (Sandbox Seeding only)
Modify the validation rule / trigger to not run for the GRAX integration user
Skip the object
INACTIVE_OWNER_OR_USER mean?Example: INACTIVE_OWNER_OR_USER: The specified user is inactive
INSUFFICIENT_ACCESS_OR_READONLY mean?Can't create a link for Email Message when it's not in draft state
Example: INSUFFICIENT_ACCESS_OR_READONLY: You can't create a link for Email Message when it's not in draft state (field=LinkedEntityId)
When a ContentDocumentLink attached to an EmailMessage is deleted, there are some guards that prevent us from placing it back. Salesforce only allows links to emails that are either in state draft, or created with IsClientManaged set to true.
If none of these apply, our recommended workaround is to archive the email message and then restore it using GRAX. In this process we're going to set that IsClientManaged flag, and then restore ContentDocumentLinks related to it.
Insufficient access rights on object id or record type
Example: INSUFFICIENT_ACCESS_OR_READONLY: insufficient access rights on object id or INSUFFICIENT_ACCESS_OR_READONLY: invalid record type
The GRAX integration user does not have sufficient permissions to delete / create this object or record type. Please review the permissions for the GRAX integration user / GRAX integration user permission set and ensure it has correct object and record type access.
Invalid sharing type
Example: INSUFFICIENT_ACCESS_OR_READONLY: Invalid sharing type V (field=ShareType)
You are trying to seed or restore a Content Document (ContentVersion + ContentDocumentLink) and you are getting an "Invalid sharing type V" error message. To resolve it, you need to configure file sharing to Set by Record in your Salesforce org settings:
Go to Setup and enter Files in the Quick Find box.
Under Salesforce Files, choose General Settings.
Enable the Set by Record option for files attached to records.
Data Validation rules
There is a validation rule that is blocking the record from being created as it currently exists.
Review the message details to determine the cause and options to resolve by setting an override (such as setting the state field to Draft for the example above)
Skip the object.
Insufficient permissions
The GRAX integration user does not have sufficient permissions to create the record. Review the permissions for the GRAX integration user / GRAX integration user permission set and ensure it has access to create the records for that object
INVALID_CROSS_REFERENCE_KEY or INSUFFICIENT_ACCESS_ON_CROSS_REFERENCE_ENTITY mean?Example: INVALID_CROSS_REFERENCE_KEY: Missing reference: field TaskId ref ID: 00T46000008US01AAG
When Salesforce needs to do a many-to-many link between two objects (such as Accounts can have many tasks, task can be linked to many accounts), it uses a junction object called a cross-reference. This error means that the cross-reference GRAX is trying to create is missing or unable to access one side of the junction. This most commonly has two likely causes:
Missing reference - One of the 2 records that are being linked is missing:
Check to see if there are other errors in the job that may indicate the record was skipped
The record was manually excluded
The record was not included in the job due to a filter such as max children
this ID value isn't valid for the user or You do not have the level of access necessary - The GRAX integration user doesn't have sufficient permissions to create the cross-reference record type. Check the permissions for the GRAX integration user / GRAX integration user permission set and ensure it has access to create the cross-reference record type.
INVALID_OR_NULL_FOR_RESTRICTED_PICKLIST mean?Example: INVALID_OR_NULL_FOR_RESTRICTED_PICKLIST: bad value for restricted picklist field: Direct_Traffic (field=originalSource)
The picklist value GRAX is trying to set is not a valid value for the field. This can happen if the option was deleted since the record was backed up or in Sandbox seeding if the option was added to the source org after the target org was created. Possible options to resolve this are:
Add the missing picklist value to the picklist options for the field
Add an override to the job to set the field value to a valid picklist option
STORAGE_LIMIT_EXCEEDED mean?Example: STORAGE_LIMIT_EXCEEDED: storage limit exceeded
Your Salesforce target org does not have sufficient storage space to load the data you are trying to seed. GRAX provides an estimated size for all records vs the available space reported by the seeding target. Since these are estimates, its possible to exceed the available space when the data is actually loaded. Please edit the seeding job and reduce the amount of data being seeded, some options are:
Reduce file size by skipping the ContentDocument or Attachment objects
Skip objects that are not meaningful to the use cases for the seeded data. For example running a test on Opportunity data doesn't require EmailMessages
Clicking on the Object in the graph or list will show you the estimated size of the records for that object. You can use this to determine which objects are consuming the most space and use this info to decide which objects to skip
UNABLE_TO_LOCK_ROW mean?Example: UNABLE_TO_LOCK_ROW: unable to obtain exclusive access to this record or 10 records: ...
Salesforce is attempting to update a separate record based on the changes to the record that reported this error while a different record in the same batch is also updating that record. This can happen as a result of a roll-up summary field or a trigger, such as rolling up the number of opportunities on an account. If you're seeing this error, it means the Salesforce and GRAX ROW LOCK resolution processes were unable to separate the child records that are blocking each other.
Review the trigger logic to ensure it can properly handle batches of records (and doesn't assume only one record will be updated at a time)
For Sandbox Seeding, reduce the number of records being seeded
For Restore, retry the restore job for the failed records
UNKNOWN_EXCEPTION mean?UNKNOWN_EXCEPTION is a catch-all for errors that Salesforce does not provide a specific error code for. The error message usually has a general description of the cause of the error. Search the internet or contact Salesforce support to help identify the root cause. Based on the cause of the error, it may be possible to correct it by setting an override or skipping the object.
This section covers issues blocking general operations of GRAX including networking failures, boot failures, restart behaviors, etc. This guide includes commands specific to the Amazon Linux 2 AMI maintained by AWS, but attempts to remain otherwise infrastructure-agnostic where possible. Some steps may not work as intended if you have made heavy customizations to networking, image, or environment.
First, let's note the locations of the GRAX binary and environment file. Keeping track of their paths helps validate service configurations in later steps. In a typical installation, GRAX is stored under /home/ec2-user/graxinc/grax and the .env file is stored under /home/ec2-user. Thus, we'll use the following paths:
GRAX binary: /home/ec2-user/graxinc/grax/grax
GRAX command-line tools: /home/ec2-user/graxinc/grax/graxctl
Environment file: /home/ec2-user/.env
To ensure that Linux knows GRAX is an executable, we can check permissions on the file as follows:
The "x" in the permissions strings at the beginning of each line denotes an executable file. If the grax and graxctl files aren't executable, we can mark them as such:
There are several important rules to remember for .env files:
Only one key-value pair per line
Only = is supported as a key-value separator
Comments aren't supported
Comments are the most commonly seen issue as teams often attempt to label values for later reference. Unfortunately, this causes most .env parsers to immediately return (sometimes non-fatally). This can lead to partial configurations and thus cause indeterminate symptoms.
This guide assumes you're operating GRAX as a permanent service on the instance with systemd. The most common issues with systemd are configuration issues in the service file. In a typical installation, the GRAX service file is at the path /lib/systemd/system/grax.service.
Validate Configuration
We can see the contents of the service configuration by using cat:
Check the following:
EnvironmentFile is a valid absolute path that points to your GRAX .env file.
ExecStart is a valid absolute path that points to your GRAX executable.
Restart is "always" to ensure GRAX is always running regardless of exit-singaling.
Service Status
The services run via systemd are managed and interacted with via the systemctl command. To see the current status of the GRAX service, we can use the status subcommand:
We always expect the GRAX service to be "active" unless under maintenance or GRAX was intentionally taken offline. If the service isn't "active" (that is "failed" or "stopped"), you can restart GRAX at any time by running the restart subcommand:
If the GRAX service is entirely disabled, you can enable it with the enable subcommand, and then enforce a start immediately with start:
A successful start of the service (and thus app) outputs logs in the app log file. If you have a regular health check configured, you'll see logs in relation to those calls being submitted to the log if the app is active.
GRAX is a web server and API. It offers an endpoint for an external health check to see if the app is available. The health check endpoint for GRAX is an HTTP/1.1 HTTPS-only GET handler on /health. In a typical installation, GRAX runs on port 8000.
We can manually check the status of the app from the instance by curling the local route:
The expected value from the endpoint HTTP status 200; this signifies a healthy service. A failed call, either via timeout, rejection, or different status is a sign of a failed service/app. This endpoint is designed for load balancer registration and de-registration, not for instance replacement.
The GRAX Application is a data-processor at its core. To process data, it must be able to retrieve that data, write it to storage, and read it back. When you add app maintenance, licensing, and telemetry to the equation, connectivity is critical to ensure proper operation.
Only some pieces of overall connectivity requirements are possible to test from the instance. These are the egress connections that are used to push or pull data to/from remote resources. Ingress communications, as they start from other sources, are harder to test.
Timeouts, rejections, or broken connections during the following tests are considered failures. All failures should be investigated.
GRAX HQ
Communication to GRAX HQ is egress-only, and can be tested relatively simply. To start, we can verify connectivity to the GRAX packaging API, which allows downloading of the app in the first place:
The -L flag is utilized to follow the ALB redirect that points to HQ. -o is used to avoid printing the binary data to the terminal. A successful download of several dozen MB can be considered a passing test.
Next, let's confirm POST requests to HQ succeed:
This may seem an unusual result, but the 401 return is a good enough response to know that your POST request made it to HQ without requiring you construct a valid set of credentials for a simple test. If you don't get a JSON response in line with the above, consider the test a failure.
Salesforce
To read and write Salesforce data via the Salesforce API, GRAX must first be able to connect. We can test that connectivity much the same as above:
The response to the above should be an HTML document, too large to post here. Repeat that test for the following:
https://login.salesforce.com
Any custom/my-domain paths configured in your organization
Postgres
To test connectivity to your DB instance, we use postgresql, a Linux command-line tool that allows direct interaction with Postgres clusters. Installing the tool may be unnecessary depending on image, but can easily be done like the following:
As you can see, our typical installation already includes the right tooling. We can connect in two ways:
Copy the DATABASE_URL value from your .env file, and run psql [database_url]
Use the graxctl psql subcommand
A valid connection results in an interactive psql shell, which can be exited with \q:
If a connection cannot be made, graxctl tries again every 5 seconds for a few minutes. This is usually a sign of the following:
DATABASE_URL isn't set
DATABASE_URL isn't properly formatted
DATABASE_URL contains invalid cluster name
DATABASE_URL contains a password with special characters that need to be escaped
DATABASE_URL isn't exported to current environment
Route tables are forcing DB traffic outside of the VPC
Security groups aren't allowing traffic from the Instance into the DB
If a connection can be made but you receive a Postgres error about DB existence, credentials, etc., then you likely have an issue with correctness in your DATABASE_URL value (that is username, password, or DB name).
No. There should be no effect on any GRAX functionality due to the latest Salesforce release. You can read more about this release . Additionally, the RFC 7230 Validation for Apex RestResponse Headers that were enforced in Spring '24 will also not impact GRAX.
No. The Cookies Having Independent Partitioned State (CHIPS) change that Google will be making will not have any effect on your GRAX connection in Salesforce or your GRAX Application; you can read more about this .
Yes. meet the latest browser requirements and are now being used in all Salesforce orgs. As long as you're using OAuth to login to your Salesforce environments, the Enhanced Domain Enforcement impact to GRAX is minimal. After making the domain change, you'll need to take the following steps:
Yes. is a new architecture that allows customers to run Salesforce applications on public cloud infrastructure providers such as Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure. Hyperforce was announced by Salesforce in December 2020 as part of its Dreamforce event.
In this case, Salesforce publishes which will need to be allowed to communicate with your GRAX Application.
This configuration is uncommon, and a simpler solution is to use the .
Please review for additional information.
No. Salesforce Platform Events don't support all objects needed by GRAX and are VERY limiting (when backing up ALL objects). Currently, captures , (Attachment, Content, Chatter Files, etc), and Salesforce system-tables. Data and binary files MUST be captured at the same rate/time or you risk damaging referential integrity or completeness of data. To fulfill our customers backup needs Salesforce Platform Events aren't an option due to incomplete object support. If you have questions please reach out to the .
Yes. should not impact your ability to use GRAX, as GRAX is designed to work seamlessly with Salesforce's native encryption.
Try out Shield Platform Encryption at no charge in a .
Yes. routes SFDC traffic via Salesforce-managed public cloud VPCs instead of letting egress traffic cross the public internet. This is a network-layer configuration; if configured correctly, the GRAX Application won't be able to tell the difference between a public or private connection. GRAX isn't responsible for configuring or maintaining Private Connect. Private Connect requires additional Salesforce licensing. Private Connect is only available for self-managed GRAX deployments.
No. For orgs created after Winter 2020, the is enabled by default. The transition to the ICU Locale format will not impact your GRAX Application.
Next, you will need to set up a GRAX deployment and connect a storage bucket as described in the .
Then, you will using an account that has the Salesforce Administrator role and run GRAX Auto Config. This will assign the permission sets for the GRAX Application to operate.
Once the App is connected, log in with Salesforce OAuth and enable Auto Backup if it is not yet running. Auto Backup will automatically run as it protects all of your data. Note: There can be a backlog of data to synchronize when you initially connect to a Salesforce Org, and you will notice the noted as "backfilling" as this process is catching up.
Once Auto Backup is complete, you can disconnect your Salesforce org, but first you will need to create a local user account via email and password to log in as GRAX will no longer have access to Salesforce OAuth to provide authentication. This can be done under Settings -> User Management in the GRAX Application. Your password link will be sent via email. Note that will need to enable this feature on your org.
The GRAX integration user does not have sufficient permissions to create the records for inactive users. Please review the permissions for the GRAX integration user / GRAX integration user permission set and ensure it has the Set Inactive Owners permission. See the page.
Additional is available from Salesforce.
For a total example of a valid .env file, see our .
If you have exhausted the steps here and require further assistance (or have recommendations for quality/completeness of this guide), contact .