# Troubleshooting This document includes detailed instructions and frequently asked questions to help you troubleshoot your GRAX Application. Click on a link below to be directed to the section of your choice: * [Frequently Asked Salesforce Questions](#frequently-asked-salesforce-questions) * [Common Salesforce Errors](#common-salesforce-errors) * [Troubleshooting Network and Infrastructure Issues](#troubleshooting-network-and-infrastructure-issues) ## Frequently Asked Salesforce Questions #### Why is the record count I see in GRAX different than the one in Salesforce? The record count you see in the org setup under "Storage Usage" is an estimate. Same for the [Record Count platform API](https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/resources_record_count.htm). To get an actual count of records from Salesforce you need to issue a `COUNT` SOQL query, though note it may timeout when there are a lot of records. In this case you may want to issue multiple queries counting records in different time ranges, or use the [Data Loader](https://help.salesforce.com/s/articleView?id=sf.exporting_data.htm\&type=5) to export records IDs and count them from a file instead. #### How can I see how many records GRAX has in backup? What about how many records were archived? Just like Salesforce, GRAX stores estimated record counts. This allows this data to be displayed, grouped, and segmented quickly on demand. These estimates are displayed in the Backup and Archive dashboards as well as in GRAX Insights. They can also be queried programmatically via the [Estimated Record Counts API](https://api.grax.com/#tag/records/GET/api/v2/record-counts). Please note there is a margin of error of approximately 2% for these numbers. They can become outdated if you reset Backup; if this is the case, contact us. If you need a precise count of records, use [Global Search](https://documentation.grax.com/reuse-data/global-search). #### Does the Salesforce Winter '25 Release Affect the GRAX Application? No. There should be no effect on any GRAX functionality due to the latest Salesforce release. You can read more about this release [here](https://help.salesforce.com/s/articleView?id=release-notes.salesforce_release_notes.htm\&release=252\&type=5). Additionally, the RFC 7230 Validation for Apex RestResponse Headers that were enforced in Spring '24 will also not impact GRAX. #### Will the upcoming depreciation of third party cookies for certain browsers affect Salesforce or my GRAX Application? No. The Cookies Having Independent Partitioned State (CHIPS) change that Google will be making will not have any effect on your GRAX connection in Salesforce or your GRAX Application; you can read more about this [here](https://developers.google.com/privacy-sandbox/3pcd/chips). #### Does GRAX support Enhanced Domains? Yes. [Enhanced domains](https://help.salesforce.com/s/articleView?id=sf.domain_name_enhanced.htm\&type=5) meet the latest browser requirements and are now being used in all Salesforce orgs. As long as you're using OAuth to login to your Salesforce environments, the Enhanced Domain Enforcement impact to GRAX is minimal. After making the domain change, you'll need to take the following steps: * Login to your SFDC environment * In a separate tab navigate directly to your GRAX Application URL and append /web to the end and select `Sign-in with Salesforce` or `Connect with Auto Config` - if you don't know this URL, you can also navigate to the GRAX Application by clicking on the `Schedule` tab within the GRAX managed package * Click on the `Allow` button on the Allow Access pop up window if it appears (there are some orgs that do not require this step) These steps should give you access back to GRAX. Once you have access, double check the Salesforce connection within GRAX (in the `Settings` section) and that the Integration User is connected to confirm the change. #### Does GRAX support Salesforce Hyperforce? Yes. [Salesforce Hyperforce](https://www.salesforce.com/products/platform/hyperforce/) is a new architecture that allows customers to run Salesforce applications on public cloud infrastructure providers such as Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure. Hyperforce was announced by Salesforce in December 2020 as part of its Dreamforce event. The Hyperforce API is the same as the Salesforce API. From a developer's perspective, this means that you can continue to use the Salesforce API to build and customize your applications, and the API remains the same regardless of whether you are running your Salesforce applications on Hyperforce or on Salesforce's own infrastructure. Therefore GRAX works with Hyperforce deployments with no changes. If you do use: * Hyperforce * GRAX LWC in Apex (non-iframe) mode * Self-managed GRAX deployment Your AWS deployment has a Web Application Firewall (WAF), which may block the LWC API requests coming from your Hyperforce Apex servers. In this case, Salesforce publishes [a list of Hyperforce external IPs](https://compliance.salesforce.com/en/documents/a006e0000121zduAAA) which will need to be allowed to communicate with your GRAX Application. This configuration is uncommon, and a simpler solution is to use the [GRAX LWC in IFrame mode](https://documentation.grax.com/reuse-data/managed-package/first-generation/features#configuration). #### Does an instance refresh impact my GRAX connection? No. A Salesforce instance refresh/migration occurs when SFDC upgrades the infrastructure supporting your instance in their data centers. Following this maintenance, your instance will move to a new data center, and the name of your instance will change. Prior to an instance refresh/maintenance, SFDC will provide customers a date/timeline when the refresh will occur, specifying a maintenance window where their instance will be down. GRAX will be available during this SFDC maintenance period but GRAX will not be able to perform any backup or archive operations until the SFDC org is back up and available. GRAX uses OAuth to access SFDC, so you may need to reauthorize the integration. Once the instance refresh/migration is complete, you’ll want to re-authenticate the GRAX integration user by navigating to the GRAX Application and logging in with the integration user. After the integration User has been authenticated, GRAX will resume all activities (backup, archive, etc.) from the point where the service was disrupted. Please review [SFDC Instance Refresh Maintenance best practices](https://help.salesforce.com/s/articleView?id=000387056\&type=1) for additional information. #### Does GRAX use Salesforce Platform Events? No. Salesforce Platform Events don't support all objects needed by GRAX and [Event Allocations](https://developer.salesforce.com/docs/atlas.en-us.platform_events.meta/platform_events/platform_event_limits.htm) are VERY limiting (when backing up ALL objects). Currently, [GRAX Backup](https://documentation.grax.com/protect-data/backup) captures [Salesforce objects](https://documentation.grax.com/protect-data/backup/supported-objects), [Binary Files](doc:supported-objects#file-objects) (Attachment, Content, Chatter Files, etc), and Salesforce system-tables. Data and binary files MUST be captured at the same rate/time or you risk damaging referential integrity or completeness of data. To fulfill our customers backup needs Salesforce Platform Events aren't an option due to incomplete object support. If you have questions please reach out to the [GRAX team](https://app.gitbook.com/s/Ujmw35eIn7K36mAY2dSt/get-support). #### Can GRAX be used with Salesforce Shield Platform Encryption? Yes. [Salesforce.com's Shield Platform Encryption](https://help.salesforce.com/s/articleView?id=sf.security_pe_overview.htm\&type=5) should not impact your ability to use GRAX, as GRAX is designed to work seamlessly with Salesforce's native encryption. Salesforce provides various encryption options for data at rest and in transit, such as Shield Platform Encryption and Transport Layer Security (TLS). These encryption options are designed to protect data from unauthorized access and breaches, but they do not affect the functionality of third-party applications like GRAX. You can encrypt certain fields on standard and custom objects, data in Chatter, and search index files. With some exceptions, encrypted fields work normally throughout the Salesforce user interface, business processes, and APIs. GRAX operates through the Salesforce API, which allows it to access data regardless of whether it is encrypted or not. GRAX creates backups of your data, and these backups are stored in a separate, encrypted data store. When you need to restore data, GRAX decrypts the backup and restores it to Salesforce. *Helpful Links* * [GRAX Compliance](https://app.gitbook.com/s/DAX8eQtnPw2DBugi77sX/) * [Salesforce Shield Platform Encryption](https://help.salesforce.com/s/articleView?id=sf.security_pe_overview.htm\&type=5) * [What You Can Encrypt](https://help.salesforce.com/s/articleView?id=sf.security_pe_encryptable_data.htm\&type=5) * Try out Shield Platform Encryption at no charge in a [Salesforce Developer Edition orgs](https://developer.salesforce.com/signup). #### Does GRAX support Salesforce Private Connect? Yes. [Salesforce Private Connect](https://help.salesforce.com/s/articleView?id=sf.private_connect_overview.htm\&type=5) routes SFDC traffic via Salesforce-managed public cloud VPCs instead of letting egress traffic cross the public internet. This is a network-layer configuration; if configured correctly, the GRAX Application won't be able to tell the difference between a public or private connection. GRAX isn't responsible for configuring or maintaining Private Connect. Private Connect requires additional Salesforce licensing. Private Connect is only available for self-managed GRAX deployments. #### Will the ICU Locale format update impact my GRAX Application? No. For orgs created after Winter 2020, the [ICU Locale format](https://help.salesforce.com/s/articleView?id=sf.icu_migration_enable.htm\&type=5) is enabled by default. The transition to the ICU Locale format will not impact your GRAX Application. #### Can I use GRAX to decommission a Salesforce org? Yes. Your company may have Salesforce orgs that you no longer want to maintain after a consolidation, merger, or business reorganization. GRAX makes it easy to protect this data, search, and reuse it without paying Salesforce to keep the org active. After disconnecting from Salesforce, the backup of your data remains safe with GRAX and accessible through powerful tools including Global Search. Record data can be exported to CSV and Data Lake may be used to send a copy of data to your downstream systems such as your data lake or analytics tools. #### How do I set up GRAX and decommission a Salesforce Org? 1. First, you will need a GRAX license. Please contact your account manager or GRAX Sales. 2. Next, you will need to set up a GRAX deployment and connect a storage bucket as described in the [Platform documentation](https://app.gitbook.com/s/qjG14Rorm2T7JgMy0hrc/). 3. Then, you will [connect to your Salesforce org](https://documentation.grax.com/other/settings/connecting-salesforce) using an account that has the Salesforce Administrator role and run GRAX `Auto Config`. This will assign the [GRAX Integration User](https://documentation.grax.com/other/permissions-and-access/integration-user) permission sets for the GRAX Application to operate. 4. Once the App is connected, log in with Salesforce OAuth and enable Backup if it is not yet running. Backup will automatically run as it protects all of your data. Note: There can be a backlog of data to synchronize when you initially connect to a Salesforce Org, and you will notice the [backup status](https://documentation.grax.com/protect-data/backup) noted as "backfilling" as this process is catching up. 5. Once Backup is complete, you can disconnect your Salesforce org, but first you will need to create a local user account via email and password to log in as GRAX will no longer have access to Salesforce OAuth to provide authentication. This can be done under Settings -> User Management in the GRAX Application. Your password link will be sent via email. Note that [GRAX Support](https://app.gitbook.com/s/Ujmw35eIn7K36mAY2dSt/get-support) will need to enable this feature on your org. 6. Now you can safely decommission your Salesforce Org. > Note that any changes to data in the Salesforce org made after disconnecting GRAX will no longer be captured by GRAX Backup. ### Common Salesforce Errors {% hint style="info" %} For Sandbox Seeding, the errors will need to be resolved in the seeding target org unless otherwise specified {% endhint %} #### What does `CANNOT_EXECUTE_FLOW_TRIGGER` or `CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY` mean? Your Salesforce org has a trigger interfering with record creation or deletion. **Apex CPU time limit exceeded** *Example:* `CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY: EmailMessageTrigger: System.LimitException: Apex CPU time limit exceeded` The Apex code in this trigger is taking too long to run on a record and failing with a `System.LimitException` error due to a SFDC platform time limit. The Salesforce platform enforces that Apex code from triggers must run within a short amount of time to prevent excessive resource consumption. Possible options to resolve this are: * Use the `Disable automations` option in the seeding job to disable the trigger for the duration of the seeding job (Sandbox Seeding only) * Modify the trigger to not run for the GRAX integration user * Optimize the trigger code to run in less time **General Trigger Execution** *Example:* `CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY: TaskTrigger: execution of AfterInsert` The code in the trigger is blocking the operation based on how the object or related objects are currently configured. Possible options to resolve this are: * Add an override to the job to set the field value that will allow the record to pass the trigger validation rule (Sandbox Seeding or Restore) * Use the `Disable automations` option in the job to disable the trigger for the duration of the seeding job (Sandbox Seeding only) * Modify the trigger to not run for the GRAX integration user * Review the trigger code to allow the operation to succeed #### What does `DUPLICATE_VALUE` or `DUPLICATES_DETECTED` mean? *Example:* `DUPLICATE_VALUE: A topic with the name, why-GRAX-is-amazing, already exists.` Salesforce requires unique values for certain fields (such as topic names) and GRAX has tried to insert a duplicate. * For Sandbox Seeding, this can happen when re-seeding a dataset without undoing the previous seed first. Undo the previous seed that included this record or select a different dataset to seed. * For Restore, this can happen if a record you are trying to restore was previously un-deleted from the Salesforce Recycle bin. To ensure data integrity and records are properly linked, always restore records from GRAX. #### What does `ENTITY_IS_DELETED` mean? *Example:* `ENTITY_IS_DELETED: entity is deleted` The record was deleted in Salesforce since the last backup was taken. Wait for the next Backup to complete and try the activity again. #### What does `FIELD_CUSTOM_VALIDATION_EXCEPTION` mean? *Example:* `FIELD_CUSTOM_VALIDATION_EXCEPTION: Activity can't be created for Closed enquiry. Please Re Open Enquiry to save Activity.` The `FIELD_CUSTOM_VALIDATION_EXCEPTION` error indicates that there is a custom validation rule or trigger on the object that is blocking the record from being created. Review the message details to determine the cause, possible options to resolve this are: * Add an override to the seeding job to set the field value that will allow the record to pass the validation rule (Sandbox Seeding or Restore) * Use the `Disable automations` option in the seeding job to disable the trigger for the duration of the seeding job (Sandbox Seeding only) * Modify the validation rule / trigger to not run for the GRAX integration user * Skip the object #### What does `INACTIVE_OWNER_OR_USER` mean? *Example:* `INACTIVE_OWNER_OR_USER: The specified user is inactive` The GRAX integration user does not have sufficient permissions to create the records for inactive users. Please review the permissions for the GRAX integration user / GRAX integration user permission set and ensure it has the `Set Inactive Owners` permission. See the [Restore Best Practices](https://documentation.grax.com/protect-data/restore/restore-best-practices) page. #### What does `INSUFFICIENT_ACCESS_OR_READONLY` mean? **Can't create a link for Email Message when it's not in draft state** *Example:* `INSUFFICIENT_ACCESS_OR_READONLY: You can't create a link for Email Message when it's not in draft state (field=LinkedEntityId)` When a ContentDocumentLink attached to an EmailMessage is deleted, there are some guards that prevent us from placing it back. Salesforce only allows links to emails that are either in state draft, or created with `IsClientManaged` set to true. If none of these apply, our recommended workaround is to archive the email message and then restore it using GRAX. In this process we're going to set that `IsClientManaged` flag, and then restore ContentDocumentLinks related to it. **Insufficient access rights on object id or record type** *Example:* `INSUFFICIENT_ACCESS_OR_READONLY: insufficient access rights on object id` or `INSUFFICIENT_ACCESS_OR_READONLY: invalid record type` The GRAX integration user does not have sufficient permissions to delete / create this object or record type. Please review the permissions for the GRAX integration user / GRAX integration user permission set and ensure it has correct object and record type access. **Invalid sharing type** *Example:* `INSUFFICIENT_ACCESS_OR_READONLY: Invalid sharing type V (field=ShareType)` You are trying to seed or restore a Content Document (ContentVersion + ContentDocumentLink) and you are getting an "Invalid sharing type V" error message. To resolve it, you need to configure file sharing to `Set by Record` in your Salesforce org settings: 1. Go to `Setup` and enter `Files` in the `Quick Find` box. 2. Under `Salesforce Files`, choose `General Settings`. 3. Enable the `Set by Record` option for files attached to records. Additional [documentation](https://help.salesforce.com/s/articleView?id=release-notes.rn_files_sharing_set_by_record.htm\&type=5\&release=222) is available from Salesforce. **Data Validation rules** There is a validation rule that is blocking the record from being created as it currently exists. * Review the message details to determine the cause and options to resolve by setting an override (such as setting the `state` field to `Draft` for the example above) * Skip the object. **Insufficient permissions** The GRAX integration user does not have sufficient permissions to create the record. Review the permissions for the GRAX integration user / GRAX integration user permission set and ensure it has access to create the records for that object #### What does `INVALID_CROSS_REFERENCE_KEY` or `INSUFFICIENT_ACCESS_ON_CROSS_REFERENCE_ENTITY` mean? *Example:* `INVALID_CROSS_REFERENCE_KEY: Missing reference: field TaskId ref ID: 00T46000008US01AAG` When Salesforce needs to do a many-to-many link between two objects (such as Accounts can have many tasks, task can be linked to many accounts), it uses a junction object called a cross-reference. This error means that the cross-reference GRAX is trying to create is missing or unable to access one side of the junction. This most commonly has two likely causes: 1. `Missing reference` - One of the 2 records that are being linked is missing: * Check to see if there are other errors in the job that may indicate the record was skipped * The record was manually excluded * The record was not included in the job due to a filter such as max children 2. `this ID value isn't valid for the user` or `You do not have the level of access necessary` - The GRAX integration user doesn't have sufficient permissions to create the cross-reference record type. Check the permissions for the GRAX integration user / GRAX integration user permission set and ensure it has access to create the cross-reference record type. #### What does `INVALID_OR_NULL_FOR_RESTRICTED_PICKLIST` mean? *Example:* `INVALID_OR_NULL_FOR_RESTRICTED_PICKLIST: bad value for restricted picklist field: Direct_Traffic (field=originalSource)` The picklist value GRAX is trying to set is not a valid value for the field. This can happen if the option was deleted since the record was backed up or in Sandbox seeding if the option was added to the source org after the target org was created. Possible options to resolve this are: * Add the missing picklist value to the picklist options for the field * Add an override to the job to set the field value to a valid picklist option #### What does `STORAGE_LIMIT_EXCEEDED` mean? *Example:* `STORAGE_LIMIT_EXCEEDED: storage limit exceeded` Your Salesforce target org does not have sufficient storage space to load the data you are trying to seed. GRAX provides an estimated size for all records vs the available space reported by the seeding target. Since these are estimates, its possible to exceed the available space when the data is actually loaded. Please edit the seeding job and reduce the amount of data being seeded, some options are: * Reduce file size by skipping the `ContentDocument` or `Attachment` objects * Skip objects that are not meaningful to the use cases for the seeded data. For example running a test on `Opportunity` data doesn't require `EmailMessages` * Clicking on the Object in the graph or list will show you the estimated size of the records for that object. You can use this to determine which objects are consuming the most space and use this info to decide which objects to skip #### What does `UNABLE_TO_LOCK_ROW` mean? *Example:* `UNABLE_TO_LOCK_ROW: unable to obtain exclusive access to this record or 10 records: ...` Salesforce is attempting to update a separate record based on the changes to the record that reported this error while a different record in the same batch is also updating that record. This can happen as a result of a roll-up summary field or a trigger, such as rolling up the number of opportunities on an account. If you're seeing this error, it means the Salesforce and GRAX ROW LOCK resolution processes were unable to separate the child records that are blocking each other. * Review the trigger logic to ensure it can properly handle batches of records (and doesn't assume only one record will be updated at a time) * For Sandbox Seeding, reduce the number of records being seeded * For Restore, retry the restore job for the failed records #### What does `UNKNOWN_EXCEPTION` mean? `UNKNOWN_EXCEPTION` is a catch-all for errors that Salesforce does not provide a specific error code for. The error message usually has a general description of the cause of the error. Search the internet or contact Salesforce support to help identify the root cause. Based on the cause of the error, it may be possible to correct it by setting an override or skipping the object. ### Troubleshooting Network and Infrastructure Issues This section covers issues blocking general operations of GRAX including networking failures, boot failures, restart behaviors, etc. This guide includes commands specific to the Amazon Linux 2 AMI maintained by AWS, but attempts to remain otherwise infrastructure-agnostic where possible. *Some steps may not work as intended if you have made heavy customizations to networking, image, or environment.* #### Where are GRAX files located? First, let's note the locations of the GRAX binary and environment file. Keeping track of their paths helps validate service configurations in later steps. In a typical installation, GRAX is stored under `/home/ec2-user/graxinc/grax` and the `.env` file is stored under `/home/ec2-user`. Thus, we'll use the following paths: * GRAX binary: `/home/ec2-user/graxinc/grax/grax` * GRAX command-line tools: `/home/ec2-user/graxinc/grax/graxctl` * Environment file: `/home/ec2-user/.env` #### Is GRAX executable? To ensure that Linux knows GRAX is an executable, we can check permissions on the file as follows: ```bash [root@grax-test-runtime grax]# ls -la total 278108 drwxr-xr-x 2 root root 137 Aug 20 10:09 . drwxr-xr-x 3 root root 18 Aug 18 12:38 .. -rwxr-xr-x 1 root root 71471248 Aug 20 10:09 grax -rwxr-xr-x 1 root root 56687264 Aug 19 15:48 graxctl -rw-r--r-- 1 root root 52411443 Aug 18 12:39 master.zip ``` The "x" in the permissions strings at the beginning of each line denotes an executable file. If the `grax` and `graxctl` files aren't executable, we can mark them as such: ```bash [root@grax-test-runtime grax]# chmod +x grax graxctl ``` #### How should the Environment file be formatted? There are several important rules to remember for `.env` files: 1. Only one key-value pair per line 2. Only `=` is supported as a key-value separator 3. Comments aren't supported Comments are the most commonly seen issue as teams often attempt to label values for later reference. Unfortunately, this causes most `.env` parsers to immediately return (sometimes non-fatally). This can lead to partial configurations and thus cause indeterminate symptoms. For a total example of a valid `.env` file, see our [Linux Install Guide](https://app.gitbook.com/s/d9R0vX0Xh14BwQLNh5fI/install-guides/install-on-linux). #### Is the service (systemd) working properly? This guide assumes you're operating GRAX as a permanent service on the instance with `systemd`. The most common issues with `systemd` are configuration issues in the service file. In a typical installation, the GRAX service file is at the path `/lib/systemd/system/grax.service`. **Validate Configuration** We can see the contents of the service configuration by using `cat`: ```bash [root@grax-test-runtime grax]# cat /lib/systemd/system/grax.service [Install] WantedBy=multi-user.target [Service] EnvironmentFile=/home/ec2-user/.env ExecStart=/home/ec2-user/graxinc/grax/grax Restart=always Type=simple [Unit] Description=grax daemon ``` Check the following: 1. `EnvironmentFile` is a valid absolute path that points to your GRAX `.env` file. 2. `ExecStart` is a valid absolute path that points to your GRAX executable. 3. `Restart` is "always" to ensure GRAX is always running regardless of exit-singaling. **Service Status** The services run via `systemd` are managed and interacted with via the `systemctl` command. To see the current status of the GRAX service, we can use the `status` subcommand: ```bash [root@grax-test-runtime grax]# systemctl status grax.service ● grax.service - grax daemon Loaded: loaded (/usr/lib/systemd/system/grax.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-08-19 17:35:29 UTC; 5 days ago Main PID: 13125 (grax) CGroup: /system.slice/grax.service └─13125 /home/ec2-user/graxinc/grax/grax (Log Lines omitted for brevity) Hint: Some lines were ellipsized, use -l to show in full. ``` We always expect the GRAX service to be "active" unless under maintenance or GRAX was intentionally taken offline. If the service isn't "active" (that is "failed" or "stopped"), you can restart GRAX at any time by running the `restart` subcommand: ```bash [root@grax-test-runtime grax]# systemctl restart grax.service ``` If the GRAX service is entirely disabled, you can enable it with the `enable` subcommand, and then enforce a start immediately with `start`: ```bash [root@grax-test-runtime grax]# systemctl enable grax.service; systemctl start grax.service ``` A successful start of the service (and thus app) outputs logs in the app log file. If you have a regular health check configured, you'll see logs in relation to those calls being submitted to the log if the app is active. #### Is the Web Server Serving Requests? GRAX is a web server and API. It offers an endpoint for an external health check to see if the app is available. The health check endpoint for GRAX is an HTTP/1.1 HTTPS-only GET handler on `/health`. In a typical installation, GRAX runs on port 8000. We can manually check the status of the app from the instance by curling the local route: ```bash [root@grax-test-runtime grax]# curl -k https://localhost:8000/health ok ``` The expected value from the endpoint HTTP status 200; this signifies a healthy service. A failed call, either via timeout, rejection, or different status is a sign of a failed service/app. This endpoint is designed for load balancer registration and de-registration, not for instance replacement. #### Is Connectivity intact? The GRAX Application is a data-processor at its core. To process data, it must be able to retrieve that data, write it to storage, and read it back. When you add app maintenance, licensing, and telemetry to the equation, connectivity is critical to ensure proper operation. Only some pieces of overall connectivity requirements are possible to test from the instance. These are the egress connections that are used to push or pull data to/from remote resources. Ingress communications, as they start from other sources, are harder to test. Timeouts, rejections, or broken connections during the following tests are considered failures. All failures should be investigated. **GRAX HQ** Communication to GRAX HQ is egress-only, and can be tested relatively simply. To start, we can verify connectivity to the GRAX packaging API, which allows downloading of the app in the first place: ```bash [root@grax-test-runtime grax]# curl -L -o testgrax https://hq.grax.com/api/v2/download/graxinc/grax/master/linux/amd64 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 68 100 68 0 0 4270 0 --:--:-- --:--:-- --:--:-- 4533 100 48.6M 0 48.6M 0 0 6344k 0 --:--:-- 0:00:07 --:--:-- 9270k ``` The `-L` flag is utilized to follow the ALB redirect that points to HQ. `-o` is used to avoid printing the binary data to the terminal. A successful download of several dozen MB can be considered a passing test. Next, let's confirm POST requests to HQ succeed: ```bash [root@grax-test-runtime grax]# curl -L -X POST https://hq.grax.com/api/v2/dd/logs/api/v2/logs {"cause":"","status":401,"message":"Unauthorized"} ``` This may seem an unusual result, but the 401 return is a good enough response to know that your POST request made it to HQ without requiring you construct a valid set of credentials for a simple test. If you don't get a JSON response in line with the above, consider the test a failure. **Salesforce** To read and write Salesforce data via the Salesforce API, GRAX must first be able to connect. We can test that connectivity much the same as above: ```bash [root@grax-test-runtime grax]# curl https://test.salesforce.com ``` The response to the above should be an HTML document, too large to post here. Repeat that test for the following: 1. `https://login.salesforce.com` 2. Any custom/my-domain paths configured in your organization **Postgres** To test connectivity to your DB instance, we use `postgresql`, a Linux command-line tool that allows direct interaction with Postgres clusters. Installing the tool may be unnecessary depending on image, but can easily be done like the following: ```bash [root@grax-test-runtime grax]# yum install postgresql Loaded plugins: extras_suggestions, langpacks, priorities, update-motd amzn2-core | 3.7 kB 00:00:00 amzn2extra-docker | 3.0 kB 00:00:00 Package postgresql-9.2.24-6.amzn2.x86_64 already installed and latest version Nothing to do ``` As you can see, our typical installation already includes the right tooling. We can connect in two ways: 1. Copy the `DATABASE_URL` value from your `.env` file, and run `psql [database_url]` 2. Use the `graxctl psql` subcommand A valid connection results in an interactive `psql` shell, which can be exited with `\q`: ```bash [root@grax-test-runtime grax]# ./graxctl psql 2022/08/25 16:25:54 trace C9tATg VBlnGV start main mainWithCode:152 e=0s 2022/08/25 16:25:54 pprof addr: [::]:46569 2022/08/25 16:25:54 trace C9tATg VBlnGV info config setTemplateDefaults:427 msg="loading general template v1.0.0 defaults" template=virtual-appliance e=0s 2022/08/25 16:25:54 trace C9tATg VBlnGV info config/secrets New:175 secretStore=database e=11ms psql (9.2.24, server 14.5) WARNING: psql version 9.2, server version 14.0. Some psql features might not work. SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256) Type "help" for help. grax=> \q ``` If a connection cannot be made, `graxctl` tries again every 5 seconds for a few minutes. This is usually a sign of the following: 1. `DATABASE_URL` isn't set 2. `DATABASE_URL` isn't properly formatted 3. `DATABASE_URL` contains invalid cluster name 4. `DATABASE_URL` contains a password with special characters that need to be escaped 5. `DATABASE_URL` isn't exported to current environment 6. Route tables are forcing DB traffic outside of the VPC 7. Security groups aren't allowing traffic from the Instance into the DB If a connection can be made but you receive a Postgres error about DB existence, credentials, etc., then you likely have an issue with correctness in your `DATABASE_URL` value (that is username, password, or DB name). #### *Is additional assistance available?* If you have exhausted the steps here and require further assistance (or have recommendations for quality/completeness of this guide), contact [GRAX Support](https://app.gitbook.com/s/Ujmw35eIn7K36mAY2dSt/get-support).