Documentation
Login

Telemetry

GRAX monitors deployed software via egress-only log and telemetry streaming technologies. This allows the GRAX team to best ensure reliability without need for ingress connections or direct environment access, both of which have security implications. This document explains both of the methods in use today by GRAX and describes the data contained by both.

GRAX Metrics

Metrics are quantitative expressions of app performance, health, and configuration; their numerical nature assists in detection of failures. GRAX calculates and streams metrics continuously as the app runs. Telemetry regarding system resources (CPU, RAM, Disk, etc.) is streamed continuously, but the GRAX Application submits specialized telemetry events occasionally. These include, in part:

  • At time-of-boot
  • At time-of-update
  • Changing configuration of:
    • Auto Backup
    • Search
    • Data Lake
    • General Settings
  • Processing tasks change status (created, started, ended, failed, etc.):
    • Archives
    • Restores

GRAX Logs

Logs are more detailed and structured than app metrics; they're traditionally used for investigating issues -- not detecting them. GRAX streams logs with an authenticated egress-only connection, making application logs available to GRAX Engineering for the sake of supportability and bug-fixing. Here are some related key details:

  • GRAX logging never contains your Salesforce records or any system secrets.
  • Logs use the same authenticated hq.grax.com connection as licensing and metrics.
  • Access to these logs is tightly controlled internally at GRAX.
  • Logs are retained indefinitely.
  • This cannot be disabled.

Logging uses a forward-only collector and won't transmit logs from an earlier point in time.

What do GRAX logs contain?

Logs emitted by GRAX never contain customer CRM data, PII, or secrets. These logs are intentionally designed to be useful for GRAX engineers; as such, they contain:

  • Source Function Names
  • Source filenames
  • Function Timing Information
  • API Request Methods
  • API Request Paths
  • CPU Performance Metrics / Profiles
  • Memory Performance Metrics / Profiles
  • Storage Performance Metrics / Profiles
  • Function Metadata (Object Names, Batch Sizes, Record Counts, etc.)

As you can see, the data logged within the GRAX logging system is strictly related to operation and performance of the GRAX Application with no exposure of protected data at any time.

As stated above, logs from GRAX are intended for consumption by GRAX engineers. We don't publish documentation nor provide training on understanding the internals of the GRAX Application. This means that logs won't provide value to teams monitoring GRAX directly without the assistance of GRAX support.

Who can view GRAX logs?

Your logs are only visible to the engineers who directly support and manage operation of the GRAX Application. For more information about security controls, audits, and compliance, see here.

Network Considerations

This is required for the GRAX Application to operate. As such, egress to hq.grax.com is required at all times from the app. A static IP for this communication isn't currently available. Without this access, the app won't boot or run; this isn't configurable.

Data Security

GRAX takes security of customer data seriously. As such, none of your Salesforce data ever leaves the app environment. A breakdown of collected data follows:

  • Names of Salesforce objects (Standard and Custom) covered by backup and archive operations.
  • Number of records for Salesforce objects (Standard and Custom) covered by backup and archive operations.
  • GRAX backup/archive/restore configurations (schedule, start time, etc) and statuses.
  • Size, performance, and internal metrics for the proprietary GRAX storage layer in your storage bucket.
  • Size and performance metrics for the attached Postgres database.
  • Total data size sent and received to/from Salesforce.
  • Feature status (feature flags, feature access levels)
  • Structured app logging (optional)

No sensitive, classified, or restricted data or PII is included in telemetry communications. The content of backed up records isn't inspected for telemetry, nor is it made available to any GRAX engineers.

All data is encrypted with HTTPS and TLS 1.2+ while in flight, and encrypted on disk when at rest. Access to the telemetry dataset is restricted within the GRAX team to only engineers whose roles require access.

Third Party Tools

GRAX uses third party tools and services to protect, store, and analyze telemetry data. The current telemetry tool set includes:

Telemetry archives are encrypted at rest with AES-256 encryption.

Frequently Asked Questions

Can I save GRAX Application logs?

GRAX forwards app logs to a central GRAX service to help with automatic monitoring and customer support. GRAX retains these logs indefinitely. For more details see here here.

However, in self-managed GRAX deployments, you are in control of the configuration of your GRAX Application servers, and can configure them to forward logs to your own external storage systems.

How to do this depends on how your server is configured to run GRAX, as well as what log storage system you use. If you are using the standard GRAX AWS templates, GRAX logs go through:

  • systemd journal
  • rsyslog
  • /var/log/grax.log
  • AWS CloudWatch

Consult with your cloud team and logging service provider on how to configure the server to forwarding from one of these subsystems.

Can I use GRAX Application logs for auditing?

In self-managed GRAX deployments, after you forward logs to your own logging service, you also are in control of how these are used, and can use them for auditing, monitoring and alerting.

While the format of application logs is subject to change, every API call is logged and includes:

  • HTTP method, path, and status
  • Remote address and user agent
  • Salesforce Org ID and User ID of the caller

Here is an example log for when the user with ID 0054o000002Yz6lAAC updated storage settings:

{"keys":{"jwtAuth":"sfid_00D770000008i1kEAA_0054o000002Yz6lAAC","method":"PUT","path":"/api/v1/secrets/StorageSecret","remoteAddr":"[2600:1700:9da3:c850:3c02:8217:4d9f:2507]:52529","route":"/api/v1/secrets/:id","status":200,"userAgent":"Mozilla/5.0"}}

Updated about 21 hours ago