Panther can be configured to write processed log data to an AWS-based Snowflake database cluster. Once the Panther-Snowflake integration is set up, Data Explorer, Indicator Search and other back-end processes will automatically switch to using Snowflake instead of Athena.
Integrating Panther with Snowflake enables Panther data to be used in your Business Intelligence tools to make dashboards tailored to you operations. In addition, you can join Panther data (e.g., Panther alerts) to your business data, enabling assessment of your security posture with respect to your organization.
For example, you can tally alerts by organizational division (e.g., Human Resources) or by infrastructure (e.g., Development, Test, Production).
Panther uses Snowpipe to copy the data into your Snowflake cluster.
Panther uses a
panther_readonly Snowflake user to query the data in Snowflake. By default, this user's role
panther_readonly_role is only endowed with a minimal set of grants to enable it to access the data in the panther databases. However, if you wish to add your own preexisting datasets to your Panther data-explorer queries (such as HR data, in-house or vendor-provided whitelists/blacklists) you can easily make that data accessible to the role.
GRANT USAGEON DATABASE my_database_nameTO ROLE panther_readonly_role;GRANT USAGEON SCHEMA my_database_name.my_schema_nameTO ROLE panther_readonly_role;GRANT SELECTON TABLE my_database_name.my_schema_name.my_table_nameTO ROLE panther_readonly_role;
Note that the newly granted database, schema and table will not populate in the Panther sidebar, but you will be able to access it using regular SQL.
Panther implements a set of basic data load status self-monitoring tables:
a pipe status monitoring table
a file load status monitoring table
a data source table history
a monitoring history, tracking any load errors encountered
Under the default snowflake configuration settings, a master stack variable named
SnowflakeMonitorRunFrequency is set to run a monitoring sweep every 180 minutes. This variable can be adjusted as desired down to a minimum of once every 2 minutes, or up to a maximum of once every 10080 minutes (a week).
If a data loading error is found by the monitor sweep, this should trigger an alarm in CloudWatch, under the
/aws/lambda/panther-snowflake-admin-api log group. The data loading errors are also stored in the data monitoring history table.
The monitoring tables are implemented in the
panther_monitor database, and can be queried via the WebUI or the Panther Data Explorer. Please note that the monitor database is not currently one of the pre-populated data sets in the Panther UI.
Customers should run the latest version of the snowflake setup instructions before upgrading to a version including snowflake data load monitoring.
The monitors are designed to capture information from a variety of sources: 1. The pipe status captures information about the state of all the snowpipes used by the Panther system, and will keep a history of it in the
pipe_history table. Pipes that are not in a
RUNNING state will generate an alert in CloudWatch and will be recorded as an entry in
Load status of every file for every table is stored in
load_history -- if a file failed to load (state not
LOAD_IN_PROGRESS) for any reason, the even will be recorded in CloudWatch and also as an entry in the
Table History -- at every run of the monitoring sweep, the table history will update, including such information as row and byte counts and row and byte count deltas. Please note that due to clustering compaction operations by Snowflake, byte counts may decrease occasionally between sweeps. Additional useful information in this table is a last_altered time which indicates the last time changes were made to the data in the table, and a last altered delta, indicating the time in seconds between when these changes were made and the last monitor sweep. As different data sources can have dramatically different load frequencies (ranging from seconds to weeks) no default alerting is currently driven from the table history table.
Monitor History -- if there are errors encountered during the load process (an example would be s3 access errors caused by a change in s3 permissions) such errors would be summarized in the monitor_history table.