Custom Log Types

Overview

Panther allows users to define their own log types by adding a Custom Log Type entry. Custom Log Types are identified by a Custom. prefix in their name and can be used wherever a 'native' Log Type is used:

  • You can use a Custom Log Type when onboarding data through SQS or S3

  • You can write Rules for these Log Types.

  • You can query the data in Data Explorer. Panther will create a new table for the Custom Log Type, once you onboard a source that uses it.

  • You can query the data through Indicator Search

Limitations

Currently Panther does not allow editing of Custom Log Types. Changing the schema of a Log Type requires extensive checks and possibly a migration of log tables. This will be allowed in the future once all corner cases are dealt with.

The only editing action allowed for a Custom Log Type is 'Delete' and this only succeeds if no Source is using it. Deleting a Custom Log Type does not affect any data already stored in the data lake. All data is still queryable through Data Explorer or Indicator Search. To provide this, Panther has to reserve the name of a Custom Log Type even after it has been deleted. Adding another one with the same name will fail with a conflict.

Adding a Custom Log Type

You can add a Custom Log Type by navigating to Log Analysis -> Custom Logs and clicking on the 'New Schema' button in the upper right corner.

Create custom logs screen

Here you must enter a name for the Custom Log Type (ie Custom.SampleAPI) and write or paste your YAML Log Schema definition. Use the 'Validate Syntax' button at the bottom to verify your schema contains no errors and hit 'Save'.

Note that the 'Validate Syntax' button only checks the syntax of the Log Schema. 'Save' might still fail due to name conflicts.

If all went well, you can now navigate to Log Analysis -> Sources and add either add a new source or modify an existing one to use the new Custom.SampleAPI _Log Type. Once Panther receives events from this Source, it will and process the logs and store the Log Events to the custom_sampleapi table. You can now write Rules to match against these logs and query them using the Data Explorer.

Writing a Log Schema for JSON logs

To parse log files where each line is JSON you have to define a Log Schema that describes the structure of each log entry.

For example, suppose you have logs that adhere to the following JSON structure:

{
"method": "GET",
"path": "/-/metrics",
"format": "html",
"controller": "MetricsController",
"action": "index",
"status": 200,
"params": [],
"remote_ip": "1.1.1.1",
"user_id": null,
"username": null,
"ua": null,
"queue_duration_s": null,
"correlation_id": "c01ce2c1-d9e3-4e69-bfa3-b27e50af0268",
"cpu_s": 0.05,
"db_duration_s": 0,
"view_duration_s": 0.00039,
"duration_s": 0.0459,
"tag": "test",
"time": "2019-11-14T13:12:46.156Z"
}

You can define a Log Schema for these logs using:

version: 0
fields:
- name: time
description: Event timestamp
required: true
type: timestamp
timeFormat: rfc3339
isEventTime: true
- name: method
description: The HTTP method used for the request
type: string
- name: path
description: The path used for the request
type: string
- name: remote_ip
description: The remote IP address the request was made from
type: string
indicators: [ ip ] # the value will be appended to `p_any_ip_addresses` if it's a valid ip address
- name: duration_s
description: The number of seconds the request took to complete
type: float
- name: format
description: Response format
type: string
- name: user_id
description: The id of the user that made the request
type: string
- name: params
type: array
element:
type: object
fields:
- name: key
description: The name of a Query parameter
type: string
- name: value
description: The value of a Query parameter
type: string
- name: tag
description: Tag for the request
type: string
- name: ua
description: UserAgent header
type: string

Such a YAML file can either be input directly in the Panther UI or prepared offline in your editor/IDE of choice. For more information on the structure and fields in a Log Schema, see the Log Schema Reference.

Writing a Log Schema for text logs

Panther handles logs that are not structured as JSON by using a 'parser' that translates each log line into key/value pairs and feeds it as JSON to the rest of the pipeline. You can define a text parser using the parser field of the Log Schema. Panther provides the following parsers for non-JSON formatted logs:

Name

Description

fastmatch

Match each line of text against one or more simple patterns

regex

Use regular expression patterns to handle more complex matching such as conditional fields, case-insensitive matching etc

csv

Treat log files as CSV mapping colunm names to field names

Testing a Log Schema

Panther provides a simple CLI tool to validate and test a log schema YAML file. The tool is called customlogs and an executable for each platform is provided with the release. The tool validates a schema file and uses it to parse log files in new-line delimited JSON format. Processed logs are writen to stdout and errors to stderr.

For example, to parse logs in sample_logs.jsonl with the log schema in schema.yml, use:

$ ./customlogs -s schema.yml sample_logs.jsonl

The tool can also accept input via stdin so it can be used in a pipeline:

$ cat sample_logs.jsonl | ./customlogs -s schema.yml