Log Schema Reference
In this guide, you will find common fields used to build YAML-based schemas when onboarding custom log types (logs that do not have built-in support in Panther).

LogSchema fields

    version (0,required): The version of the log schema. This field should be set to zero (0). Its purpose is to allow backwards compatibility with future versions of the log schema.
    fields ([]FieldSchema, required): The fields in each Log Event.
    parser (ParserSpec): Define a parser that will convert non-JSON logs to JSON.

Example

1
version: 0
2
parser:
3
csv:
4
delimiter: ','
5
fields:
6
- name: action
7
type: string
8
required: true
9
- name: time
10
type: timestamp
11
timeFormat: unix
Copied!

ParserSpec

A ParserSpec specifies a parser to use to convert non-JSON input to JSON. Only one of the following fields can be specified:
    fastmatch (FastmatchParser{}): Use fastmatch parser
    regex (RegexParser{}): Use regex parser
    csv (CSVParser{}): Use csv parser

Parser fastmatch fields

    match ([]string, required, non-empty): One or more patterns to match log lines against.
    emptyValues ([]string): Values to consider as null.
    expandFields (map[string]string): Additional fields to be injected by expanding text templates.
    trimSpace (bool): Trim space surrounding each value

Parser regex fields

    match ([]string, required, non-empty): A pattern to match log lines against (can be split it into parts for documentation purposes).
    patternDefinitions (map[string]string): Additional named patterns to use in match pattern.
    emptyValues ([]string): Values to consider as null.
    expandFields (map[string]string): Additional fields to be injected by expanding text templates.
    trimSpace (bool): Trim space surrounding each value

Parser csv

    delimiter (string, required, non-empty): A character to use as field delimiter.
    hasHeader (bool): Use first row to derive column names (unless columns is set also in which case the header is just skipped).
    columns ([]string, required(without hasHeader), non-empty): Names for each column in the CSV file. If not set, the first row is used as a header.
    emptyValues ([]string): Values to consider as null.
    trimSpace (bool): Trim space surrounding each value
    expandFields (map[string]string): Additional fields to be injected by expanding text templates.

FieldSchema

A FieldSchema defines a field and its value. The field is defined by:
    name (String,required): The name of the field.
    required (Boolean): If the field is required or not.
    description (String): Some text documenting the field.
And its value is defined using the fields of a ValueSchema.

ValueSchema

A ValueSchema defines a value and how it should be processed. Each ValueSchema has a type field that can be of the following values:
Value Type
Description
string
A string value
int
A 32-bit integer number in the range -2147483648, 2147483647
smallint
A 16-bit integer number in the range -32768, 32767
bigint
A 64-bit integer number in the range -9223372036854775808, 9223372036854775807
float
A 64-bit floating point number
boolean
A boolean value true / false
timestamp
A timestamp value
array
A JSON array where each element is of the same type
object
A JSON object of known keys
json
Any valid JSON value (JSON object, array, number, string, boolean)
The fields of a ValueSchema depend on the value of the type field.
Type
Field
Value
Description
object
fields (required)
An array of FieldSpec objects describing the fields of the object.
array
element (required)
A ValueSchema describing the elements of an array.
timestamp
timeFormat (required)
String
The format to use for parsing the timestamp. (see Timestamps)
timestamp
isEventTime
Boolean
A flag to tell Panther to use this timestamp as the Log Event Timestamp.
string
indicators
[]String
Tells Panther to extract indicators from this value (see Indicators)
string
validate
Validation rules for the string value

Timestamps

Timestamps are defined by setting the type field to timestamp and specifying the timestamp format using the timeFormat field. Timestamp formats can be either one of the built-in timestamp formats:
    rfc3339 The most common timestamp format.
    unix Timestamp expressed in seconds since UNIX epoch time. It can handle fractions of seconds as a decimal part.
    unix_ms Timestamp expressed in milliseconds since UNIX epoch time.
    unix_us Timestamp expressed in microseconds since UNIX epoch time.
    unix_ns Timestamp expressed in nanoseconds since UNIX epoch time.
or you can define a custom format by using strftime notation. For example:
1
# The field is a timestmap using a custom timestamp format like "2020-09-14 14:29:21"
2
- name: ts
3
type: timestamp
4
timeFormat: "%Y-%m-%d %H:%M:%S" # note the quotes required for proper YAML syntax
Copied!
Timestamp values can also be marked with isEventTime: true to tell Panther that it should use this timestamp as the p_event_time field. It is possible to set isEventTime on multiple fields. This covers the cases where some logs have optional or mutually exclusive fields holding event time information. Since there can only be a single p_event_time for every Log Event, the priority is defined using the order of fields in the schema.

Indicators

Values of string type can be used as indicators. In order to mark a field as an indicator, you must set the indicators field to an array of indicator scanner names. This will instruct Panther to parse the string and store any indicator values it finds to the relevant field. For example:
1
# Will scan the value as IP address and store it to `p_any_ip_addresses`
2
- name: remote_ip
3
type: string
4
indicators: [ ip ]
5
6
# Will scan the value as URL and store domain name or ip address to `p_any_domain_names` or `p_any_ip_addresses`
7
- name: target_url
8
type: string
9
indicators: [ url ]
Copied!

Validation - Allow/Deny lists

Values of string type can be further restricted by declaring a list of values to allow or deny. This allows to have different log types that have common overlapping fields but differ on values of those fields.
1
# Will only allow 'login' and 'logout' event types to match this log type
2
- name: event_type
3
type: string
4
validate:
5
allow: [ "login", "logout"]
6
7
# Will match any event type other than 'login' and 'logout'
8
- name: event_type
9
type: string
10
validate:
11
deny: [ "login", "logout"]
Copied!

Validation by string type

This feature will be available in 1.24
Values of string type can be restricted to match well-known formats. Currently, Panther supports the ip and cidr formats to require that a string value be a valid IP address or CIDR range. Note that the ip and cidr validation types can be combined with allow or deny rules but it is somewhat redundant, for example, if you allow two IP addresses, then adding an ip validation will simply ensure that your validation will not include false positives if the IP addresses in your list are not valid.
1
# Will allow valid ipv4 IP addresses e.g. 100.100.100.100
2
- name: address
3
type: string
4
validate:
5
ip: "ipv4"
6
7
# Will allow valid ipv6 CIDR ranges
8
# e.g. 2001:0db8:85a3:0000:0000:0000:0000:0000/64
9
- name: address
10
type: string
11
validate:
12
cidr: "ipv6"
13
14
# Will allow any valid ipv4 or ipv6 address
15
- name: address
16
type: string
17
validate:
18
ip: "any"
Copied!

Embedded JSON values

Sometimes JSON values are delivered 'embedded' in a string. For example the input JSON could be of the form:
1
{
2
"timestamp": "2021-03-24T18:15:23Z",
3
"message": "{\"foo\":\"bar\"}"
4
}
Copied!
To have Panther parse the escaped JSON inside the string you can use an isEmbeddedJSON: true flag. This flag is valid for values of type object, array and json.
By defining message as:
1
- name: message
2
type: object
3
isEmbeddedJSON: true
4
fields:
5
- name: foo
6
type: string
Copied!
each event will be stored as:
1
{
2
"timestamp": "2021-03-24T18:15:23Z",
3
"message": {
4
"foo": "bar"
5
}
6
}
Copied!

Using JSON Schema in an IDE

If your editor/IDE supports JSON Schema, you can use this JSON Schema file for validation and autocompletion. You can also use the resources below as well to create custom JSON schemas:

JetBrains

You can find instructions on how to configure JetBrains IDEs to use custom JSON Schemas here.

VSCode

You can find instructions on how to configure VSCode to use JSON Schema here.
Last modified 30d ago