GCP Logs
Connecting GCP logs to your Panther Console
Overview
Panther supports ingesting Google Cloud Platform (GCP) logs via common Data Transport options.
To connect GCP logs with Panther, it's recommended to use the Pub/Sub Data Transport source with a log sink, as it results in the lowest latency—roughly five minutes.
Alternatively, using the Google Cloud Storage (GCS) Data Transport source with a log sink will result in logs being delivered to Panther only on an hourly basis.
How to onboard GCP logs to Panther
Prerequisite
Set a default Data Access audit logging configuration for your Google Cloud services:
In your GCP console, navigate to the IAM & Admin service. In the navigation bar, click Audit Logs.
Click Save.
These instructions for setting a default Data Access audit log configuration for your Google Cloud services are also found in the GCP documentation: Set the default configuration.
Step 1: Create a Google Cloud source in Panther
In the left-hand navigation bar of your Panther Console, click Configure > Log Sources.
Click Create New.
Search for "GCP" then click the Google Cloud tile.
In the slide-out panel, in the Transport Mechanism dropdown in the upper right corner, select Google Cloud Pub/Sub.
It is possible to use any of the Data Transport options, but is recommended to use Pub/Sub in conjunction with a log sink, which you will configure in the next step.
Follow the Panther documentation for configuring your selected Data Transport.
If you selected Pub/Sub, follow the Pub/Sub Source instructions.
Step 2: Configure GCP to push logs to the Data Transport source
See GCP's documentation for instructions on how to forward logs to your selected Data Transport source.
If you are using Pub/Sub or Google Cloud Storage as your Data Transport, configure a log sink.
Video walkthrough: Setup using GCS
Panther-managed detections
See Panther-managed rules for Google Cloud Platform in the panther-analysis GitHub repository.
Supported log types
GCP.AuditLog
The GCP.AuditLog schema supports ingesting all four types of Google Cloud audit logs:
For more information, see the GCP Documentation on Cloud Audit Logs.
schema: GCP.AuditLog
description: |
Cloud Audit Logs maintains audit logs for each Google Cloud project, folder, and organization: Admin Activity, Data Access, System Event, and Policy Denied.
Google Cloud services write audit log entries to these logs to help you answer the questions of "who did what, where, and when?" within your Google Cloud resources.
referenceURL: https://cloud.google.com/logging/docs/audit
fields:
- name: logName
required: true
description: The resource name of the log to which this log entry belongs.
type: string
- name: severity
description: The severity of the log entry. The default value is LogSeverity.DEFAULT.
type: string
- name: insertId
description: A unique identifier for the log entry.
type: string
- name: resource
description: The monitored resource that produced this log entry.
type: object
fields:
- name: type
required: true
description: Type of resource that produced this log entry
type: string
- name: labels
description: Labels describing the resource
type: json
- name: timestamp
description: The time the event described by the log entry occurred.
type: timestamp
timeFormats:
- rfc3339
isEventTime: true
- name: receiveTimestamp
required: true
description: The time the log entry was received by Logging.
type: timestamp
timeFormats:
- rfc3339
isEventTime: true
- name: labels
description: A set of user-defined (key, value) data that provides additional information about the log entry.
type: json
- name: operation
description: Information about an operation associated with the log entry, if applicable.
type: object
fields:
- name: id
description: Log entries with the same identifier are assumed to be part of the same operation.
type: string
- name: producer
description: An arbitrary producer identifier. The combination of id and producer must be globally unique.
type: string
- name: first
description: This is the first entry in an operation
type: boolean
- name: last
description: This is the last entry in an operation
type: boolean
- name: trace
description: Resource name of the trace associated with the log entry, if any. The trace field provides the link between logs and traces.
type: string
- name: httpRequest
description: Information about the HTTP request associated with this log entry, if applicable.
type: object
fields:
- name: requestMethod
description: The request HTTP method.
type: string
- name: requestURL
description: The scheme (http, https), the host name, the path and the query portion of the URL that was requested.
type: string
indicators:
- url
- name: requestSize
description: The size of the HTTP request message in bytes, including the request headers and the request body.
type: bigint
- name: status
description: The response HTTP status code
type: smallint
- name: responseSize
description: The size of the HTTP response message sent back to the client, in bytes, including the response headers and the response body.
type: bigint
- name: userAgent
description: The user agent sent by the client.
type: string
- name: remoteIP
description: The IP address (IPv4 or IPv6) of the client that issued the HTTP request.
type: string
indicators:
- ip
- name: serverIP
description: The IP address (IPv4 or IPv6) of the origin server that the request was sent to.
type: string
indicators:
- ip
- name: referer
description: The referer URL of the request
type: string
indicators:
- url
- name: latency
description: The request processing latency in seconds on the server, from the time the request was received until the response was sent.
type: string
- name: cacheLookup
description: Whether or not a cache lookup was attempted.
type: boolean
- name: cacheHit
description: Whether or not an entity was served from cache (with or without validation).
type: boolean
- name: cacheValidatedWithOriginServer
description: Whether or not an entity was served from cache (with or without validation).
type: boolean
- name: cacheFillBytes
description: Whether or not an entity was served from cache (with or without validation).
type: bigint
- name: protocol
description: Protocol used for the request.
type: string
- name: spanId
description: The span ID within the trace associated with the log entry.
type: string
- name: traceSampled
description: The sampling decision of the trace associated with the log entry.
type: boolean
- name: sourceLocation
description: Source code location information associated with the log entry, if any.
type: object
fields:
- name: file
description: Source file name. Depending on the runtime environment, this might be a simple name or a fully-qualified name.
type: string
- name: line
description: Line within the source file. 1-based; 0 indicates no line number available.
type: bigint
- name: function
description: Human-readable name of the function or method being invoked, with optional context such as the class or package name. The format can vary by language
type: string
- name: protoPayload
required: true
description: The AuditLog payload
type: object
fields:
- name: '@type'
required: true
description: The type of payload
type: string
- name: serviceName
description: The name of the API service performing the operation
type: string
- name: methodName
description: The name of the service method or operation. For API calls, this should be the name of the API method.
type: string
- name: resourceName
description: The resource or collection that is the target of the operation. The name is a scheme-less URI, not including the API service name.
type: string
- name: numResponseItems
description: The number of items returned from a List or Query API method, if applicable.
type: bigint
- name: status
description: The status of the overall operation.
type: object
fields:
- name: code
description: The status code, which should be an enum value of google.rpc.Code.
type: int
- name: message
description: A developer-facing error message, which should be in English.
type: string
- name: details
description: A list of messages that carry the error details. There is a common set of message types for APIs to use.
type: json
- name: authenticationInfo
description: Authentication information.
type: object
fields:
- name: principalSubject
description: String representation of identity of requesting party. Populated for both first and third party identities.
type: string
- name: serviceAccountKeyName
description: The name of the service account key used to create or exchange credentials for authenticating the service account making the request. This is a scheme-less URI full resource name.
type: string
indicators:
- domain
- name: principalEmail
description: The email address of the authenticated user making the request.
type: string
indicators:
- email
- name: authoritySelector
description: The authority selector specified by the requestor, if any. It is not guaranteed that the principal was allowed to use this authority.
type: string
- name: thirdPartyPrincipal
description: The third party identification (if any) of the authenticated user making the request. When the JSON object represented here has a proto equivalent, the proto name will be indicated in the @type property.
type: json
- name: serviceAccountDelegationInfo
description: Identity delegation history of an authenticated service account that makes the request. It contains information on the real authorities that try to access GCP resources by delegating on a service account. When multiple authorities present, they are guaranteed to be sorted based on the original ordering of the identity delegation events.
type: array
element:
type: object
fields:
- name: firstPartyPrincipal
description: First party (Google) identity as the real authority.
type: object
fields:
- name: principalEmail
description: The email address of a Google account.
type: string
indicators:
- email
- name: serviceMetadata
description: Metadata about the service that uses the service account.
type: json
- name: thirdPartyPrincipal
description: Third party identity as the real authority.
type: object
fields:
- name: thirdPartyClaims
description: Metadata about third party identity.
type: json
- name: principalSubject
description: String representation of identity of requesting party.
type: string
- name: authorizationInfo
description: Authorization information. If there are multiple resources or permissions involved, then there is one AuthorizationInfo element for each {resource, permission} tuple.
type: array
element:
type: object
fields:
- name: resource
description: The resource being accessed, as a REST-style string.
type: string
- name: permission
description: The required IAM permission
type: string
- name: granted
description: Whether or not authorization for resource and permission was granted.
type: boolean
- name: resourceAttributes
description: Resource attributes used in IAM condition evaluation. This field contains resource attributes like resource type and resource name. To get the whole view of the attributes used in IAM condition evaluation, the user must also look into AuditLog.request_metadata.request_attributes.
type: object
fields:
- name: service
description: The name of the service that this resource belongs to, such as pubsub.googleapis.com. The service may be different from the DNS hostname that actually serves the request.
type: string
- name: name
description: The stable identifier (name) of a resource on the service.
type: string
- name: type
description: The type of the resource. The syntax is platform-specific because different platforms define their resources differently.
type: string
- name: labels
description: The labels or tags on the resource, such as AWS resource tags and Kubernetes resource labels.
type: string
- name: uid
description: The unique identifier of the resource. UID is unique in the time and space for this resource within the scope of the service. It is typically generated by the server on successful creation of a resource and must not be changed. UID is used to uniquely identify resources with resource name reuses. This should be a UUID4.
type: string
- name: requestMetadata
description: Metadata about the request
type: object
fields:
- name: callerIP
description: The IP address of the caller.
type: string
indicators:
- ip
- name: callerSuppliedUserAgent
description: The user agent of the caller. This information is not authenticated and should be treated accordingly.
type: string
- name: callerNetwork
description: The network of the caller. Set only if the network host project is part of the same GCP organization (or project) as the accessed resource.
type: string
- name: requestAttributes
description: Request attributes used in IAM condition evaluation. This field contains request attributes like request time and access levels associated with the request.
type: json
- name: destinationAttributes
description: The destination of a network activity, such as accepting a TCP connection.
type: json
- name: request
description: The operation request. This may not include all request parameters, such as those that are too large, privacy-sensitive, or duplicated elsewhere in the log record. When the JSON object represented here has a proto equivalent, the proto name will be indicated in the @type property.
type: json
- name: response
description: The operation response. This may not include all response parameters, such as those that are too large, privacy-sensitive, or duplicated elsewhere in the log record. When the JSON object represented here has a proto equivalent, the proto name will be indicated in the @type property.
type: json
- name: metadata
description: Other service-specific data about the request, response, and other information associated with the current audited event.
type: json
- name: serviceData
description: Other service-specific data about the request, response, and other activities.
type: jsonGCP.HTTPLoadBalancer
External HTTP(S) Load Balancing distributes HTTP and HTTPS traffic to backends hosted on a variety of Google Cloud platforms (such as Compute Engine, Google Kubernetes Engine (GKE), Cloud Storage, and so on), as well as external backends connected over the internet or via hybrid connectivity. HTTP(S) load balancing logs provide information for monitoring and debugging web traffic.
For more information, see the HTTPLoadBalancer documentation.
schema: GCP.HTTPLoadBalancer
parser:
native:
name: GCP.HTTPLoadBalancer
fields:
- name: httpRequest
required: true
description: httpRequest
type: object
fields:
- name: referer
description: referer
type: string
indicators:
- url
- name: latency
required: true
description: latency
type: string
- name: remoteIp
required: true
description: remoteIp
type: string
indicators:
- ip
- name: requestMethod
required: true
description: requestMethod
type: string
- name: requestSize
required: true
description: requestSize
type: bigint
- name: requestUrl
required: true
description: requestUrl
type: string
indicators:
- url
- name: responseSize
description: responseSize
type: bigint
- name: serverIp
description: serverIp
type: string
indicators:
- ip
- name: status
description: status
type: bigint
- name: userAgent
description: userAgent
type: string
- name: insertId
required: true
description: insertId
type: string
- name: jsonPayload
required: true
description: jsonPayload
type: object
fields:
- name: '@type'
required: true
description: '@type'
type: string
- name: remoteIp
required: true
description: remoteIp
type: string
indicators:
- ip
- name: statusDetails
required: true
description: statusDetails
type: string
- name: logName
required: true
description: logName
type: string
- name: receiveTimestamp
required: true
description: receiveTimestamp
type: timestamp
timeFormat: rfc3339
- name: resource
required: true
description: resource
type: object
fields:
- name: labels
required: true
description: labels
type: object
fields:
- name: backend_service_name
required: true
description: backend_service_name
type: string
- name: forwarding_rule_name
required: true
description: forwarding_rule_name
type: string
- name: project_id
required: true
description: project_id
type: string
- name: target_proxy_name
required: true
description: target_proxy_name
type: string
- name: url_map_name
required: true
description: url_map_name
type: string
- name: zone
required: true
description: zone
type: string
- name: type
required: true
description: type
type: string
- name: severity
required: true
description: severity
type: string
- name: spanId
required: true
description: spanId
type: string
- name: timestamp
required: true
description: timestamp
type: timestamp
timeFormat: rfc3339
isEventTime: true
- name: trace
required: true
description: trace
type: string
indicators:
- trace_idLast updated
