GCP Audit to S3 via Fluentd

Objectives

The objective of this recipe is to stream Google Cloud Project Audit Logs into Panther. Many businesses use several cloud providers, and these steps will allow teams to gather API calls that happen with a GCP account into Panther.
We’ll implement this using a combination of primitives across GCP and AWS.

Solution Brief

At a high level, we’ll be implementing the following flow:
  1. 1.
    Audit logs are generated in GCP and routed to PubSub
  2. 2.
    Fluentd polls PubSub and forwards to an S3 Bucket
  3. 3.
    The S3 bucket is onboarded into Panther for normalization, detection, and long-term storage

Steps

Step 1: Create a New Pub/Sub

Pub/Sub console
  • In GCP, open the Pub/Sub console
  • Create a new Topic, call it Panther-Audit
    • Uncheck ‘Add a default subscription’
    • Select CREATE TOPIC
  • Click Subscriptions > Create subscription
    • Input Panther-Audit as the Subscription ID
    • Select the Panther-Audit topic
    • Leave all other options or tune the expiration/retention as needed (per your intended spending/budgeting)
    • Click CREATE
Write down the Topic name (projects/<project-name>/topics/Panther-Audit) and Topic subscription (Panther-Audit); we’ll use it later!

Step 2: Create a Logs Router

Logging Console
  • Open the Logging console
  • Click Logs Router
  • Click CREATE SINK
  • Set the name to Panther-Audit
  • Set the Sink Destination
    • Cloud Pub/Sub topic
    • Select the Panther-Audit Topic
  • Click CREATE SINK
You can validate this pipeline is working by going to Pub/Sub, clicking the Topic ID of Panther-Audit, and viewing the ACTIVITY to see Audit events.
Activity

Step 3: Create a Service Account

  • Open IAM & Admin
  • Click Service Accounts
  • Click CREATE SERVICE ACCOUNT
    • Set the Service account name to Panther-Audit. Add a description if you like.
    • Click Create and Continue
    • Under Grant this service account access to project, set the service account access role to Pub/Sub Viewer and Pub/Sub Subscriber
    • Click Continue
    • Click Done
  • Under Service accounts -> Actions, click Manage keys, ADD KEY, Create new key, select JSON, and hit CREATE to download your credentials.
  • Keep this credential file safe! We’ll use it soon.
Service Account Actions

Step 4: Configure AWS Infrastructure

On the Getting Started with Fluentd page, review and deploy the Firehose & S3 stack

Step 5: Launch Your Instance in AWS

  • Open the AWS EC2 Console (in the same region where you launched the stack above) and launch an Ubuntu Instance.
    • Click Launch Instance
    • Select Ubuntu Server 20.04 LTS
    • Select t2.medium (or a more powerful instance type, if you’d like)
    • In the IAM Role section, select the value of the InstanceProfileName copied in Step 4, with the format “<stack-name>-FirehoseInstanceProfile-<random-string>”
    • Click Add Storage, and add a 64GiB capacity drive
    • Set your Security Group, Key Pair, and other preferences as you’d like
    • Click Launch

Step 6: Install and Configure Fluentd

  • Add your keypair to your ssh agent
    • ssh-add <path-to-keypair>
  • SCP your GCP credentials downloaded in Step 3 to the instance
  • SSH to your newly launched EC2 instance
    • Follow the instructions for Ubuntu Focal
  • Install the official AWS Kinesis plugin
    • sudo td-agent-gem install fluent-plugin-kinesis
  • Install the GCP plugin
    • sudo td-agent-gem install fluent-plugin-gcloud-pubsub-custom
Overwrite the default fluentd config in /etc/td-agent/td-agent.conf:
1
<system>
2
log_level debug
3
</system>
4
5
<source>
6
@type gcloud_pubsub
7
tag gcp.audit
8
project <YOUR-GCP-PROJECT-ID>
9
key <PATH-TO-YOUR-KEYPAIR>
10
topic <PANTHER-AUDIT-TOPIC-ID>
11
subscription <PANTHER-AUDIT-SUBSCRIPTION-ID>
12
max_messages 1000
13
return_immediately true
14
pull_interval 1
15
pull_threads 2
16
parse_error_action exception
17
<parse>
18
@type json
19
</parse>
20
</source>
21
22
<match gcp.**>
23
@type kinesis_firehose
24
region <YOUR-FIREHOSE-REGION>
25
delivery_stream_name <YOUR-FIREHOSE-NAME>
26
27
<assume_role_credentials>
28
duration_seconds 3600
29
role_arn <YOUR-FIREHOSE-ROLE-ARN>
30
role_session_name "#{Socket.gethostname}-panther-audit"
31
</assume_role_credentials>
32
</match>
Copied!
  • Restart td-agent
    • sudo systemctl restart td-agent

Step 7: Onboard data into Panther

Since GCP audit logs are supported out of the box, you may configure the S3 bucket as a data transport to start ingesting the logs through Panther.

Troubleshooting

  • Note: logs may take ~5 minutes to show up in the S3 bucket because of the IntervalInSecondsand SizeInMBs parameters within the CloudFormation template.
  • Monitor the td-agent logs for any errors
    • sudo tail -f /var/log/td-agent/td-agent.log
  • If you need more verbose logging, run:
    • td-agent -vv
Last modified 1mo ago