Managing AWS S3 Log Sources with Terraform

Manage S3 log sources as code in Terraform

Overview

You can define your S3 log source—meaning you can create the S3 bucket and associated infrastructure in AWS, and onboard it to your Panther instance—all in Terraform. Panther is a Terraform provider.

Other methods to create an S3 log source include using the Panther API directly and manual creation in the Panther Console.

How to define your Panther S3 log source in Terraform

The following sections outline how to define your S3 log source in HashiCorp Configuration Language (HCL). You will define both AWS and Panther infrastructure in HCL.

Prerequisite

Before starting, ensure you have a GraphQL API URL and token with the Manage Log Sources permission. This is required to complete Step 4.
- Follow these instructions for creating an API token in the Panther Console.

Step 1: Define variables

Define a *.tfvars file with the following AWS and Panther variables.

variable "aws_account_id" {
  type        = string
  description = "The AWS account ID where the template is being deployed"
}

variable "panther_aws_account_id" {
  type        = string
  description = "The AWS account ID of your Panther instance"
}

variable "panther_aws_region" {
  type = string
  default = "us-east-1"
  description = "The region where the Panther instance is deployed"
}

variable "panther_aws_partition" {
  type        = string
  default     = "aws"
  description = "AWS partition of the account running the Panther backend e.g aws, aws-cn, or aws-us-gov"
}

variable "s3_bucket_name" {
  type        = string
  description = "The S3 Bucket name to onboard"
}

variable "log_source_name" {
  type = string
  description = "The name of the log source to be created in Panther"
}

variable "panther_api_token" {
  type = string
}

variable "panther_api_url" {
  type = string
}

Step 2: Define Terraform providers

Include both the the AWS and Panther Terraform providers.

terraform {
  required_providers {
    panther = {
      source = "panther-labs/panther"
    }
    aws = {
      source  = "hashicorp/aws"
    }
  }
}

Step 3: Define AWS infrastructure

In AWS, you need to create an S3 bucket and an SNS topic. To ingest logs from an S3 bucket, the bucket must write notifications to an SNS topic on object creation. A subscription on this topic is needed to push object information onto Panther's input queue.

Define S3 bucket

The following HCL configuration defines the S3 bucket and associated IAM role for accessing its contents. This role requires read permissions on the S3 bucket, as it will be assumed by your Panther instance to read incoming logs.

resource "aws_s3_bucket" "log_bucket" {
  bucket = var.s3_bucket_name
}

resource "aws_iam_role" "log_processing_role" {
  name = "PantherLogProcessingRole-${var.s3_bucket_name}"

  # Policy that grants an entity permission to assume the role.
  assume_role_policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Action : "sts:AssumeRole",
        Effect : "Allow",
        Principal : {
          AWS : "arn:${var.aws_partition}:iam::${var.panther_aws_account_id}:root"
        }
        Condition : {
          Bool : { "aws:SecureTransport" : true }
        }
      }
    ]
  })

  tags = {
    Application = "Panther"
  }
}


# Provides an IAM role inline policy for reading S3 Data
resource "aws_iam_role_policy" "read_data_policy" {
  name = "ReadData"
  

  role = aws_iam_role.log_processing_role.id
  policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Effect : "Allow",
        Action : [
          "s3:GetBucketLocation",
          "s3:ListBucket",
        ],
        Resource : "arn:${var.aws_partition}:s3:::${aws_s3_bucket.log_bucket.bucket}"
      },
      {
        Effect : "Allow",
        Action : "s3:GetObject",
        Resource : "arn:${var.aws_partition}:s3:::${aws_s3_bucket.log_bucket.bucket}/*"
    }, ]
  })
}

The following HCL configuration creates the SNS topic and related policy for enabling S3 bucket notifications. It also creates a subscription to forward messages to Panther's input data notifications queue.

The same SNS topic can be used for multiple S3 buckets integrations.

resource "aws_sns_topic" "panther_notifications_topic" {
  name = "panther-notifications-topic"
}

resource "aws_sns_topic_policy" "default" {
  arn = aws_sns_topic.panther_notifications_topic.arn

  policy = data.aws_iam_policy_document.panther_notifications_topic_policy.json
}

data "aws_iam_policy_document" "panther_notifications_topic_policy" {
  statement {
    sid = "AllowS3EventNotifications"
    actions = [
      "sns:Publish",
    ]
    effect = "Allow"
    principals {
      type = "Service"
      identifiers = ["s3.amazonaws.com"]
    }
    resources = [
      aws_sns_topic.panther_notifications_topic.arn,
    ]
  }
  statement {
    sid = "AllowCloudTrailNotification"
    actions = [
      "sns:Publish",
    ]
    effect = "Allow"
    principals {
      type = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    resources = [
      aws_sns_topic.panther_notifications_topic.arn,
    ]
  }
  statement {
    sid = "AllowSubscriptionToPanther"
    actions = [
      "sns:Subscribe",
    ]
    effect = "Allow"
    principals {
      type = "AWS"
      identifiers = ["arn:${var.panther_aws_partition}:iam::${var.panther_aws_account_id}:root"]
    }
    resources = [
      aws_sns_topic.panther_notifications_topic.arn,
    ]
  }
}

resource "aws_s3_bucket_notification" "panther_event_notifications" {
  bucket = aws_s3_bucket.log_bucket.id

  topic {
    topic_arn = aws_sns_topic.panther_notifications_topic.arn
    events        = ["s3:ObjectCreated:*"]
  }
}

resource "aws_sns_topic_subscription" "panther_notifications_subscription" {
  topic_arn = aws_sns_topic.panther_notifications_topic.arn
  protocol  = "sqs"
  endpoint  = "arn:${var.panther_aws_partition}:sqs:${var.panther_aws_region}:${var.panther_aws_account_id}:panther-input-data-notifications-queue"
  raw_message_delivery = false
}

Step 4: Define Panther S3 log source

The following HCL configuration defines the S3 log source in Panther. Note that to complete this section, you will need the API URL and token outlined in the Prerequisite section.

Note that panther_managed_bucket_notifications_enabled is set to false. This indicates that all of the infrastructure related to this log source is being managed externally, in this case through Terraform.

provider "panther" {
  token = var.panther_api_token
  url   = var.panther_api_url
}

resource "panther_s3_source" "demo_source" {
  aws_account_id                          = var.aws_account_id
  name                                    = var.log_source_name
  log_processing_role_arn                 = aws_iam_role.log_processing_role.arn
  log_stream_type                         = "JSON"
  panther_managed_bucket_notifications_enabled = false
  bucket_name                             = aws_s3_bucket.log_bucket.bucket
  prefix_log_types = [{
    excluded_prefixes = []
    log_types         = ["AWS.CloudTrail"]
    prefix            = ""
  }]
}

PreviousTerraform Nextpantherlog Tool

Last updated 8 months ago