Syslog to S3 via Fluentd

Overview

This guide provides a method to deliver syslog messages to S3 using Fluentd. There are two different pipeline flows: via an AWS Firehose delivery stream and directly to an AWS S3 bucket.

Prerequisites

This guide assumes that an S3 bucket or Firehose has already been created. If you need to create either of these resources, please see the Getting Started with Fluentd guide. If you have already provisioned the resources, you can adapt the guide below to fit your needs.

Setup Fluentd

Step 1. Install Fluentd

Follow the Fluentd install guide for the environment of the server from which you want to collect syslog messages.

Step 2. Edit Fluentd Configuration

You have two options when configuring Fluentd: using the Firehouse plugin or S3 plugin. Below are the configuration files for both options.
We recommend the Firehose plugin option as it is the more performant of the two, however both will deliver the logs to S3. Two different authentication types are shown in the configuration: assume role and access keys. Use the authentication type that best suits your environment.
Install the following Fluentd plugin:
1
td-agent-gem install fluent-plugin-kinesis
Copied!
Edit the Fluentd configuration/etc/td-agent/td-agent.conf with the below config. This allows Fluentd to listen for syslog events over udp port 5140 and output to Kinesis Firehose. Update the region, delivery_stream_name and role_arn in the configuration below:
1
<source>
2
@type syslog
3
port 5140
4
bind 0.0.0.0
5
tag syslog
6
<parse>
7
message_format auto
8
</parse>
9
</source>
10
11
<filter syslog.** >
12
@type record_transformer
13
<record>
14
tag ${tag}
15
time ${time}
16
</record>
17
</filter>
18
19
<match syslog.**>
20
@type kinesis_firehose
21
region <FIREHOSE-REGION>
22
delivery_stream_name <FIREHOSE-NAME>
23
24
<assume_role_credentials>
25
duration_seconds 3600
26
role_arn <FIREHOSE-ROLE-ARN>
27
role_session_name "#{Socket.gethostname}-panther-audit"
28
</assume_role_credentials>
29
<format>
30
@type json
31
</format>
32
</match>
Copied!

Via S3 Plugin

Edit the Fluentd configuration /etc/td-agent/td-agent.conf with the below config. This will allow Fluentd to listen for syslog events over udp port 5140 and output to a S3 bucket. Update the s3_bucket, s3_region, aws_key_id, and aws_sec_key in the configuration below:
1
<source>
2
@type syslog
3
port 5140
4
bind 0.0.0.0
5
tag syslog
6
<parse>
7
message_format auto
8
</parse>
9
</source>
10
11
<filter syslog.** >
12
@type record_transformer
13
<record>
14
tag ${tag}
15
time ${time}
16
</record>
17
</filter>
18
19
<match syslog.**>
20
@type s3
21
aws_key_id <ACCESS-KEY-ID>
22
aws_sec_key <SECRET-KEY>
23
s3_bucket <BUCKET-NAME>
24
s3_region <BUCKET-REGION>
25
path syslog/%Y/%m/%d/
26
store_as gzip
27
<buffer tag,time>
28
@type file
29
path /var/log/td-agent/buffer/s3
30
timekey 300 # 5 min partition
31
timekey_wait 2m
32
timekey_use_utc true # use utc
33
chunk_limit_size 256m
34
</buffer>
35
<format>
36
@type json
37
</format>
38
</match>
Copied!

Step 3: Start Fluentd

After configuring Fluentd, start it by running the below command:
1
$ sudo systemctl start td-agent.service
Copied!
Verify that Fluentd is running:
1
$ sudo systemctl status td-agent.service
Copied!
See the Fluentd install guide on starting Fluentd in your environment if systemctl is not available.

Configure rsyslog

Configure rsyslog to forward to local Fluentd

Configure rsyslog to forward messages to the local Fluentd daemon by adding these two lines to the bottom of /etc/rsyslog.d/50-default.conf or /etc/rsyslog.conf in some environments:
1
# Send log messages to Fluentd
2
*.* @127.0.0.1:5140
Copied!
Restart rsyslog with the below command:
1
$ sudo systemctl restart rsyslog.service
Copied!
This step can be duplicated for other servers to forward syslog to the Fluentd server that was previously configured in this guide. Just replace the local address *.* @127.0.0.1:5140 with the IP address of the Fluentd server. You may need to update security groups or host-based firewalls to allow sending udp/5140 traffic to the server.

Verify Logging

After 5-10 minutes have passed, verify that syslog messages are being logged to the S3 bucket. Logs should be showing up under the syslog/ prefix within the bucket.
You can now onboard the data in the Panther UI by onboarding the S3 bucket and using the Fluentd.Syslog3164 log type.
Last modified 2mo ago