Overview

There are two methods of WAF log retrieval: the new method and the old method.

The new method allows users to enable WAF log-related features by opting in with a WAF log retrieval checkbox.

The old method is the same as the one available on the old WafCharm Dashboard, which uses Lambda to transfer WAF logs.

If you are using the Legacy rule policy, you can use both, but it is recommended to switch to the new method.

WAF log retrieval is optional.

There are two ways to configure if you choose the old method.

• Exporting WAF logs directly to an S3 bucket
• Using Amazon Data Firehose to export WAF logs to an S3 bucket

Please keep in mind that WAF log retrieval does not support WAF logs outputted via CloudWatch Logs.

Preparations

To use the old method, you are required to create an IAM role and configure Lambda on the AWS management console. Therefore, the AWS account you use on the AWS management console must have the permissions listed below.

If you have permissions equivalent to the ones listed below, you should be able to complete all the steps.

• Exporting WAF logs directly to an S3 bucket
  • AWSLambda_FullAccess
  • IAMFullAccess
  • CloudWatchFullAccess
• Using Amazon Data Firehose to export WAF logs to an S3 bucket
  • AWSLambda_FullAccess
  • IAMFullAccess
  • CloudWatchFullAccess
  • AmazonKinesisFullAccess

Notes

• It is recommended that you periodically (e.g., every month) transfer WAF logs outputted to your S3 bucket to S3 Glacier or delete them using the life cycle feature.
• If you want to confirm that the WAF logs are successfully transferred to WafCharm, please check the two points below, and send the latest WAF log file name and the target Web ACL ID to the WafCharm support team.
  • WAF logs are outputted to your S3 buckets.
  • There are no error logs in the CloudWatch event logs (START/END/REPORT lines should be logged when an event is triggered)
    • How to check for an error
    • Open CloudWatch -> Open Log groups -> Open /aws/lambda/Lambda function name -> Click on the latest log stream (available at the top of the list) -> Check if any lines contain the word "ERROR"
• If you edit the IAM role/policy after you have deployed the Lambda, the changes made to roles or policies may not be applied to the working Lambda scripts. In that case, please add an empty line at the bottom of the index.mjs and click the [Deploy] button again.
• If you have registered WAF Configs (web ACLs) for different AWS WAF versions (Classic or v2) on WafCharm and wish to use this feature for both WAF Configs, please prepare separate Amazon Data Firehose and Lambda for each AWS WAF version.
  • If the AWS WAF version is the same, you can share Amazon Data Firehose and Lambda between multiple web ACLs.
  • The content of index.mjs for AWS WAF v2 and AWS WAF Classic are different.
• Log filtering provided by AWS WAF is not recommended if you wish to use this feature.
  • If you filter WAF logs to only save logs with action: Block
    • Only WAF logs with blocked requests are available, so the monthly report and detection notification operate based on the blocked logs.
  • If you filter WAF logs to only save logs with action: Count
    • The monthly report and the detection notification will not work properly because the Count action for a single rule and rule group are treated differently by AWS WAF in WAF logs.

Procedure

Exporting WAF logs directly to an S3 bucket

How to use the feature

The architecture overview is as follows.

AWS WAF will output WAF logs directly to an S3 bucket on your AWS environment.

When WAF logs are output to an S3 bucket (S3 putObject Event Trigger), it triggers the Lambda to execute the script.

The data is transferred to WafCharm and monthly reports can be created and viewed without any additional configurations at the beginning of each month.

If you configure email notification settings for WafCharm rule detections on the WafCharm Console, you can use the WAF log alert (notification detection) feature.

Output WAF logs on AWS WAF

1. Sign in to the AWS management console.
2. Open the list of web ACLs on AWS WAF.

   

3. Open the target web ACL.
4. Click [Logging and metrics].

   

5. Click [Enable].

   

6. Select [S3 bucket] under [Logging destination].

   

7. Click [Create new] on the [Amazon S3 bucket] section.
8. Enter a name that begins with [aws-waf-logs-] in [Bucket name].

   Per AWS specification, you are required to add [aws-waf-logs-] at the beginning of the S3 bucket name.

   

9. Go back to the AWS WAF page and select the S3 bucket you just created from the drop-down menu.

   

10. Click [Save].

    

11. Access the target resources and make sure that WAF logs are outputted to the expected S3 bucket.
12. Open the S3 bucket and check the path where WAF logs are outputted.

    The format should be like below. You will use this information later.

    aws-waf-logs-[specified bucket name]/AWSLogs/account-id/WAFLogs/Region/web-acl-name

    Other than the S3 bucket name, account-id, Region, and web-acl-name should change based on your environment.

    

Create an IAM policy and role used by Lambda

You will be creating the three items below.

• Read permissions to your S3 bucket.
• Put permissions to transfer WAF logs to WafCharm's S3 bucket.
• The IAM role with the permissions above to be used by Lambda.

1. Open IAM.
2. Create a Read permission to your S3 bucket.
   1. Open [Policies] and click [Create policy].

      

   2. Configure the options below.
      • Service: S3
      • Action: GetObject
      • Resources: arn:aws:s3:::aws-waf-logs-指定したバケット名/AWSLogs/account-id/WAFLogs/Region/web-acl-name/*

        Write the S3 bucket path you obtained in the previous steps. The path specified under the [Resources] must end with /* .

      

   3. Proceed to the Review policy page and enter the policy name and description (optional).

      The name can be any value.

      Example: wafcharm-waflog-s3-read

      

3. Create a Put permissions to transfer WAF logs to WafCharm's S3 bucket.
   1. Open [Policies] and click [Create policy].
   2. Configure the options below.
      • Service: S3
      • Action: PutObject, PutObjectACL
      • Resources: arn:aws:s3:::wafcharm.com/*

        Specify WafCharm's S3 bucket to transfer WAF logs.

      

   3. Proceed to the Review policy page and enter the policy name and description (optional).

      The name can be any value.

      Example: wafcharm-waflog-s3-put

      

4. Create an IAM role for Lambda.
   1. Open [Roles] and click [Create role].

      

   2. Choose [Lambda] for [Use case].
   3. Search for the permissions below under [Permissions policies] and check to add them.
      • AWSLambdaExecute
      • Read permission created in the previous steps (e.g.: wafcharm-waflog-s3-read)
      • Put permission created in the previous steps (e.g.: wafcharm-waflog-s3-put)

      

   4. Proceed to the next page and enter the role name and description (optional).

      The name can be any value.

      Example: wafcharm-waflog

Configure Lambda

1. Switch to the same region as your S3 bucket.
2. Open Lambda.
3. Click [Create function].
4. Enter [Function name].

   The name can be any value.

   Example: wafcharm-waflog

5. Configure the options below.

   Runtime: Node.js 18.x ~ Node.js 20.x

   Execution Role: Use an existing role

   Existing role: Select the IAM role you created in the previous steps (e.g.: wafcharm-waflog)

6. Click [Create function].
7. Paste the index.mjs available below in the editor under the [Code source].

   index.mjs

   

8. Click the [Deploy] button.
9. Click the [Add trigger] button.
10. Configure the options below.

    Select a source: S3

    Bucket: the S3 bucket you created in the previous steps

    Event types: All object create events

    Prefix: AWSLogs/account-id/WAFLogs/Region/web-acl-name/

    The account-id, Region, and web-acl-name should change based on your environment.

    

11. Click [Add].
12. Click [Configuration].
13. Click the [Edit] button to the right of [General configuration].
14. Configure the options below.

    Description (optional)

    Timeout: 1 minute

    

15. Click [Save].

    

Configure retention period for Lambda's execution logs on CloudWatch Logs

This configuration is optional.

If you want to change the retention period for your logs, please follow the steps below.

1. Execute the Lambda script.

   Access the target resources to output WAF logs. The Lambda script is triggered when the WAF logs are outputted to the S3 buckets.

2. Open CloudWatch.
3. Click [Log groups].
4. Check the checkbox of the Lambda's log group.
5. Click [Edit retention setting(s)] from the [Action] menu.
6. Change the period under the [Expire events after:] drop-down menu.

   The default is set to [Never expire]. Please adjust the period as necessary.

Using Amazon Data Firehose to export WAF logs to an S3 bucket

How to use the feature

The architecture overview is as follows.

AWS WAF will output WAF logs to an S3 bucket via Amazon Data Firehose on your AWS environment.

When WAF logs are output to an S3 bucket (S3 putObject Event Trigger), it triggers the Lambda to execute the script.

The data is transferred to WafCharm and monthly reports can be created and viewed without any additional configurations at the beginning of each month.

If you configure email notification settings for WafCharm rule detections on the WafCharm Console, you can use the WAF log alert (notification detection) feature.

Configure Amazon Data Firehose

1. Sign in to the AWS management console.
2. Switch to the same region as your web ACL.

   If you are attaching CloudFront to your web ACL, select US East (N. Virginia).

3. Open [Amazon Data Firehose].
4. Click [Create Firehose stream].

   

5. Select the options below under the [Choose source and destination] section.
   • Source: Direct PUT
   • Destination: Amazon S3
   • Firehose stream name: enter a name that begins with [aws-waf-logs-]

     Per AWS specification, you are required to add [aws-waf-logs-] in the beginning of the stream name.

   • S3 bucket: select any S3 bucket folder you wish to use (e.g.: csc-waftest)
   • S3 bucket prefix (optional): enter a prefix (e.g.: waflog/)

     If you want to add a prefix, make sure to add a slash ( / ) at the end.

   • S3 bucket and S3 error output prefix time zone: UTC

     Please choose UTC here because the feature may not work properly with different timezone.

   • Buffer size: 5MB is recommended

     Buffer interval: 60 seconds is recommended

     *WAF logs are exported to the S3 bucket when they reach the specified buffer interval or buffer size.

   • Compression for data records: GZIP

   

   

6. Create or select service access (IAM role).

   

7. Click [Create Firehose stream].

Output WAF logs on AWS WAF

1. Open the list of web ACLs on AWS WAF.

   

2. Open the target web ACL.
3. Click [Logging and metrics].

   

4. Click [Enable].

   

5. Select [Amazon Data Firehose] under [Logging destination].

   

6. Select the Data Firehose you just created.
7. Click [Save].
8. Access the target resources and make sure that WAF logs are outputted to the expected S3 bucket.
9. Open the S3 bucket and check the path where WAF logs are outputted.

   The format should be like below. You will use this information later.

   csc-waftest/waflog/

   The S3 bucket name and prefix (or lack thereof) will depend on the options you chose in the previous steps.

   

Create an IAM policy and role used by Lambda

You will be creating the three items below.

• Read permissions to your S3 bucket.
• Put permissions to transfer WAF logs to WafCharm's S3 bucket.
• The IAM role with the permissions above to be used by Lambda.

1. Open IAM.
2. Create a Read permission to your S3 bucket.
   1. Open [Policies] and click [Create policy].

      

   2. Configure the options below.
      • Service: S3
      • Action: GetObject
      • Resources: arn:aws:s3:::csc-waftest/waflog/*

        Write the S3 bucket path you obtained in the previous steps. The path specified under the [Resources] must end with /* .

      

   3. Proceed to the Review policy page and enter the policy name and description (optional).

      The name can be any value.

      Example: wafcharm-waflog-s3-read

      

3. Create a Put permissions to transfer WAF logs to WafCharm's S3 bucket.
   1. Open [Policies] and click [Create policy].
   2. Configure the options below.
      • Service: S3
      • Action: PutObject, PutObjectACL
      • Resources: arn:aws:s3:::wafcharm.com/*

        Enter WafCharm's S3 bucket (as specified above) to enable WAF log transfer. The path specified under the [Resources] must end with /* .

      

   3. Proceed to the Review policy page and enter the policy name and description (optional).

      The name can be any value.

      Example: wafcharm-waflog-s3-put

      

4. Create an IAM role for Lambda.
   1. Open [Roles] and click [Create role].
   2. Choose [Lambda] for [Use case].

      

   3. Search for the permissions below under [Permissions policies] and check to add them.
      • AWSLambdaExecute
      • Read permission created in the previous steps (e.g.: wafcharm-waflog-s3-read)
      • Put permission created in the previous steps (e.g.: wafcharm-waflog-s3-put)

      

      

   4. Proceed to the next page and enter the role name and description (optional).

      The name can be any value.

      Example: wafcharm-waflog

      

Configure Lambda

1. Switch to the same region as your S3 bucket.
2. Open Lambda.
3. Click [Create function].
4. Enter [Function name].

   The name can be any value.

   Example: wafcharm-waflog

5. Configure the options below.

   Runtime: Node.js 18.x ~ Node.js 20.x

   Execution Role: Use an existing role

   Existing role: Select the IAM role you created in the previous steps (e.g.: wafcharm-waflog)

6. Click [Create function].
7. Paste the index.mjs available below in the editor under the [Code source].

   index.mjs

   

8. Click the [Deploy] button.
9. Click the [Add trigger] button.
10. Configure the options below.

    Select a source: S3

    Bucket: the S3 bucket you created in the previous steps

    Event types: All object create events

    Prefix: prefix of the S3 bucket you created in the previous steps (e.g.: waflog/ )

    

11. Click [Add].
12. Click [Configuration].
13. Click the [Edit] button to the right of [General configuration].
14. Configure the options below.

    Description (optional)

    Timeout: 1 minute

    

15. Click [Save].

    

Configure retention period for Lambda's execution logs on CloudWatch Logs

This configuration is optional.

If you want to change the retention period for your logs, please follow the steps below.

1. Execute the Lambda script.

   Access the target resources to output WAF logs. The Lambda script is triggered when the WAF logs are outputted to the S3 buckets.

2. Open CloudWatch.
3. Click [Log groups].
4. Check the checkbox of the Lambda's log group.
5. Click [Edit retention setting(s)] from the [Action] menu.
6. Change the period under the [Expire events after:] drop-down menu.

   The default is set to [Never expire]. Please adjust the period as necessary.