April 12, 2024

AWS

When NotFound Errors are plenty in AWS CloudTrail! Exploring Solutions and Best Practices

Shintaro Kurihara

Cover Image for When NotFound Errors are plenty in AWS CloudTrail! Exploring Solutions and Best Practices

Hey, I found a bunch of NotFound error events in AWS CloudTrail!

Hello. I am Kurihara from the Cloud Center of Excellence (CCoE) team at KINTO Technologies, who couldn't dislike alcohol even after watching the Japanese series "Drinking Habit 50." As Tada from my team previously introduced CCoE Activities and Providing Google Cloud Security Preset Environments, we are working every day to keep our cloud environment secure. While analyzing AWS CloudTrail logs to check the AWS health of our account, I noticed that there were a lot of NotFound-type errors on a regular basis. This may sound boring, but if you are an AWS user, chances are you have encountered the same event. Despite searching extensively on Google, I couldn't find any relevant information, so I decided to document my investigation through a blog post.

Conclusion

Overall, when analyzing AWS CloudTrail, NotFound-type errors via the service link crawl in the AWS Config recorder should be excluded and analyzed. Error events inevitably occur due to the behavior of AWS Config, so they should be properly filtered to reduce analysis noise.

Details of Investigation

KINTO Technologies has a multi-account configuration where Landing Zones are managed in AWS Control Tower in accordance with best practices for AWS multi-account management. Therefore, AWS Config manages configuration information and AWS CloudTrail manages audit logs.

While analyzing AWS CloudTrail logs to check the AWS health of our account, I found that NotFound-type error events were occurring in large numbers and on a regular basis.

Here are the results of the AWS Athena analysis of CloudTrail logs for about a month from a certain AWS account. This account is issued with minimal security settings and no workload has been built.

-- Analyze the top of errorCode
WITH filterd AS (
  SELECT
    *
  FROM
    cloudtrail_logs
  WHERE
    errorCode IS NOT NULL
)
SELECT
  errorCode,
  count(errorcode) as eventCount,
  count(errorCode) * 100 / (select count(*) from filterd) as errorRate
FROM
  filterd
GROUP BY

errorCode	eventCount	errorRate
ResourceNotFoundException	1,515	18
ReplicationConfigurationNotFoundError	1,112	13
ObjectLockConfigurationNotFoundError	958	11
NoSuchWebsiteConfiguration	954	11
NoSuchCORSConfiguration	952	11
InvalidRequestException	627	7
Client.RequestLimitExceeded	609	7

-- Check the frequency of occurrence of a specific erroCode
SELECT
  date(from_iso8601_timestamp(eventtime)) as "date"
  count(*) as count
FROM
  cloudtrail_logs
WHERE
  errorcode = 'ResourceNotFoundException'
GROUP BY
  date(from_iso8601_timestamp(eventtime))
ORDER BY
  "date" ASC
LIMIT 5

date	count
2023-10-19	52
2023-10-20	80
2023-10-21	80
2023-10-22	80
2023-10-23	80

I picked up a few error codes and looked at the AWS CloudTrail records (the actual AWS CloudTrail logs are listed at the end of this article) and found that all of them were recorded in the arn field of the userIdentity that was the access source as arn:aws:sts::${AWS_ACCOUNT_ID}:assumed-role/AWSServiceRoleForConfig/${SESSION_NAME}. This is the Service-Linked Roles attached to AWS Config. I could not figure out why NotFound would occur even though the target resource exists, but when I checked the eventName section, I realized that it is not an API to get configuration information of the resource itself, but rather for each of its dependent resources.

Resource	errorCode	API that was called (eventName)
Lambda	ResourceNotFoundException	GetPolicy20150331v2
S3	ReplicationConfigurationNotFoundError	GetBucketReplication
S3	NoSuchCORSConfiguration	GetBucketCors

Although it is not an error that affects the workload, we would like to eliminate it as it is noise in general monitoring and troubleshooting. To do so, we need to take non-essential actions such as "configure something in the related resource" (for example, adding a Lambda resource-based policy that allows InvokeFunction actions only from its own account).

We came to the corresponding conclusion that our CCoE team excludes access from the AWS Config service-linked role when analyzing AWS CloudTrail. If you analyze with AWS Athena, it is an image of executing the following query.

SELECT
  *
FROM
  cloudtrail_logs
WHERE
  userIdentity.arn not like '%AWSServiceRoleForConfig%'

A Brief Deep Dive

I will delve a bit further into the process of recording configuration information in AWS Config, based on insights gained during this investigation. There are two points that are not explicitly stated in the official documentation, but were found in this investigation.

Dependent (supplemental) resources (I named it myself) recording behavior
Frequency of recording dependent (supplemental) resources

Dependent (supplemental) resource recording behavior

AWS Config not only records configuration information of the resource itself, but also related resources (relationships). They are named direct relationship and indirect relationship.

AWS Config derives the relationships for most resource types from the configuration field, which are called "direct" relationships. A direct relationship is a one-way connection (A→B) between a resource (A) and another resource (B), typically obtained from the describe API response of resource (A). In the past, for some resource types that AWS Config initially supported, it also captured relationships from the configurations of other resources, creating "indirect" relationships that are bidirectional (B→A). For example, the relationship between an Amazon EC2 instance and its security group is direct because the security groups are included in the describe API response for the Amazon EC2 instance. On the other hand, the relationship between a security group and an Amazon EC2 instance is indirect because describing a security group does not return any information about the instances it is associated with. As a result, when a resource configuration change is detected, AWS Config not only creates a CI for that resource, but also generates CIs for any related resources, including those with indirect relationships. For example, when AWS Config detects changes in an Amazon EC2 instance, it creates a CI for the instance and a CI for the security group that is associated with the instance.

-- https://docs.aws.amazon.com/config/latest/developerguide/faq.html#faq-1

There are resources, which I name them on my own, dependent (supplemental) resource, that are separate from related resources and appear to be settings for the resource itself, but they also have separate acquisition APIs. In the case of Lambda, Lambda itself is a resource that can be obtained with GetFunction, whereas resource-based policy is another resource that can be obtained with GetPolicy. Looking at the Configuration Item (CI), the resource-based policy that is a dependent (supplemental) resource, is recorded in the supplementaryConfiguration field as follows:

{
  "version": "1.3",
  "accountId": "<$AWS_ACCOUNT_ID>",
  "configurationItemCaptureTime": "2023-12-15T09:52:19.238Z",
  "configurationItemStatus": "OK",
  "configurationStateId": "************",
  "configurationItemMD5Hash": "",
  "arn": "arn:aws:lambda:ap-northeast-1:<$AWS_ACCOUNT_ID>:function:check-config-behavior",
  "resourceType": "AWS::Lambda::Function",
  "resourceId": "check-config-behavior",
  "resourceName": "check-config-behavior",
  "awsRegion": "ap-northeast-1",
  "availabilityZone": "Not Applicable",
  "tags": {
    "Purpose": "investigate"
  },
  "relatedEvents": [],

  # Related resources
  "relationships": [
    {
      "resourceType": "AWS::IAM::Role",
      "resourceName": "check-config-behavior-role-nkmqq3sh",
      "relationshipName": "Is associated with "
    }
  ],

  ... Omitted

  # Dependent (supplemental) resources
  "supplementaryConfiguration": {
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"test-poilcy\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::<$AWS_ACCOUNT_ID>:root\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:ap-northeast-1:<$AWS_ACCOUNT_ID>:function:check-config-behavior\"}]}",
    "Tags": {
      "Purpose": "investigate"
    }
  }
}

Frequency of recording dependent (supplemental) resources

The frequency of recording CIs in AWS Config depends on the setting of RecordingMode, but this does not seem to be the case for dependent (supplemental) resources. If it was a NotFound-type error, it may have been due to retry attempts. However, the observed behavior indicated that recording was attempted once every 12 or 24 hours. This also does not seem to be a regularity subject to the type of dependent (supplemental) resources. This is the result of my investigation, although it is quite a black box behavior.

Summary

The above introduced the identity of the mysterious NotFound-type error events output to AWS CloudTrail and countermeasures. The details will be further investigated in the future, but it has been confirmed that similar error events are occurring from the service-linked roles in Macie. Although AWS CloudTrail analysis is a tedious task, it is also an opportunity to gain a deeper understanding of the behavior of AWS services. Therefore, let's perform it proactively! For engineers who want to leverage AWS to its fullest, and who think Keisuke Koide is a talented actor, the Platform Group is currently seeking to hire you!

Finally, I will conclude this article by listing each AWS CloudTrail error event. Thank you for reading.

Lambda: ResourceNotFoundException

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "************:LambdaDescribeHandlerSession",
    "arn": "arn:aws:sts::<$AWS_ACCOUNT_ID>:assumed-role/AWSServiceRoleForConfig/LambdaDescribeHandlerSession",
    "accountId": "<$AWS_ACCOUNT_ID>",
    "accessKeyId": "*********",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "*********",
        "arn": "arn:aws:iam::<$AWS_ACCOUNT_ID>:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig",
        "accountId": "<$AWS_ACCOUNT_ID>",
        "userName": "AWSServiceRoleForConfig"
      },
      "webIdFederationData": {},
      "attributes": {
        "creationDate": "2023-12-03T09:09:17Z",
        "mfaAuthenticated": "false"
      }
    },
    "invokedBy": "config.amazonaws.com"
  },
  "eventTime": "2023-12-03T09:09:19Z",
  "eventSource": "lambda.amazonaws.com",
  "eventName": "GetPolicy20150331v2",
  "awsRegion": "ap-northeast-1",
  "sourceIPAddress": "config.amazonaws.com",
  "userAgent": "config.amazonaws.com",
  "errorCode": "ResourceNotFoundException",
  "errorMessage": "The resource you requested does not exist.",
  "requestParameters": {
    "functionName": "**************"
  },
  "responseElements": null,
  "requestID": "******************",
  "eventID": "******************",
  "readOnly": true,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "<$AWS_ACCOUNT_ID>",
  "eventCategory": "Management"
}

S3: ReplicationConfigurationNotFoundError

{
  "eventVersion": "1.09",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "**********:AWSConfig-Describe",
    "arn": "arn:aws:sts::<$AWS_ACCOUNT_ID>:assumed-role/AWSServiceRoleForConfig/AWSConfig-Describe",
    "accountId": "<$AWS_ACCOUNT_ID>",
    "accessKeyId": "*************",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "*************",
        "arn": "arn:aws:iam::<$AWS_ACCOUNT_ID>:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig",
        "accountId": "<$AWS_ACCOUNT_ID>",
        "userName": "AWSServiceRoleForConfig"
      },
      "attributes": {
        "creationDate": "2023-12-03T13:09:16Z",
        "mfaAuthenticated": "false"
      }
    },
    "invokedBy": "config.amazonaws.com"
  },
  "eventTime": "2023-12-03T13:09:55Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "GetBucketReplication",
  "awsRegion": "ap-northeast-1",
  "sourceIPAddress": "config.amazonaws.com",
  "userAgent": "config.amazonaws.com",
  "errorCode": "ReplicationConfigurationNotFoundError",
  "errorMessage": "The replication configuration was not found",
  "requestParameters": {
    "replication": "",
    "bucketName": "*********",
    "Host": "*************"
  },
  "responseElements": null,
  "additionalEventData": {
    "SignatureVersion": "SigV4",
    "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "bytesTransferredIn": 0,
    "AuthenticationMethod": "AuthHeader",
    "x-amz-id-2": "**************",
    "bytesTransferredOut": 338
  },
  "requestID": "**********",
  "eventID": "*************",
  "readOnly": true,
  "resources": [
    {
      "accountId": "<$AWS_ACCOUNT_ID>",
      "type": "AWS::S3::Bucket",
      "ARN": "arn:aws:s3:::***********"
    }
  ],
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "<$AWS_ACCOUNT_ID>",
  "vpcEndpointId": "vpce-***********",
  "eventCategory": "Management"
}

S3: NoSuchCORSConfiguration

{
  "eventVersion": "1.09",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "***********:AWSConfig-Describe",
    "arn": "arn:aws:sts::<$AWS_ACCOUNT_ID>:assumed-role/AWSServiceRoleForConfig/AWSConfig-Describe",
    "accountId": "<$AWS_ACCOUNT_ID>",
    "accessKeyId": "***************",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "*************",
        "arn": "arn:aws:iam::<$AWS_ACCOUNT_ID>:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig",
        "accountId": "<$AWS_ACCOUNT_ID>",
        "userName": "AWSServiceRoleForConfig"
      },
      "attributes": {
        "creationDate": "2023-12-03T13:09:16Z",
        "mfaAuthenticated": "false"
      }
    },
    "invokedBy": "config.amazonaws.com"
  },
  "eventTime": "2023-12-03T13:09:55Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "GetBucketCors",
  "awsRegion": "ap-northeast-1",
  "sourceIPAddress": "config.amazonaws.com",
  "userAgent": "config.amazonaws.com",
  "errorCode": "NoSuchCORSConfiguration",
  "errorMessage": "The CORS configuration does not exist",
  "requestParameters": {
    "bucketName": "********",
    "Host": "*************************8",
    "cors": ""
  },
  "responseElements": null,
  "additionalEventData": {
    "SignatureVersion": "SigV4",
    "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "bytesTransferredIn": 0,
    "AuthenticationMethod": "AuthHeader",
    "x-amz-id-2": "*********************",
    "bytesTransferredOut": 339
  },
  "requestID": "***********",
  "eventID": "*****************",
  "readOnly": true,
  "resources": [
    {
      "accountId": "<$AWS_ACCOUNT_ID>",
      "type": "AWS::S3::Bucket",
      "ARN": "arn:aws:s3:::*************"
    }
  ],
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "<$AWS_ACCOUNT_ID>",
  "vpcEndpointId": "vpce-********",
  "eventCategory": "Management"
}

We are hiring!

【クラウドセキュリティエンジニア】SCoE G／東京・大阪

Security Center of Excellence ( SCoE ) グループについてマルチクラウド ( AWS, Google Cloud, Azure ) 環境のセキュリティガバナンスを担当しています。KINTOテクノロジーズ内だけでなく、グループ内の関連組織とも協力しながら、業務に行います。

【データエンジニア】分析G／東京・名古屋・大阪

分析グループについてKINTOにおいて開発系部門発足時から設置されているチームであり、それほど経営としても注力しているポジションです。決まっていること、分かっていることの方が少ないぐらいですので、常に「なぜ」を考えながら、未知を楽しめるメンバーが集まっております。