Amazon EventBridge: How To Manage API Connection Credentials With AWS Secrets Manager

Original Photo by alexandru vicol on Unsplash

In my previous article, Amazon EventBridge: API Destinations Demystified-Part I, we discussed API Destinations and its components. We looked at creating a Connection with OAuth credentials and configuring a routing rule with API Destination as the target.

In that, I mentioned Amazon EventBridge uses AWS Secrets Manager to store API credentials. However, Secrets Manager’s pricing can become a concern when a high volume of events gets routed to multiple destinations.

The other concern is about protecting the credentials without exposing them to the infrastructure code.

This article aims to address the following.

• How to minimize Secret Manager’s cost by working with EventBridge?
• How to protect the credentials associated with API Destinations?

The Cost Of AWS Secrets Manager

Source: AWS Console

AWS Secrets Manager is a service to store and manage secrets. It also enables to rotate credentials such as API keys, access tokens, and secrets. We can use its APIs to fetch the secrets at our application’s runtime, thus preventing the exposure of sensitive data.

What’s in a Secret?

A secret in Secrets Manager is not a single value but a collection of key-value pairs of data represented as a JSON string, known as SecretString.

For example, to store the OAuth credentials for the CustomerAccounts connection we saw in Part 1, we create a secret called CustomerAccountsClientSecret and store the ClientID and ClientSecretand APIKey.

Secret name: CustomerAccountsClientSecret

Secret string:

{
  "ClientID": "shdg3bsdu8lhxvs",
  "ClientSecret": "ab@ls-r£x8M9",
  "APIKey": "hfvbdifusdf"
}

To retrieve a secret, we supply the secret name to the GetSecretValue API, and it returns the result. From the result, we get the secret as a JSON string. We can then pick the required value from the SecretString.

Note: If the secret is stored as binary, then you need to get the binary value and decode it.

How much we pay?

To be frank, Secrets Manager’s pricing can be a big disappointment for many. For an average environment with 100s of secrets and millions of requests to fetch them, the monthly cost can be substantial.

• Monthly cost of a secret = $0.40 (per month)
• Cost of 10,000 Secrets Manager API calls = $0.05

For example, say, there are 100 secrets and every day EventBridge routes 20,000 events to lambda targets. Lambda functions then invoke 3rd party APIs. Every time it routes an event, the lambda fetches a secret for authentication.

• Cost of secrets = 100 secrets x $0.40 = $40.00 per month
• Cost of API calls = 600,000 calls @ $0.05/10,000 calls = $3.00 per month

$43.00 per month may not look much if you are paying thousands of dollars every month. Think of situations where secrets are short-lived and rotated regularly. Also, it is not uncommon that millions of events get routed via EventBridge to several targets every day.

Luckily, if we use EventBridge’s API Destinations, there is a way to reduce the cost which I explain in the next section!

Service-Linked Role — The Secret Behind Cost Saving

What is a service-linked role?

A service-linked role is a special IAM role that is linked directly to an AWS service. This role permits an AWS service to invoke resources in another AWS service on our behalf.

In the case of API Destinations, Amazon EventBridge uses the service-linked role to call AWS Secrets Manager to fetch connection credentials from Secrets Manager while invoking an event target.

Because EventBridge performs the Secrets Manager interaction internally, it absorbs the cost. In short, as a user of EventBridge, we won’t pay for the Secrets Manager API calls that EventBridge will make on our behalf!

How does EventBridge accomplish this?

First, when we create a connection for an API Destination, a service-linked role AWSServiceRoleForAmazonEventBridgeApiDestinations is added to the account.

Then, for managing secrets in the Secrets Manager, a special policy AmazonEventBridgeApiDestinationsServiceRolePolicy is attached to the above role. This policy has the Allow effect for all Secrets Manager actions, as shown below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:CreateSecret",
                "secretsmanager:UpdateSecret",
                "secretsmanager:DescribeSecret",
                "secretsmanager:DeleteSecret",
                "secretsmanager:GetSecretValue",
                "secretsmanager:PutSecretValue"
            ],
            "Resource": "arn:aws:secretsmanager:*:*:secret:events!connection/*"
        }
    ]
}

If we look at the Resource value in the above policy, it has a forbidden character !and indicates that it’s created internally by EventBridge. Later, we will see the same pattern being used when it creates its secret.

Creating API Connection With Secrets

It’s easy to summarize the configuration of an API Destination’s Connection with secrets in three steps.

1. User step: Creation of secrets in the Secrets Manager
2. User step: Creation of API Connection with secrets
3. Internal step: Housekeeping by CloudFormation!

1. Creation of secrets

It’s the normal process of creating a secret in Secrets Manager to store the API client credentials.

The way teams create secrets depends on the process they follow and the environment — manually updated in the console by an admin, supplied as input to a script, reading from a file, or other approaches that are in place.

The point here is about separating the creation of the secrets away from the API Destinations setup. We will refer to the secrets when we create the connection instead of the actual values.

For context, here is a sample CloudFormation script to create a secret for OAuth connection credentials.

Resources:
  ConnectionSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub "${SecretName}"
      Description: "OAuth connection credential secret."   
      SecretString: !Sub >-
        {"ClientID": "${id}", "ClientSecret": "${secret}", "APIKey": "${key}"}

2. Referencing the secrets

This is the core of what we’ve been discussing so far. When we setup a Connection for an API Destination, we use the reference to the secret in the Secrets Manager rather than the actual value.

The following script will create the API connection we saw in Part 1.

Resources:
  OAuthConnection:
    Type: AWS::Events::Connection
    Properties:
      Name: 'CustomerAccounts'
      Description: 'Customer accounts API Destination connection'
      AuthorizationType: OAUTH_CLIENT_CREDENTIALS
      AuthParameters:
        OAuthParameters:
          ClientParameters:
            ClientID: '{{resolve:secretsmanager:CustomerAccountsClientSecret:SecretString:ClientID}}'
            ClientSecret: '{{resolve:secretsmanager:CustomerAccountsClientSecret:SecretString:ClientSecret}}'
          OAuthHttpParameters:
            BodyParameters:
              - Key: 'scope'
                Value: 'customers.create'
                IsValueSecret: false
              - Key: 'grant_type'
                Value: 'client_credentials'
                IsValueSecret: false
            HeaderParameters:
              - Key: 'x-api-key'
                Value: '{{resolve:secretsmanager:CustomerAccountsClientSecret:SecretString:APIKey}}'
                IsValueSecret: true
          AuthorizationEndpoint: 'https://youAuth.com/oauth2/token'
          HttpMethod: POST

The highlighted lines in the script are the places where we provide the reference to the secret values that we already have in Secret Manager.

If the secrets are from a different account, then you need to supply the ARN of the secret along with the version stage, as below.

resolve:secretsmanager:arn:aws:secretsmanager:us-west-1:123456789012:secret:CustomerAccountsClientSecret-eOsoa9:SecretString:APIKey

What EventBridge does with a given reference is explained in the following section.

3. Housekeeping by CloudFormation!

Here is what happens when CloudFormation creates an API connection with the secret references.

• From the reference we supply, CloudFormation will retrieve the actual secret and supply it to EventBridge.
• EventBridge then creates a new (copy) secret (based on the original) in the same account, but with a different naming format (events!connection/…), as shown below.

List of secrets

• In its service-linked secret, EventBridge keeps additional details such as the target endpoint, invocation parameters, and access token. For example, below is the SecretString value of the service-linked secret.

{
  "client_id": "shdg3bsdu8lhxvs",
  "client_secret": "ab@ls-r£x8M9",
  "authorization_endpoint": "https://youAuth.com/oauth2/token",
  "http_method": "POST",
  "oauth_http_parameters": {
    "header_parameters": [
      {
        "key": "x-api-key",
        "value": "hfvbdifusdf",
        "is_value_secret": true
      }
    ],
    "body_parameters": [
      {
        "key": "scope",
        "value": "customers.create",
        "is_value_secret": false
      },
      {
        "key": "grant_type",
        "value": "client_credentials",
        "is_value_secret": false
      }
    ]
  },
  "access_token": "eyJraWQi...3ZLW"
}

• While invoking an API Destination, EventBridge fetches from the copy secret that it created and not from the original.
• EventBridge uses the role AWSServiceRoleForAmazonEventBridgeApiDestinations that we saw earlier and the service-linked policy AmazonEventBridgeApiDestinationsServiceRolePolicy allows it to interact with the Secrets Manager.
• Most importantly, as with the above, EventBridge absorbs the cost incurred for its interaction with the Secrets Manager.

Things To Remember

• You are responsible for the original secret and the cost associated with it.
• If you update the original secret, the changes won’t be reflected in the copy.
• If you have a rotation policy for your secret, then it can become a challenge to propagate the changes to the linked secret.
• When you create an API Connection, it will try and authorize the connection by calling the OAuth endpoint with the details. If it fails to authorize, then the resource creation may fail.
• When you delete an API Connection, its linked secret will also be deleted.

There you have it! A secure and cost-effective way to use secrets with API Destinations. Now that we have been through the details, it’s easy to see the cost savings it brings.

Think of situations where your application has several API clients and each subscribing to notifications via webhooks. Rather than invoking your HTTP targets via lambda functions and fetching secrets directly from the Secrets Manager, if your use case can work with API Destinations, you can see the benefits it brings. Not to mention the Functionless approach with no lambda functions to write for the integration to dispatch events. It’s a win-win situation!

Conclusion

I hope the details I shared in Part 1 and here have helped you understand the functioning and the use of EventBridge’s API Destinations.

As the serverless ecosystem matures, new features and patterns enable us to optimize our solution. The use of asynchronous communications and the power of event-driven architectures redefine the way we build modern applications to deliver business value. Growing on these foundations will drive the next wave of serverless evolution.

Let our serverless journey continue to take us to endless destinations!