When AWS CDK 2 does not deliver and AWS SDK for Javascript comes to the rescue

Introduction

At FourCo we use Infrastructure as Code on a daily basis for our customer projects and since the introduction of AWS CDK in July 2019 it is possible to build infrastructures with real code! During the past 3 years AWS CDK evolved to a full fledged framework so you can code your infrastructure in a modularized way. But sometimes bugs/omissions creep in and you need to find a workaround to solve a problem. This blogs covers extending AWS CDK with Javascript AWS API SDK to solve these problems in an elegant way.

Lifecycle of AWS CDK constructs

As new AWS Services come available, AWS first delivers the API backend before CloudFormation definitions are prepared. The CDK constructs follow after this. First with level 1 constructs (CFN) and later with higher level. Sometimes these higher level constructs need to mature a bit to fit “in the big picture”.

The Architecture

A couple of weeks ago we demoed our Data Hub solution where Amazon Managed Streaming for Kafka (AWS MSK) was used to ingest events from IoT devices. As a storage solution AWS OpenSearch was used. To process events from AWS MSK we leveraged the topic event trigger to kick off a lambda which processes the events before storing them in OpenSearch. By using AWS Services the individual components are modern and low maintenance (NoOps!).

The flow of data is visualized in below architecture.

The CDK code for creating the AWS MSK Cluster and a user

Above deployment of AWS MSK is defined with just a few lines of code and adding users is easy!

import * as msk from 'aws-cdk-lib/aws-msk-alpha';

declare const vpc: ec2.Vpc;

const cluster = new msk.Cluster(this, 'Cluster', {
  clusterName: 'myCluster',
  kafkaVersion: msk.KafkaVersion.V2_8_1,
  vpc,
});

cluster.addUser("opensearch-sink");

Defining the Lambda eventListener

Adding the Lambda to process the events on the “iot-topic” is shown below.

import { Secret } from 'aws-cdk-lib/aws-secretsmanager';
import { ManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

// FIXME: Below construct does not return the full ARN including the 6 random characters
// which is needed by the creation of the EventSource. 
const secret = new Secret(this, 'Secret', { secretName: 'AmazonMSK_myCluster_opensearch-sink' });

declare const myFunction: lambda.Function;

myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn: cluster.clusterArn,
  topic: "iot-topic,
  secret: secret, // FIXME: This fails due to the lack of the fully qualified ARN.
  batchSize: 10,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
}));

Unfortunately above construct does not work and the deployment of the stack fails and results in a rollback.

Lookup of the full ARN of the Secret with AWS SDK for Javascript

Even looking up the secret by partial ARN results in the 6 symbols being replaced with the “??????” wildcard, which still results in a deployment failure and a rollback.

It is however possible the use AWS SDK for Javascript to perform a dynamic lookup of the Secret!

First we install the NPM module:

bas@host:~/datahub-demo$ npm install @aws-sdk/client-secrets-manager

Import the module at the top of our CDK construct:

import { SecretsManagerClient, ListSecretsCommand } from "@aws-sdk/client-secrets-manager";

And prepare our API call to AWS SecretsManager with the SDK. Mind the mixing in of the “this.region” CDK construct to direct the API calls to the correct region.

const client = new SecretsManagerClient({ region: this.region });
const command = new ListSecretsCommand({
  "Filters": [
    {
       "Key": "name",
       "Values": [ "AmazonMSK_KafkaCluster_opensearch-sink" ]
    }
 ],
});

After that we can easily process the found Secret and create the CDK construct with the fully qualified ARN. The Javascript SDK call is a-synchronous, so we process the results in CDK resources in the callback function. We place the code in a separate Stack, so this is called after the cluster is deployed and the user is added.

client.send(command).then(
  (data: any) => {
    let kafkaSecret:secretsmanager.ISecret;
    for (var secret of <Array>data.SecretList) {
      console.log(`Using secret '${secret.ARN}' for Kafka event source`);
      kafkaSecret = secretsmanager.Secret.fromSecretAttributes(this, 'KafkaClusterSecret', {
        secretCompleteArn: secret.ARN
      });
      kafkaSecret.grantRead(myFunction)
    }
    myFunction.addEventSource(new events.ManagedKafkaEventSource({
      batchSize: 10,
      clusterArn: cluster.clusterArn,
      maxBatchingWindow: cdk.Duration.seconds(10),
      secret: kafkaSecret,
      startingPosition: lambda.StartingPosition.TRIM_HORIZON,
      topic: "iot-topic",
    }));
  },
  (error) => {
    // error handling.
    console.log(`Error: ${error}`)
  }
);

Conclusion, alternatives and further ideas

Mixing in AWS SDK functionality in your CDK code gives additional options to add in dynamic data to your solutions. Of course there are alternatives like CDK Custom Constructs, but these are semi-static.

After sharing this solution internally during a Knowledge Sharing sesstion, multiple colleagues got new inspiration to use this in their own customer projects. Like integrating it in global deployments of AWS Transit Gateway or looking up endpoints or route entries.