This diagram shows the important components that will be deployed by the quickstart. Arrows on the diagrams explain the important flows between them. The pods shown in yellow run the kafka console producer and kafka console consumers command line applications. These pods are used to demonstrate the record encryption in action.

The diagram omits the operators.

You’ll need the following software installed:

Let’s get the Kubernetes Cluster up and running. Minikube defaults work just fine for this quickstart.

Now we need a Kafka Cluster. This is the Kafka Cluster that will be proxied. We’ll just use one from the Strimzi Quickstart. It will create a cluster in the kafka namespace.

We’ll need the boostrap server address of the kafka cluster later, so let’s assign a variable containing it now.

Next, we’ll deploy a Kroxylicious proxy instance using the record encryption example included in the install zip you downloaded when you installed the operator

It will create an instance of Kroxylicious that will proxy the kafka cluster you created above. It will configure it to use the Record Encryption filter to encrypt as records are sent by producers and decrypt them as records get fetched by consumers. The proxy will be created the my-proxy namespace

You need to update the example to find keys in your chosen KMS.

Let’s wait for the Proxy to be ready:

and wait for the virtual cluster to be ready for connections.

Finally, let’s assign a variable containing the virtual cluster’s bootstrap address. We’ll use this later to produce and consumer records through the proxy.

Record Encryption needs an encryption key to use when encrypting the records produced to the topic. Let’s create an encryption key in our KMS now.

The filter is configured to expect a key to exist in the KMS with the name KEK_<topic name>. We are going to be using a topic called trades so we will create a key that can be referred to using the name KEK_trades.

Now let’s use Kafka’s console producer CLI to send a few records to a topic called trades. The Record Encryption filter will encrypt the records before they reach broker, but this is completely transparent to the console producer.

Now let’s use Kafka’s console consumer to fetch the records. You’ll see the two records we sent above. The Record Encryption filter will decrypt the records before they reach consumer, but this is completely transparent to the console consumer.

Consuming the same records we wrote is a bit underwhelming! And, in fact, it’s what we’d expect to see with a vanilla, unproxied, Kafka cluster. So how do we know the records really encrypted on the broker? Let’s use the console consumer again, but this time we’ll consume straight from the Kafka cluster, bypassing the proxy. We’ll get back the records, but they’ll contain ciphertext, rather than the plaintext.

To clean up, remove the proxy and the kafka cluster

Remove the Kroxylicious and Strimzi Operator.

Remove the KMS.

And finally remove the namespaces.

Learn more from the Kroxylicious Record Encryption guide and from the Kroxylicious Operator for Kubernetes guide

You are also welcome to come talk to us. Chat with us in Slack or start a Github Discussion.