About this guide

This guide covers using the Kroxylicious Record Encryption Filter to provide encryption-at-rest for Apache Kafka. Refer to other Kroxylicious guides for information on running the proxy or for advanced topics such as plugin development.

The Kroxylicious Record Encryption filter enhances the security of Kafka messages. The filter uses industry-standard cryptographic techniques to apply encryption to Kafka messages, ensuring the confidentiality of data stored in the Kafka Cluster. By centralizing topic-level encryption, Kroxylicious provides streamlined protection across Kafka clusters.

To use the filter, follow these steps:

  1. Set up a Key Management System (KMS)

  2. Establish encryption keys within the KMS for securing the topics

  3. Configure the filter within Kroxylicious

The filter integrates with a Key Management Service (KMS), which is responsible for the safe storage of sensitive key material. Kroxylicious supports the following KMS providers:

  • HashiCorp Vault

  • AWS Key Management Service.

  • Fortanix DSM

You can provide implementations for your specific KMS systems. Additional KMS support may be added based on demand.

1. How encryption works

The Record Encryption filter uses envelope encryption to encrypt records with symmetric encryption keys. The filter encrypts records from produce requests and decrypts records from fetch responses.

Envelope encryption

Envelope encryption is an industry-standard technique suited for encrypting large volumes of data in an efficient manner. Data is encrypted with a Data Encryption Key (DEK). The DEK is encrypted using a Key Encryption Key (KEK). The KEK is stored securely in a Key Management System (KMS).

Symmetric encryption keys

AES(GCM) 256 bit encryption symmetric encryption keys are used to encrypt and decrypt record data.

The process is as follows:

  1. The filter intercepts produce requests from producing applications and transforms them by encrypting the records.

  2. The produce request is forwarded to the broker.

  3. The filter intercepts fetch responses from the broker and transforms them by decrypting the records.

  4. The fetch response is forwarded to the consuming application.

The filter encrypts the record value only. Record keys, headers, and timestamps are not encrypted.

The entire process is transparent from the point of view of Kafka clients and Kafka brokers. Neither are aware that the records are being encrypted, nor do they have any access to the encryption keys or have any influence on the ciphering process to secure the records.

1.1. How the filter encrypts records

The filter encrypts records from produce requests as follows:

  1. Filter selects a KEK to apply.

  2. Requests the KMS to generate a DEK for the KEK.

  3. Uses an encrypted DEK (DEK encrypted with the KEK) to encrypt the record.

  4. Replaces the original record with a ciphertext record (encrypted record, encrypted DEK, and metadata).

The filter uses a DEK reuse strategy. Encrypted records are sent to the same topic using the same DEK until a time-out or an encryption limit is reached.

1.2. How the filter decrypts records

The filter decrypts records from fetch responses as follows:

  1. Filter receives a cipher record from the Kafka broker.

  2. Reverses the process that constructed the cipher record.

  3. Uses KMS to decrypt the DEK.

  4. Uses the decrypted DEK to decrypt the encrypted record.

  5. Replaces the cipher record with a decrypted record.

The filter uses an LRU (least recently used) strategy for caching decrypted DEKs. Decrypted DEKs are kept in memory to reduce interactions with the KMS.

1.3. How the filter uses the KMS

To support the filter, the KMS provides the following:

  • A secure repository for storing Key Encryption Keys (KEKs)

  • A service for generating and decrypting Data Encryption Keys (DEKs)

KEKs stay within the KMS. The KMS generates a DEK (which is securely generated random data) for a given KEK, then returns the DEK and an encrypted DEK. The encrypted DEK has the same data but encrypted with the KEK. The KMS doesn’t store encrypted DEKs; they are stored as part of the cipher record in the broker.

The KMS must be available during runtime. If the KMS is unavailable, the filter will not be able to obtain new encrypted DEKs on the produce path or decrypt encrypted DEKs on the consume path. The filter will continue to use previously obtained DEKs, but eventually, production and consumption will become impossible. It is recommended to use the KMS in a high availability (HA) configuration.

1.4. Practicing key rotation

Key rotation involves periodically replacing cryptographic keys with new ones and is considered a best practice in cryptography.

The filter allows the rotation of Key Encryption Keys (KEKs) within the Key Management System (KMS). When a KEK is rotated, the new key material is eventually used for newly produced records. Existing records, encrypted with older KEK versions, remain decryptable as long as the previous KEK versions are still available in the KMS.

If your encrypted topic is receiving regular traffic, the Data Encryption Key (DEK) will be refreshed as new records flow through. However, if messages are infrequent, the DEK might be used for up to 2 hours (by default) after its creation.

When the KEK is rotated in the external KMS, it will take up to 1 hour (by default) before all records produced by the filter contain a DEK encrypted with the new key material. This is because existing encrypted DEKs are used for a configurable amount of time after creation, the Filter caches the encrypted DEK, one hour after creation they are eligible to be refreshed.

If you need to rotate key material immediately, execute a rolling restart of your cluster of Kroxylicious instances.

If an old KEK version is removed from the KMS, records encrypted with that key will become unreadable, causing fetch operations to fail. In such cases, the consumer offset must be advanced beyond those records.

1.5. What part of a record is encrypted?

The record encryption filter encrypts only the values of records, leaving record keys, headers, and timestamps untouched. Null record values, which might represent deletions in compacted topics, are transmitted to the broker unencrypted. This approach ensures that compacted topics function correctly.

1.6. Unencrypted topics

You may configure the system so that some topics are encrypted and others are not. This supports scenarios where topics with confidential information are encrypted and Kafka topics with non-sensitive information can be left unencrypted.

Additional resources

2. Preparing your KMS

This section assumes that you already have a supported KMS instance up and running. It describes how to prepare the KMS for use with the filter.

2.1. Preparing HashiCorp Vault

To use HashiCorp Vault with the Record Encryption filter, use the following setup:

  • Enable the Transit Engine as the Record Encryption filter relies on its APIs.

  • Create a Vault policy specifically for the filter with permissions for generating and decrypting Data Encryption Keys (DEKs) for envelope encryption.

  • Obtain a Vault token that includes the filter policy.

2.1.1. Enable the Transit Engine

The filter integrates with the HashiCorp Vault Transit Engine. Vault does not enable the Transit Engine by default. It must be enabled before it can be used by the filter.

Vault Transit Engine URL

The Vault Transit Engine URL is required so the filter knows the location of the Transit Engine within the Vault instance.

The URL is formed from the concatenation of the Api Address (reported by Vault reported by during starts up) with the complete path to Transit Engine, including the name of the engine itself. If Namespacing is used on the Vault instance, the path needs to include the namespace(s). The URL will end with /transit unless the -path parameter was used when enabling the engine.

If namespacing is not in use, the URL will look like this:

https://myvaultinstance:8200/v1/transit

If namespacing is in use, the path must include the namespaces. For example, if there is a parent namespace is a and a child namespace is b, the URL will look like this:

https://myvaultinstance:8200/v1/a/b/transit

If the name of the Transit engine was changed (using the -path argument to the vault secrets enable transit command) the URL will look like this:

https://myvaultinstance:8200/v1/mytransit
Establish the naming convention for keys within Vault hierarchy

Establish a naming convention for keys to keep the filter’s keys separate from those used by other systems. Here, we use a prefix of KEK_ for filter key name. Adjust the instructions if a different naming convention is used.

Role of the administrator

To use the filter, an administrator or an administrative process must create the encryption keys within Vault, which are used by the envelope encryption process.

The organization deploying the Record Encryption filter is responsible for managing this administrator or process.

The administrator must have permissions to create keys beneath transit/keys/KEK_* in the Vault hierarchy.

As a guideline, the minimal Vault policy required by the administrator is as follows:

path "transit/keys/KEK_*" {
  capabilities = ["read", "write"]
}
Establish an application identity for the filter

The filter must authenticate to Vault in order to perform envelope encryption operations, such as generating and decrypting DEKs Therefore, a Vault identity with sufficient permissions must be created for the filter.

Create a Vault policy for the filter:

vault policy write kroxylicious_encryption_filter_policy - << EOF
path "transit/keys/KEK_*" {
  capabilities = ["read"]
}
path "/transit/datakey/plaintext/KEK_*" {
  capabilities = ["update"]
}
path "transit/decrypt/KEK_*" {
  capabilities = ["update"]
}
EOF

Create a Periodic (long-lived) Vault Token for the filter:

vault token create -display-name "kroxylicious record encryption" \
                   -policy=kroxylicious_encryption_filter_policy \
                   -period=768h \ (1)
                   -no-default-policy \ (2)
                   -orphan (3)
1 Causes the token to be periodic (with every renewal using the given period).
2 Detach the "default" policy from the policy set for this token. This is done so the token has least-privilege.
3 Create the token with no parent. This is done so that expiration of a parent won’t expire the token used by the filter.
The example token create command illustrates the use of -no-default-policy and -orphan. The use of these flags is not functionally important. You may adapt the configuration of the token to suit the standards required by your organization.

The token create command yields the token. The token value is required later when configuring the vault within the filter.

token              hvs.CAESIFJ_HHo0VnnW6DSbioJ80NqmuYm2WlON-QxAPmiJScZUGh4KHGh2cy5KdkdFZUJMZmhDY0JCSVhnY2JrbUNEWnE
token_accessor     4uQZJbEnxW4YtbDBaW6yVzwP
token_policies     [kroxylicious_encryption_filter_policy]

The token must be renewed before expiration. It is the responsibility of the administrator to do this.

This can be done with a command like the following:

vault token renew --accessor <token_accessor>
Testing the application identity for the filter using the CLI

To test whether the application identity and the policy are working correctly, a script can be used.

First, as the administrator, create a KEK in the hierarchy at this path transit/keys/KEK_testkey.

VAULT_TOKEN=<kroxylicious encryption filter token> validate_vault_token.sh <kek path>

The script should respond Ok. If errors are reported check the policy/token configuration.

transit/keys/KEK_testkey can now be removed.

2.1.2. Creating HashiCorp Vault keys

As the administrator, use either the HashiCorp UI or CLI to create AES-256 symmetric keys following your key naming convention. The key type must be aes256-gcm96, which is Vault’s default key type.

It is recommended to use a key rotation policy.

If using the Vault CLI, the command will look like:

vault write -f transit/keys/KEK_trades type=aes256-gcm96 auto_rotate_period=90d

2.2. Preparing AWS KMS

To prepare AWS Key Management Service for use with the Record Encryption filter, use the following setup:

  • Establish an AWS KMS aliasing convention for keys

  • Create AWS KMS keys

You’ll need a privileged AWS user that is capable of creating users and policies to perform the set-up.

2.2.1. Establish an aliasing convention for keys within AWS KMS

The filter references KEKs within AWS via an AWS key alias.

Establish a naming convention for key aliases to keep the filter’s keys separate from those used by other systems. Here, we use a prefix of KEK_ for filter aliases. Adjust the instructions if a different naming convention is used.

Role of the administrator

To use the filter, an administrator or an administrative process must create the encryption keys within AWS KMS, which are used by the envelope encryption process.

The organization deploying the Record Encryption filter is responsible for managing this administrator or process.

The administrator must have permissions to create keys in AWS KMS. As a starting point, the built-in AWS policy AWSKeyManagementServicePowerUser confers sufficient key management privileges.

To get started, use the following commands to set up an administrator with permissions suitable for managing encryption keys in KMS through an AWS Cloud Shell. This example illustrates using the user name kroxylicious-admin, but you can choose a different name if preferred. Adjust the instructions accordingly if you use a different user name.

ADMIN=kroxylicious-admin
INITIAL_PASSWORD=$(aws secretsmanager get-random-password  --output text)
CONSOLE_URL=https://$(aws sts get-caller-identity --query Account --output text).signin.aws.amazon.com/console
aws iam create-user --user-name ${ADMIN}
aws iam attach-user-policy --user-name ${ADMIN} --policy-arn arn:aws:iam::aws:policy/AWSKeyManagementServicePowerUser
aws iam attach-user-policy --user-name ${ADMIN} --policy-arn arn:aws:iam::aws:policy/IAMUserChangePassword
aws iam attach-user-policy --user-name ${ADMIN} --policy-arn arn:aws:iam::aws:policy/AWSCloudShellFullAccess
aws iam create-login-profile --user-name ${ADMIN} --password "${INITIAL_PASSWORD}" --password-reset-required
echo Now log in at ${CONSOLE_URL}  with user name ${ADMIN} password "${INITIAL_PASSWORD}" and change the password.
Create an alias-based policy for KEK aliases

Create an alias-based policy granting permissions to use keys aliased by the established alias naming convention.

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
cat > /tmp/policy << EOF
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AliasBasedIAMPolicy",
			"Effect": "Allow",
			"Action": [
				"kms:Encrypt",
				"kms:Decrypt",
				"kms:GenerateDataKey*",
				"kms:DescribeKey"
			],
			"Resource": [
                "arn:aws:kms:*:${AWS_ACCOUNT_ID}:key/*"
			],
			"Condition": {
				"ForAnyValue:StringLike": {
					"kms:ResourceAliases": "alias/KEK_*"
				}
			}
		}
	]
}
EOF
aws iam create-policy --policy-name KroxyliciousRecordEncryption --policy-document file:///tmp/policy
Establish an authentication mechanism for the filter

The filter must authenticate to AWS in order to perform envelope encryption operations, such as generating and decrypting DEKs.

Authenticating using long-term IAM identity

This procedure describes how to create a long-term IAM identity for the Record Encryption filter to authenticate to AWS KMS. The process involves creating an IAM user and access key, and attaching an alias-based policy that grants permissions to perform KMS operations on specific KEKs.

Do not enable console access for this user. The filter requires only API access, and console access would unnecessarily increase the security risk.
Prerequisites
Procedure

  1. Create the IAM user and access key:

    aws iam create-user --user-name kroxylicious
aws iam create-access-key --user-name kroxylicious

    This example uses kroxylicious as the user name, but you can substitute a different name if necessary.

  2. Attach the alias-based policy to the IAM identity:

    AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
aws iam attach-user-policy --user-name kroxylicious --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KroxyliciousRecordEncryption"

    This step grants the user permission to perform KMS operations on KEKs that use the alias naming convention defined in the KroxyliciousRecordEncryption policy.

  3. Verify that the policy has been successfully attached:

    aws iam list-attached-user-policies --user-name kroxylicious
Authenticating using AWS EC2 metadata

This procedure describes how to use AWS EC2 metadata for the Record Encryption filter to authenticate to AWS KMS. The process involves creating a trust policy, creating an IAM role, and attaching an alias-based policy that grants permissions to perform KMS operations on specific KEKs.

The filter authenticates using the temporary credentials retrieved from EC2 instance metadata.

Prerequisites
Procedure

  1. Create a trust policy:

    cat > trustpolicy << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Principal": {
                "Service": [
                    "ec2.amazonaws.com"
                ]
            }
        }
    ]
}
EOF

    The trust policy specifies that the EC2 instance can assume the role, enabling it to retrieve and use temporary credentials for authentication.

  2. Create the IAM role using the trust policy:

    aws iam create-role --role-name KroxyliciousInstance --assume-role-policy-document file://trustpolicy

    This example uses KroxyliciousInstance as the role name, but you can substitute a different name if necessary.

  3. Attach the alias-based policy to the role:

    AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
aws iam attach-role-policy --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KroxyliciousRecordEncryption --role-name KroxyliciousInstance

    This step grants the role permission to perform KMS operations on KEKs that use the alias naming convention defined in the KroxyliciousRecordEncryption policy.

  4. Verify that the policy has been successfully attached:

    aws iam list-attached-role-policies --role-name KroxyliciousInstance

  5. Associate the role with the EC2 instance:

    aws ec2 associate-iam-instance-profile --instance-id <EC2_instance_id> --iam-instance-profile Name="KroxyliciousInstance"

    Replace <EC2_instance_id> with the instance ID of each AWS EC2 instance hosting a Kroxylicious instance.

  6. Verify that the role has been associated with the EC2 instance:

    aws ec2 describe-iam-instance-profile-associations --filters Name=instance-id,Values=<EC2_instance_id>

2.2.2. Creating AWS KMS keys

As the administrator, use either the AWS Console or CLI to create a Symmetric key with Encrypt and decrypt usage. Multi-region keys are supported.

It is not possible to make use of keys from other AWS accounts. For more information on this limitation, see the issue for AWS KMS serde improvements.

Give the key an alias as described in Establish an aliasing convention for keys within AWS KMS.

If using the CLI, this can be done with commands like this:

KEY_ALIAS="KEK_<name>"
KEY_ID=$(aws kms create-key | jq -r '.KeyMetadata.KeyId')
# the create key command will produce JSON output including the KeyId
aws kms create-alias --alias-name alias/${KEY_ALIAS} --target-key-id ${KEY_ID}

Once the key is created, it is recommended to use a key rotation policy.

aws kms enable-key-rotation --key-id ${KEY_ID} --rotation-period-in-days 180

2.3. Preparing Fortanix Data Security Manager (DSM)

To prepare Fortanix Data Security Manager (DSM) for use with the Record Encryption filter, use the following setup:

  • Establish a naming convention for keys and choose a Fortanix group where the keys will reside.

  • Create an application identity, with an API key, for the Record Encryption filter.

  • Create Fortanix DSM keys.

2.3.1. Integrate with Fortanix DSM

The filter integrates with the Fortanix Data Security Manager (DSM). Both Fortanix DSM software-as-a service (SaaS) or an on-premise installation are supported.

These instructions assume that you are using the Fortanix DSM CLI, but you can use the Fortanix DSM user interface if preferred.

The Fortanix KMS plugin for Record Encryption doesn’t yet support keys in the Deactivated state. For more information, see the related issue.
Fortanix DSM Cluster URL

The Record Encryption filter requires the URL of the Fortanix DSM cluster.

If you are using SaaS, the URL looks like https://<region>.smartkey.io where region is an identifier such as amer. For more information, see the link: Fortanix documentation.

If using an on-premises instance, talk to the group responsible for it within your organization to find out the URL you should use.

Establish a naming convention for keys within Fortanix DSM

Establish a naming convention for keys to keep the filter’s keys separate from those used by other systems. Here, we use a prefix of KEK_ for filter key name.

Choose the Fortanix DSM groups to keep the keys. Here, we assume a group name of topic-keks.

Adjust the instructions if a different naming convention is used.

Role of the administrator

To use the filter, an administrator or an administrative process must create the encryption keys within Fortanix DSM, which are used by the envelope encryption process.

The organization deploying the Record Encryption filter is responsible for managing this administrator or process.

The administrator must have permissions to create keys with Fortanix DSM.

Establish an application identity for the filter

The filter must authenticate to Fortanix DSM in order to perform the encryption and decryption operations.

Create a Fortanix DSM App with sufficient permissions for the filter:

sdkms-cli --api-endpoint https://<region>.smartkey.io create-app --name kroxylicious --default-group topic-keks --groups topic-keks

Retrieve the API key for the app:

sdkms-cli --api-endpoint https://<region>.smartkey.io get-app-api-key --name kroxylicious

The Record Encryption filter uses the API Key in its KMS configuration to authenticate to Fortanix DSM.

2.3.2. Creating Fortanix DSM keys

As the administrator, create AES-256 symmetric keys following your key naming convention and belonging to the required group. When creating keys specify the key operations as ENCRYPT,DECRYPT,APPMANAGEABLE. These are the minimal permissions required for record encryption to function.

Identify the ID of the group to contain the keys:

GROUP_ID=$(sdkms-cli  --api-endpoint https://<region>.smartkey.io list-groups | grep topic-keks | awk '{print $1}')

For example, here we extract the ID of the group named topic-keks.

Create a key and associate it with the group:

KEY_NAME="KEK_<name>"
sdkms-cli  --api-endpoint https://<region>.smartkey.io create-key --obj-type AES --key-size 256 --group-id ${GROUP_ID} --name ${KEY_NAME} --key-ops ENCRYPT,DECRYPT,APPMANAGEABLE
It is recommended to use a key rotation policy.

3. Configuring the Record Encryption filter

This section describes at a high level how to configure the Record Encryption filter using a previously prepared KMS. Subsections provide in-depth details.

Prerequisites
Procedure

  1. Configure the plugin for your supported KMS, as required.

  2. Create a filter configuration that references the configured KMS plugins.

    See Filter configuration

  3. Apply the filter configuration:

3.1. HashiCorp Vault plugin configuration

For HashiCorp Vault, the KMS configuration used by the filter looks like this. Use the Vault Token and Vault Transit Engine URL values from the KMS setup.

kms: VaultKmsService                                          (1)
kmsConfig:
  vaultTransitEngineUrl: <vault transit engine service url>   (2)
  tls:                                                        (3)
    # ...
  vaultToken:                                                 (4)
    passwordFile: /opt/vault/token
1 Specifies the name of the KMS provider. Use VaultKmsService.
2 Vault Transit Engine URL including the protocol part, such as https: or http:.
3 (Optional) TLS trust configuration.
4 File containing the Vault Token.

A TLS client certificate can be specified using a PKCS#12 or JKS key store file.

Example TLS client certificate configuration using a PKCS#12 key store file
key:
  storeFile: /opt/cert/server.p12 (1)
  storeType: PKCS12 (2)
  storePassword: (3)
    passwordFile: /opt/cert/store.password
  keyPassword: (4)
    passwordFile: /opt/cert/key.password
1 storeFile specifies PKCS#12 file
2 storeType speficies what the keystore file type is. Supported values include PKCS12 and JKS.
3 Optionally, a keystore file password may be specified.
4 Optionally, a password may be specified for the key entry within the file.

A set of trust anchors for the TLS client can be specified using a PKCS#12 or JKS key store file.

Example TLS client trust change configuration using a PKCS#12 key store file
trust:
  storeFile: /opt/cert/server.p12 (1)
  storeType: PKCS12 (2)
  storePassword: (3)
    passwordFile: /opt/cert/store.password
1 storeFile specifies PKCS#12 file
2 storeType specifies what the keystore file type is. Supported values include PKCS12 and JKS.
3 Optionally, a keystore file password may be specified.

3.2. AWS KMS plugin configuration

For AWS KMS the configuration for authenticating with AWS KMS services looks like this:

Configuration for authenticating with a long-term IAM identity
kms: AwsKmsService                                            (1)
kmsConfig:
  endpointUrl: https://kms.<region>.amazonaws.com             (2)
  tls:                                                        (3)
    # ...
  longTermCredentials:
    accessKeyId:
      passwordFile: /opt/aws/accessKey                        (4)
    secretAccessKey:
      passwordFile: /opt/aws/secretKey                        (5)
  region: <region>                                            (6)
1 Specifies the name of the KMS provider. Use AwsKmsService.
2 AWS KMS endpoint URL, which must include the https:// scheme.
3 (Optional) TLS trust configuration.
4 File containing the AWS access key ID.
5 File containing the AWS secret access key.
6 The AWS region identifier, such as us-east-1, specifying where your KMS resources are located. This must match the region of the KMS endpoint you’re using.

Alternatively, the configuration for authenticating with EC2 metadata looks like this:

Configuration for authenticating with EC2 metadata
kms: AwsKmsService                                            (1)
kmsConfig:
  endpointUrl: https://kms.<region>.amazonaws.com             (2)
  ec2MetadataCredentials:
    iamRole: <name_of_IAM_role>                               (3)
    metadataEndpoint: <EC2_metadata_endpoint>                 (4)
    credentialLifetimeFactor: 0.8                             (5)
  region: <region>                                            (6)
1 Specifies the name of the KMS provider. Use AwsKmsService.
2 AWS KMS endpoint URL, which must include the https:// scheme.
3 Name of the IAM role associated with the EC2 instance(s) hosting Kroxylicious.
4 (Optional) Metadata endpoint used to obtain EC2 metadata. Defaults to http://169.254.169.254/. If using IPv6, use http://[fd00:ec2::254] instead.
5 (Optional) Factor used to determine when to refresh a credential before it expires. Defaults to 0.8, which means the credential is refreshed once it reaches 80% of its lifetime.
6 The AWS region identifier, such as us-east-1, specifying where your KMS resources are located. This must match the region of the KMS endpoint you’re using.

A TLS client certificate can be specified using a PKCS#12 or JKS key store file.

Example TLS client certificate configuration using a PKCS#12 key store file
key:
  storeFile: /opt/cert/server.p12 (1)
  storeType: PKCS12 (2)
  storePassword: (3)
    passwordFile: /opt/cert/store.password
  keyPassword: (4)
    passwordFile: /opt/cert/key.password
1 storeFile specifies PKCS#12 file
2 storeType speficies what the keystore file type is. Supported values include PKCS12 and JKS.
3 Optionally, a keystore file password may be specified.
4 Optionally, a password may be specified for the key entry within the file.

A set of trust anchors for the TLS client can be specified using a PKCS#12 or JKS key store file.

Example TLS client trust change configuration using a PKCS#12 key store file
trust:
  storeFile: /opt/cert/server.p12 (1)
  storeType: PKCS12 (2)
  storePassword: (3)
    passwordFile: /opt/cert/store.password
1 storeFile specifies PKCS#12 file
2 storeType specifies what the keystore file type is. Supported values include PKCS12 and JKS.
3 Optionally, a keystore file password may be specified.

3.3. Fortanix DSM plugin configuration

For Fortanix DSM, the KMS configuration looks like this. Use the API key and Fortanix DSM Cluster URL values from the KMS setup.

kms: FortanixDsmKmsService                                    (1)
kmsConfig:
  endpointUrl: <Fortanix DSM Cluster URL>                     (2)
  apiKeySessionProvider:
    apiKey:
      passwordFile: /opt/fortanix-dsm/api-key                 (3)
1 Specifies the name of the KMS provider. Use FortanixDsmKmsService.
2 Fortanix DSM Cluster URL including the protocol part, such as https: or http:.
3 File containing the API key.

3.4. Filter configuration

This procedure describes how to configure the Record Encryption filter. Provide the filter configuration and the Key Encryption Key (KEK) selector to use. The KEK selector maps topic name to key names. The filter looks up the resulting key name in the KMS.

Prerequisites
Procedure

  1. Configure a RecordEncryption type filter.

    Example Record Encryption filter configuration
    kms: <kms_service_name> (1)
kmsConfig:
  <kms_specific_config> (2)
  # ...
selector: <KEK_selector_service_name> (3)
selectorConfig:
  template: "KEK_$(topicName)" (4)
unresolvedKeyPolicy: PASSTHROUGH_UNENCRYPTED (5)
experimental:
  encryptionDekRefreshAfterWriteSeconds: 3600 (6)
  encryptionDekExpireAfterWriteSeconds: 7200 (7)
  maxEncryptionsPerDek: 5000000 (8)
    1 The KMS service name.
    2 Configuration specific to the KMS provider.
    3 The Key Encryption Key (KEK) selector to use. The $(topicName) is a literal understood by the proxy. For example, if using the TemplateKekSelector with the template KEK_$(topicName), create a key for every topic that is to be encrypted with the key name matching the topic name, prefixed by the string KEK_.
    4 The template for deriving the KEK, based on a specific topic name.
    5 Optional policy governing the behaviour when the KMS does not contain a key. The default is PASSTHROUGH_UNENCRYPTED which causes the record to be forwarded, unencrypted, to the target cluster. Users can alternatively specify REJECT which will cause the entire produce request to be rejected. This is a safer alternative if you know that all traffic sent to the Virtual Cluster should be encrypted because unencrypted data will never be forwarded.
    6 How long after creation of a DEK before it becomes eligible for rotation. On the next encryption request, the cache will asynchronously create a new DEK. Encryption requests will continue to use the old DEK until the new DEK is ready.
    7 How long after creation of a DEK until it is removed from the cache. This setting puts an upper bound on how long a DEK can remain cached.
    8 The maximum number of records any DEK should be used to encrypt. After this limit is hit, that DEK will be destroyed and a new one created.

    encryptionDekRefreshAfterWriteSeconds and encryptionDekExpireAfterWriteSeconds properties govern the originator usage period of the DEK, which is the amount of time the DEK remains valid for encrypting records. Shortening this period helps limit the impact if the DEK key material is leaked. However, shorter periods increase the number of KMS API calls, which might affect produce and consume latency and raise KMS provider costs.

    maxEncryptionsPerDek helps prevent key exhaustion by placing an upper limit of the amount of times that a DEK may be used to encrypt records.

  2. Verify that the encryption has been applied to the specified topics by producing messages through the proxy and then consuming directly and indirectly from the Kafka cluster.

If the filter is unable to find the key in the KMS, the filter passes through the records belonging to that topic in the produce request without encrypting them.

3.5. Example proxy configuration file

If your instance of the Kroxylicious proxy runs directly on an operating system, provide the filter configuration in the filterDefinitions list of your proxy configuration. Here’s a complete example of a filterDefinitions entry configured for record encryption with Vault KMS:

Example filterDefinitions configuration
filterDefinitions:
  - name: my-encryption-filter
    type: RecordEncryption
    config:
      kms: VaultKmsService
      kmsConfig:
        vaultTransitEngineUrl: # ...
        tls: # ...
        vaultToken:
          passwordFile: /opt/vault/token
      selector: TemplateKekSelector
      selectorConfig:
        template: "KEK_$(topicName)"
      unresolvedKeyPolicy: PASSTHROUGH_UNENCRYPTED
      experimental:
        encryptionDekRefreshAfterWriteSeconds: 3600
        encryptionDekExpireAfterWriteSeconds: 7200
        maxEncryptionsPerDek: 5000000

Refer to the Kroxylicious Proxy guide for more information about configuring the proxy.

4. Monitoring the Record Encryption filter

This section describes how to monitor the Record Encryption filter.

4.1. Record Encryption filter metrics

The filter emits metrics that provide insights into its interactions with the configured KMS. They indicate the load the filter places on the KMS infrastructure and how often its interactions with the KMS fail.

The filter emits metrics that count the number of records that are being encrypted. This can help you verify that the filter is configured properly and encrypting specific topics as intended.

These metrics are made available automatically once metrics are enabled in the proxy.

4.1.1. KMS metrics

KMS metrics track and count the following types of interactions:

  • Generating DEK pairs

  • Decrypting EDEKs

  • Resolving KEK aliases

Table 1. KMS metrics
Metric Name Type Labels Description

kroxylicious_kms_operation_attempt_total

Counter

operation

Count of the number of KMS operations attempted.

kroxylicious_kms_operation_outcome_total

Counter

operation, outcome

Count of the number of KMS operations grouped by outcome.
Table 2. Labels used on the KMS metrics
Label Domain Description

operation

generate_dek_pair, decrypt_edek, resolve_alias

Type of operation performed.

outcome

SUCCESS, EXCEPTION, NOT_FOUND

Result of the operation.

4.1.2. Encryption accounting metrics

Encryption accounting metrics count the number of records sent to topics that are encrypted and the number of records sent to topics that are not configured for encryption. These metrics are discriminated by topic name. Use these metrics to confirm you configuration is having the effect you desired.

Table 3. Encryption accounting metrics
Metric Name Type Labels Description

kroxylicious_filter_record_encryption_encrypted_records

Counter

topic_name

Count of the number of records encrypted by the filter.

kroxylicious_filter_record_encryption_plain_records

Counter

topic_name

Count of the number of records not encrypted by the filter.

5. Trademark notice

  • Apache Kafka is a registered trademark of The Apache Software Foundation.

  • Kubernetes is a registered trademark of The Linux Foundation.

  • Prometheus is a registered trademark of The Linux Foundation.

  • Strimzi is a trademark of The Linux Foundation.

  • Hashicorp Vault is a registered trademark of HashiCorp, Inc.

  • AWS Key Management Service is a trademark of Amazon.com, Inc. or its affiliates.

  • Fortanix and Data Security Manager are trademarks of Fortanix, Inc.