About this guide

This guide covers using the Kroxylicious Operator to configure, deploy, secure, and operate the Kroxylicious proxy on Kubernetes. Refer to other Kroxylicious guides for information on running the proxy outside Kubernetes or for advanced topics such as plugin development.

1. Kroxylicious Operator overview

Kroxylicious Proxy is an Apache Kafka protocol-aware ("Layer 7") proxy designed to enhance Kafka-based systems.

The Kroxylicious Operator is an operator for Kubernetes which simplifies deploying and operating the Kroxylicious Proxy.

Additional resources

2. API concepts

2.1. API resources used by the Kroxylicious Proxy

The operator takes these custom resources and core Kubernetes resources as inputs:

KafkaProxy

Defines an instance of the proxy.

VirtualKafkaCluster

Represents a logical Kafka cluster that will be exposed to Kafka clients.

KafkaProxyIngress

Configures how a virtual cluster is exposed on the network to Kafka clients.

KafkaService

Specifies a backend Kafka cluster for a virtual cluster.

KafkaProtocolFilter

Specifies filter mechanisms for use with a virtual cluster.

Secret

KafkaService and KafkaProtocolFilter resources may reference a Secret to provide security-sensitive data such as TLS certificates or passwords.

ConfigMap

KafkaService and KafkaProtocolFilter resources may reference a ConfigMap to provide non-sensitive configuration such as trusted CA certificates.

Diagram showing the input resources, KafkaProxy, VirtualKafkaCluster, etc, as boxes, connected by arrows representing the references between the resources.
Figure 1. Example input resources, and the references between them.

Based on the input resources just described, the operator generates the core Kubernetes resources needed to deploy the Kroxylicious proxy, such as:

ConfigMap

Provides the proxy configuration file mounted into the proxy container.

Deployment

Manages the proxy Pod and container.

Service

Exposes the proxy over the network to other workloads in the same Kubernetes cluster.

The API is decomposed into multiple custom resources in a similar way to the Kubernetes Gateway API, and for similar reasons. You can make use of Kubernete’s Role-Based Access Control (RBAC) to divide responsibility for different aspects of the overall proxy functionality to different roles (people) in your organization.

For example, you might grant networking engineers the ability to configure KafkaProxy and KafkaProxyIngress, while giving application developers the ability to configure VirtualKafkaCluster, KafkaService, and KafkaProtocolFilter resources.

Diagram showing the output resources, Deployment, Pod, ConfigMap, Service, with arrows showing the Deployment managing the Pod, the Pod mounting the ConfigMap, and the Service selecting the Pod.
Figure 2. The generated Kubernetes resources, and the relationships between them

2.2. Compatibility

2.2.1. Custom resource APIs

Kroxylicious custom resource definitions are packaged and deployed alongside the operator. Currently, there’s only a single version of the custom resource APIs: v1alpha1.

Future updates to the operator may introduce new versions of the custom resource APIs. At that time the operator will be backwards compatible with older versions of those APIs and an upgrade procedure will be used to upgrade existing custom resources to the new API version.

3. Installing the operator

TO DO

4. Deploying a proxy

Deploy a basic proxy instance with a single virtual cluster exposed to Kafka clients on the same Kubernetes cluster.

4.1. Prerequisites

  • The operator must be installed in the Kubernetes cluster

  • A Kafka cluster to be proxied

4.2. The required resources

4.2.1. Proxy configuration to host virtual clusters

A KafkaProxy resource represents an instance of the Kroxylicious Proxy. Conceptually, it is the top-level resource that links together KafkaProxyIngress, VirtualKafkaCluster, KafkaService, and KafkaProtocolFilter resources to form a complete working proxy.

KafkaProxy resources are referenced by KafkaProxyIngress and VirtualKafkaCluster resources to define how the proxy is exposed and what it proxies.

Example KafkaProxy configuration
kind: KafkaProxy
apiVersion: kroxylicious.io/v1alpha1
metadata:
  namespace: my-proxy
  name: simple
spec: {} (1)
1 An empty spec creates a proxy with default configuration.

4.2.2. Networking configuration for on-cluster access

A KafkaProxyIngress resource defines the networking configuration that allows Kafka clients to connect to a VirtualKafkaCluster.

It is uniquely associated with a single KafkaProxy instance, but it is not uniquely associated with a VirtualKafkaCluster; it can be used by multiple VirtualKafkaCluster instances.

This example shows a KafkaProxyIngress for exposing virtual clusters to Kafka clients running in the same Kubernetes cluster as the proxy.

Example KafkaProxyIngress configuration.
kind: KafkaProxyIngress
apiVersion: kroxylicious.io/v1alpha1
metadata:
  namespace: my-proxy
  name: cluster-ip
spec:
  proxyRef: (1)
    name: simple
  clusterIP: (2)
    protocol: TCP (3)
1 The proxyRef names the KafkaProxy resource that this ingress is part of. It must be in the same namespace as the KafkaProxyIngress.
2 This ingress uses clusterIP networking, which uses Kubernetes Service resources with type: ClusterIP to configure Kubernetes DNS names for the virtual cluster.
3 The protocol is set to accept plain TCP connections. Use TLS for encrypted client-proxy communication.

4.2.3. Configuration for proxied Kafka clusters

A proxied Kafka cluster is configured in a KafkaService resource, which specifies how the proxy connects to the cluster. The Kafka cluster may or may not be running in the same Kubernetes cluster as the proxy: Network connectivity is all that’s required.

This example shows a KafkaService defining how to connect to a Kafka cluster at kafka.example.com.

Example KafkaService configuration
kind: KafkaService
metadata:
  # ...
spec:
  bootstrapServers: kafka.example.com:9092 (1)
  nodeIdRanges: (2)
    - name: brokers (3)
      start: 0 (4)
      end: 5 (5)
  # ...
1 The bootstrapServers property is a comma-separated list of addresses in <host>:<port> format. Including multiple broker addresses helps clients connect when one is unavailable.
2 nodeIdRanges declares the IDs of all the broker nodes in the Kafka cluster
3 name is optional, but specifying it can make errors easier to diagnose.
4 The start of the ID range, inclusive.
5 The end of the ID range, inclusive.

4.2.4. Virtual cluster configuration for in-cluster access without TLS

A VirtualKafkaCluster resource defines a logical Kafka cluster that is accessible to clients over the network.

The virtual cluster references the following:

  • A KafkaProxy resource that the proxy is associated with.

  • One or more KafkaProxyIngress resources that expose the virtual cluster to Kafka clients.

  • A KafkaService resource that defined the backend Kafka cluster.

  • Zero or more KafkaProtocolFilter resources that apply filters to the Kafka protocol traffic passing between clients and the backend Kafka cluster.

This example shows a VirtualKafkaCluster, exposing it to Kafka clients running on the same Kubernetes cluster. It uses plain TCP (as opposed to TLS) as the transport protocol.

Example VirtualKafkaCluster configuration
kind: VirtualKafkaCluster
apiVersion: kroxylicious.io/v1alpha1
metadata:
  name: my-cluster
  namespace: my-proxy
spec:
  proxyRef: (1)
    name: simple
  targetKafkaServiceRef: (2)
    name: my-cluster
  ingresses:
    - ingressRef: (3)
        name: cluster-ip
1 The proxyRef names the KafkaProxy hosting with this virtual cluster. It must be in the same namespace as the VirtualKafkaCluster.
2 The KafkaService that is proxied by the virtual cluster. It must be in the same namespace as the VirtualKafkaCluster.
3 Ingresses to expose the virtual cluster. Each ingress names a KafkaProxyIngress which must be in the same namespace as the VirtualKafkaCluster.

4.3. Filters

A KafkaProtocolFilter resource represents a Kroxylicious Proxy filter. It is not uniquely associated with a VirtualKafkaCluster or KafkaProxy instance; it can be used in a number of VirtualKafkaCluster instances in the same namespace.

A KafkaProtocolFilter is similar to one of the items in a proxy configuration’s filterDefinitions:

  • The resource’s metadata.name corresponds directly to the name of a filterDefinitions item.

  • The resource’s spec.type corresponds directly to the type of a filterDefinitions item.

  • The resource’s spec.configTemplate corresponds to the config of a filterDefinitions item, but is subject to interpolation by the operator.

5. Securing a proxy

Secure proxies by using TLS and storing sensitive values in external resources.

5.1. Prerequisites

  • A running Kroxylicious proxy instance

5.2. Securing the client-to-proxy connection

Secure client-to-proxy communications using TLS.

5.2.1. TLS configuration for client-to-proxy connections

This example shows a VirtualKafkaCluster, exposing it to Kafka clients running on the same Kubernetes cluster. It uses TLS as the transport protocol so that communication between Kafka clients and the proxy is encrypted.

Example VirtualKafkaCluster configuration
kind: VirtualKafkaCluster
apiVersion: kroxylicious.io/v1alpha1
metadata:
  name: my-cluster
  namespace: my-proxy
spec:
  proxyRef: (1)
    name: simple
  targetKafkaServiceRef: (2)
    name: my-cluster
  ingresses:
    - ingressRef: (3)
        name: cluster-ip
      tls: (4)
        certificateRef:
          name: server-certificate
          kind: Secret
1 The proxyRef names the KafkaProxy resource that this virtual cluster is part of. It must be in the same namespace as the VirtualKafkaCluster.
2 The virtual cluster names the KafkaService to be proxied. It must be in the same namespace as the VirtualKafkaCluster.
3 The virtual cluster can be exposed by one or more ingresses. Each ingress must reference a KafkaProxyIngress in the same namespace as the VirtualKafkaCluster.
4 If the ingress supports TLS, the tls property configures the TLS server certificate to use.

Within a VirtualKafkaCluster, an ingress’s tls property configures TLS for that ingress. The tls.certificateRef specifies the Secret resource holding the TLS server certificate that the proxy uses for clients connecting through this ingress. The referenced KafkaProxyIngress also needs to be configured for TLS.

Example KafkaProxyIngress configuration for TLS
kind: KafkaProxyIngress
apiVersion: kroxylicious.io/v1alpha1
metadata:
  name: cluster-ip
  namespace: my-proxy
spec:
  proxyRef: (1)
    name: simple
  clusterIP: (2)
    protocol: TLS (3)

<1>The ingress must reference a KafkaProxy in the same namespace as the KafkaProxyIngress. <2> Exposes the proxy to Kafka clients inside the same Kubernetes cluster using a ClusterIP service. <3> The ingress uses TLS as the transport protocol.

5.3. Securing the proxy-to-broker connection

Secure proxy-to-broker communication using TLS.

5.3.1. TLS trust configuration for proxy-to-cluster connections

By default, the proxy uses the platform’s default trust store when connecting to the proxied cluster over TLS. This works if the cluster’s TLS certificates are signed by a well-known public Certificate Authority (CA), but fails if they’re signed by a private CA instead.

It is good practice to configure trust explicitly, even when proxied cluster’s TLS certificates are signed by a public CA.

This example configures a KafkaService to trust TLS certificates signed by any Certificate Authority (CA) listed in the trusted-cas.pem entry of the ConfigMap named trusted-cas.

Example KafkaService configuration for trusting certificates.
kind: KafkaService
metadata:
  # ...
spec:
  bootstrapServers: kafka.example.com:9092
  tls:
    trustAnchorRef: (1)
      kind: ConfigMap (2)
      name: trusted-cas (3)
      key: trusted-cas.pem (4)
    # ...
1 The trustAnchorRef property references a separate Kubernetes resource which contains the CA certificates to be trusted
2 The kind is optional and defaults to ConfigMap.
3 The name of the resource of the given kind. This resource must exist in the same namespace as the KafkaService
4 The key identifies the entry in the given resource. The corresponding value must be a PEM-encoded set of CA certificates.

5.3.2. TLS authentication to proxied Kafka clusters

Some Kafka clusters require mutual TLS (mTLS) authentication. You can configure the proxy to present a TLS client certificate using the KafkaService resource.

The TLS client certificate you provide must have been issued by a Certificate Authority (CA) that’s trusted by the proxied cluster.

This example configures a KafkaService to use a TLS client certificate stored in a Secret named tls-cert-for-kafka.example.com.

Example KafkaService configuration with TLS client authentication.
kind: KafkaService
metadata:
  # ...
spec:
  bootstrapServers: kafka.example.com:9092
  tls:
    trustAnchorRef:
      kind: ConfigMap
      name: trusted-cas
      key: trusted-cas.pem
    certificateRef: (1)
      kind: Secret (2)
      name: tls-cert-for-kafka.example.com (3)
    # ...
1 The certificateRef property identifies the TLS client certificate to use.
2 The kind is optional and defaults to Secret. The Secret should have type: kubernetes.io/tls.
3 The name is the name of the resource of the given kind. This resource must exist in the same namespace as the KafkaService

5.3.3. TLS version configuration for proxy-to-cluster connections

Some older versions of TLS (and SSL before it) are now considered insecure. They remain enabled by default in order to maximise interoperability between TLS clients and servers which only support older versions.

If the Kafka cluster than you want to connect to supports newer versions of TLS you can disable the proxy’s support for older, insecure versions. For example, if the Kafka cluster supports TLSv1.1, TLSv1.2 and TLSv1.3 you might choose to enable only TLSv1.3 support. This would prevent a TLS downgrade attack.

It is good practice to disable insecure protocol versions.

This example configures a KafkaService to allow only TLS v1.3 when connecting to kafka.example.com.

Example KafkaService with restricted TLS protocol versions.
kind: KafkaService
metadata:
  # ...
spec:
  bootstrapServers: kafka.example.com:9092
  tls:
    # ...
    protocols: (1)
      allowed: (2)
        - TLSv1.3
1 The protocols property configures the TLS protocol versions
2 allowed lists the versions of TLS which are permitted.

The protocols property also supports denied, if you prefer to list the versions to exclude instead.

The names of the TLS protocol versions supported depend on the JVM in the proxy container image. See https://docs.oracle.com/en/java/javase/17/docs/specs/security/standard-names.html#sslcontext-algorithms.

5.3.4. TLS cipher suite configuration for proxy-to-cluster connections

A cipher suite is a set of cryptographic algorithms that together provide the security guarantees offered by TLS. During TLS negotiation a server and client agree on a common cipher suite that they both support.

Some older cipher suites are now considered insecure, but may be enabled on the Kafka cluster to allow older clients to connect.

The cipher suites enabled by default in the proxy depend on the JVM used in the proxy image and the TLS protocol version that is negotiated.

To prevent a TLS downgrade attack you can disable cipher suites known (or thought) to be insecure from the ones that the proxy is allowed to use. However, the proxy and the cluster must support at least one cipher suite in common.

It is good practice to disable insecure cipher suites.
Example KafkaService configured so that the proxy will negotiate TLS connection using only the listed ciphers.
kind: KafkaService
metadata:
  # ...
spec:
  bootstrapServers: kafka.example.com:9092
  tls:
    # ...
    cipherSuites: (1)
      allowed: (2)
       - TLS_AES_128_GCM_SHA256
       - TLS_AES_256_GCM_SHA384
1 The cipherSuites object configures the cipher suites.
2 allowed lists the cipher suites which are permitted.

The cipherSuites property also supports denied, if you prefer to list the cipher suites to exclude instead.

The names of the cipher suites supported depend on the JVM in the proxy container image. See https://docs.oracle.com/en/java/javase/21/docs/specs/security/standard-names.html#jsse-cipher-suite-names.

5.4. Securing filters

Secure filters by using the security features provided by each filter and storing sensitive values in external resources such as a Kubernetes Secret.

5.4.1. Security-sensitive values in filter resources

Template use and value interpolation

Interpolation is supported in spec.configTemplate for the automatic substitution of placeholder values at runtime. This allows security-sensitive values, such as passwords or keys, to be specified in Kubernetes Secret resources rather than directly in the KafkaProtocolFilter resource. Likewise, things like trusted CA certificates can be defined in ConfigMap resources.

The operator determines which Secret and ConfigMap resources are referenced by a KafkaProtocolFilter resource and declares them as volumes in the proxy Pod, mounted into the proxy container. This example shows how to configure the RecordEncryptionFilter using a Vault KMS deployed in the same Kubernetes cluster.

Example KafkaProtocolFilter configuration
kind: KafkaProtocolFilter
metadata:
  # ...
spec:
  type: RecordEncryption (1)
  configTemplate: (2)
    kms: VaultKmsService
    kmsConfig:
      vaultTransitEngineUrl: http://vault.vault.svc.cluster.local:8200/v1/transit
      vaultToken:
        password: ${secret:vault:token} (3)
    selector: TemplateKekSelector
    selectorConfig:
      template: "$(topicName)" (4)
1 The type is the Java class name of the proxy filter. If the unqualified name is ambiguous, it must be qualified by the filter package name.
2 The KafkaProtocolFilter requires a configTemplate, which supports interpolation references.
3 The password uses an interpolation reference, enclosed by ${ and } instead of a literal value. The operator supplies the value at runtime from the specified Secret.
4 The selector template is interpreted by the proxy. It uses different delimiters, $( and ), than the interpolation reference.
Structure of interpolation references

Let’s look at the example interpolation reference ${secret:vault:token} in more detail.

It starts with ${ and ends with }. Between these, it is broken into three parts, separated by colons (:):

  • secret is a provider. Supported providers are secret and configmap (note the use of lower case).

  • vault is a path. The interpretation of the path depends on the provider.

  • token is a key. The interpretation of the key also depends on the provider.

For both secret and configmap providers:

  • The path is interpreted as the name of a Secret or ConfigMap resource in the same namespace as the KafkaProtocolFilter resource.

  • The key is interpreted as a key in the data property of the Secret or ConfigMap resource.

6. Glossary

API

Application Programmer Interface.

CA

Certificate Authority. An organization that issues certificates.

CR

Custom Resource. An instance resource of a CRD. In other words, a resource of a kind that is not built into Kubernetes.

CRD

Custom Resource Definition. A Kubernetes API for defining Kubernetes API extensions.

KMS

Key Management System. A dedicated system for controlling access to cryptographic material, and providing operations which use that material.

mTLS

Mutual Transport Layer Security. A configuration of TLS where the client presents a certificate to a server, which the server authenticates.

TLS

The Transport Layer Security. A secure transport protocol where a server presents a certificate to a client, which the client authenticates. TLS was previously known as the Secure Sockets Layer (SSL).

TCP

The Transmission Control Protocol.

7. Trademark notice

  • Apache Kafka is a registered trademark of The Apache Software Foundation.

  • Kubernetes is a registered trademark of The Linux Foundation.

  • Strimzi is a trademark of The Linux Foundation.

  • Hashicorp Vault is a registered trademark of HashiCorp, Inc.

  • AWS Key Management Service is a trademark of Amazon.com, Inc. or its affiliates.

  • Fortanix and Data Security Manager are trademarks of Fortanix, Inc.