Kroxylicious Proxy

First let’s define the concepts in the landscape surrounding Kroxylicious.

Now let’s define some concepts used within Kroxylicious itself.

The proxy is written in Java, on top of Netty. The usual ChannelHandlers provided by the Netty project are used where appropriate (e.g. SSL support uses SslHandler), and Kroxylicious provides Kafka-specific handlers of its own.

The Kafka-aware parts use the Apache Kafka project’s own classes for serialization and deserialization.

Protocol filters get executed using a handler-per-filter model.

The proxy supports a range of possible deployment topologies. Which style is used depends on what the proxy is meant to achieve, architecturally speaking. Broadly speaking a proxy instance can be deployed:

We can further classify deployment topologies in how many proxy instances are used. For example:

Kroxylicious follows Semantic Versioning rules. While we are still in the initial development phase (denoted by a major version 0), we still take API compatibility very seriously. We aim to provide at least two minor releases between deprecation and the removal of that deprecated item.

We also consider our configuration file syntax a public API (though not the Java model backing it). As such, the syntax follows the same Semantic Versioning and deprecation rules.

Kubernetes custom resources are a public API, and we are making every effort to evolve Kroxylicious custom resources in line with Kubernetes best practices. Kubernetes resources have their own versioning scheme, which is independent of the Kroxylicious proxy service version. As a result, we may reach version 1.0.0 of Kroxylicious while still using alpha or beta versions of the custom resources.

Kroxylicious offers the same backwards and forwards compatibility guarantees as Apache Kafka. We support the same range of client and broker versions as the official Apache Kafka Java client.

In the Port Identifies Node scheme, the virtual cluster opens a separate port for each proxied broker in addition to a separate port for the bootstrap.

By default, this scheme assumes that the target cluster comprises three nodes with broker ids 0..2. If this is inadequate, additional configuration can be provided describing the broker topology of the target broker.

This scheme can be used with both plain and TLS downstream connections.

This scheme works best with straightforward configurations where the target cluster uses a known minimum broker ID and uses stable sets of broker IDs. For more complex cases, it is recommended to use the SNI Host Identifies Node scheme.

With the example configuration above, the gateway exposes a target cluster of up to three brokers with node ids 0, 1, 2. The advertised address is defaulted to that of the bootstrap host name. Port numbers are assigned sequentially beginning at bootstrap port number + 1.

The gateway exposes the following three broker addresses:

With the example configuration above, the gateway exposes a target cluster of up to three brokers with node ids 0, 1, 2. The advertised broker address is defined as mycluster.example.com. Port numbers are assigned sequentially beginning at 9200.

The gateway exposes the following three broker addresses:

With the example configuration above, the gateway exposes a target cluster of up to six nodes with node ids 1..3 and 101..103. The advertised broker address is defined as mycluster.example.com. Port numbers are assigned sequentially beginning at 9193 (bootstrap port number + 1).

The gateway exposes the following six broker addresses:

The advertisedBrokerAddressPattern configuration parameter accepts the $(nodeId) replacement token, which is optional. If included, $(nodeId) is replaced by the broker’s node.id (or broker.id) in the target cluster.

In the SNI Host Identifies Node scheme, unique broker hostnames are used to know where to route the traffic. As this scheme relies on SNI (Server Name Indication), which is a TLS extension, TLS connections are required. It cannot be used with plain text connections.

In this scheme, you can either share the port across multiple virtual cluster gateways or assign a separate port for each virtual cluster gateway. However, you cannot use a port that is already assigned to a virtual cluster gateway using the Port Identifies Node scheme.

With the example configuration above, the gateway accepts all traffic on port 9192. Any TLS connections received with the SNI of mycluster.example.com are routed as bootstrap. Any connections received with SNI matching mybroker-$(nodeId).mycluster.example.com are routed to the upstream broker with the same node ID. The configuration exposes a target cluster with any number of brokers. It does not need prior knowledge of the node IDs used by the brokers.

The gateway exposes the following broker addresses:

Both the advertisedBrokerAddressPattern and bootstrapAddress configuration parameters accept the $(virtualClusterName) replacement token, which is optional. If included, $(virtualClusterName) is replaced with the name of the gateway’s virtual cluster.

With the example configuration above, Kroxylicious is instructed to listen on port 9192, but advertise brokers of this virtual cluster as being available on port 443. This feature is useful where a network intermediary (such as another proxy or load balancer) is port forwarding.

The gateway exposes the following broker addresses:

You can enable low level logging on a per-virtual cluster basis. The logNetwork property controls logging of information about requests and responses at the network level, before they’ve been decoded into Kafka requests and responses. The logFrames property controls logging of the decoded requests and responses.

The proxy can run an HTTP server for exposing basic management information. This is configured with the top level management property.

Replace the token <rule definition> in the YAML configuration with either a Schema Validation rule or a JSON Syntax Validation rule depending on your requirements.

The Schema Validation rule validates that the key or value matches a schema identified by its global ID within an Apicurio Schema Registry.

If the key or value does not adhere to the schema, the record will be rejected.

Additionally, if the kafka producer has embedded a global ID within the record it will be validated against the global ID defined by the rule. If they do not match, the record will be rejected. See the Apicurio documentation for details on how the global ID could be embedded into the record. The filter supports extracting ID’s from either the Apicurio globalId record header or from the initial bytes of the serialized content itself.

The JSON Syntax Validation rule validates that the key or value contains only syntactically correct JSON.

There are two steps to using the filter.

Connection metrics count the TCP connections made from the client to the proxy (kroxylicious_client_to_proxy_request_total) and from the proxy to the broker (kroxylicious_proxy_to_server_connections_total). These metrics count connection attempts, so the connection count is incremented even if the connection attempt ultimately fails.

In addition to the count metrics, there are error metrics. * If an error occurs whilst the proxy is accepting a connection from the client the kroxylicious_client_to_proxy_errors_total metric is incremented by one. * If an error occurs whilst the proxy is attempting a connection to a broker the kroxylicious_proxy_to_server_errors_total metric is incremented by one.

Connection and connection error metrics include the following labels: virtual_cluster (the virtual cluster’s name) and node_id (the broker’s node ID). When the client connects to the boostrap endpoint of the virtual cluster, a node ID value of bootstrap is recorded.

The kroxylicious_client_to_proxy_errors_total metric also counts connection errors that occur before a virtual cluster has been identified. For these specific errors, the virtual_cluster and node_id labels are set to an empty string ("").

Message metrics count, and record the sizes of, the Kafka protocol requests and responses that flow through the proxy.

Use these metrics to help understand: * the number of messages flowing through the proxy. * the overall volume of data through the proxy. * the effect the filters are having on the messages.

The size recorded is the encoded size of the protocol message. It includes the 4 byte message size.

Filters can alter the flow of messages through the proxy or the content of the message. This is apparent through the metrics. * If a filter sends a short-circuit, or closes a connection the downstream message counters will exceed the upstream counters. * If a filter changes the size of the message, the downstream size metrics will be different to the upstream size metrics.

Message metrics include the following labels: virtual_cluster (the virtual cluster’s name), node_id (the broker’s node ID), api_key (the message type), api_version, and decoded (a flag indicating if the message was decoded by the proxy).

When the client connects to the boostrap endpoint of the virtual cluster, metrics are recorded with a node ID value of bootstrap.

Add common tags for metrics to appear as labels in the Prometheus scrape.

Micrometer uses the concept of meter binders to register metrics that provide information about the state of some aspect of the application or its container. By registering standard binders included with Micrometer, you can expose metrics about the JVM and system, such as JVM memory usage and garbage collection.

Use the static methods of Micrometer Metrics to register metrics with the global registry.

Alternatively, use Metrics.globalRegistry to get a reference to the global registry. Metrics registered this way are automatically available through the Prometheus scrape endpoint.