Kroxylicious' composable filter chains and pluggable API mean that you can write your own filters to apply your own rules to the Kafka protocol, using the Java programming language.

In this quick start you will build a custom filter and use it to modify messages being sent to/consumed from Kafka, learn about filter configuration and running custom filters, and find a starting point for developing your own custom filters with your own rules and logic.

1. Getting started

1.1. Prerequisites

To start developing your own custom filters for Kroxylicious, you will need to install JDK 21.

You’ll also need to install the Apache Maven CLI and one of either Podman or Docker.

If you are using Podman, you may encounter issues with the integration tests. There are instructions here to resolve this.

1.2. Generate a Sample Filter Project

The easiest way to learn how to build custom filters is with our kroxylicious-filter-archetype maven archetype, which will generate some basic find-and-replace filters for you to experiment with. Begin by generating a sample project:

mvn archetype:generate -DarchetypeGroupId=io.kroxylicious \
                       -DarchetypeArtifactId=kroxylicious-filter-archetype \
                       -DarchetypeVersion=0.17.0 \
                       -DgroupId=org.example \
                       -DartifactId=sample-filter \
                       -Dversion=1.0-SNAPSHOT \
                       -DinteractiveMode=false
cd sample-filter

2. Build the Sample Filter project

Building the sample project is easy!

mvn verify

Note that the sample project includes automated unit and integration tests. The project’s powerful integration tests run against an in-VM Kafka cluster, enabling you to rapidly iterate on your filter’s business logic.

3. Run the built Sample Filter module in a Proxy

To build and run your sample filter

chmod +x run.sh
./run.sh --config sample-proxy-config.yaml

Send a Kafka message containing foo through the proxy. You should see the content transformed, with the word foo replaced by baz in the message received by the consumer.

podman run -it --net host \
           --entrypoint /opt/kafka/bin/kafka-console-producer.sh \
           apache/kafka:4.1.0 \
           --bootstrap-server localhost:9192 \
           --topic hello
>foo

podman run -it --net host \
           --entrypoint /opt/kafka/bin/kafka-console-consumer.sh \
           apache/kafka:4.1.0 \
           --bootstrap-server localhost:9192 \
           --topic hello \
           --from-beginning
baz

4. Configure the Filter

Filters can be added and removed by altering the filterDefinitions list in the sample-proxy-config.yaml file. You can also reconfigure the sample filters by changing the configuration values in this file. Note that the proxy must be restarted when you modify the configuration.

The SampleFetchResponseFilter and SampleProduceRequestFilter each have two configuration values that must be specified for them to work:

  • findValue - the string the filter will search for in the produce/fetch data

  • replacementValue - the string the filter will replace the value above with

4.1. Default Configuration

The default configuration for SampleProduceRequestFilter is:

filterDefinitions:
  - name: produce-request-filter
    type: SampleProduceRequestFilterFactory
    config:
      findValue: foo
      replacementValue: bar

This means that it will search for the string foo in the produce data and replace all occurrences with the string bar. For example, if a Kafka Producer sent a produce request with data {"myValue":"foo"}, the filter would transform this into {"myValue":"bar"} and Kroxylicious would send that to the Kafka Broker instead.

The default configuration for SampleFetchResponseFilter is:

filterDefinitions:
  - name: fetch-response-filter
    type: SampleFetchResponseFilterFactory
    config:
      findValue: bar
      replacementValue: baz

This means that it will search for the string bar in the fetch data and replace all occurrences with the string baz. For example, if a Kafka Broker sent a fetch response with data {"myValue":"bar"}, the filter would transform this into {"myValue":"baz"} and Kroxylicious would send that to the Kafka Consumer instead.

4.2. Modify

Now that you know how the sample filters work, you can start modifying them! Replace the SampleFilterTransformer logic with your own code, change which messages they apply to, or whatever else you like!