Kroxylicious is a Java application based on Netty, which means it will run anywhere you can run a JVM. (That’s a lot of places!) To help you get started with Kroxylicious, we’ve created this quick setup guide.
To get started deploying Kroxylicious, you will need to install a Java Runtime Environment (JRE) with minimum version 17. This does not come included with Kroxylicious.
Some operating systems come with a JRE already installed. You can check what Java version you have installed by running the following command:
java -version
If you get an error or the command doesn’t return anything, you may not have a JRE installed, or your JRE may not have been correctly added to your system’s PATH variable.
You will also need a running Apache Kafka® cluster for Kroxylicious to proxy. The official Apache Kafka® quickstart has instructions for setting up a local bare metal cluster.
Once your cluster is set up, the cluster bootstrap address used by Kroxylicious can be changed in the configuration YAML file (see the Configure section below).
Kroxylicious can be downloaded from the releases page of the Kroxylicious GitHub repository, or from Maven Central.
In GitHub, all releases since v0.4.0 have an attached kroxylicious-app-*-bin.zip file. Download the latest version of this zip, and optionally verify the contents of the package with the attached kroxylicious-app-*-bin.zip.asc file.
If you’re trying Kroxylicious out on Linux or macOS, you may find the .tar.gz format easier to work with. We’re using the .zip files in this quickstart for cross-platform compatibility, but we recommend you use whichever format you’re most familiar with.
Extract the downloaded Kroxylicious Zip file into the directory you would like to install Kroxylicious in.
Ensure the kroxylicious-start.sh and run-java.sh files in the bin/ directory within the extracted folder have at least read and execute (r-x) permissions for the owner.
Kroxylicious is configured with YAML. From the configuration file you can specify how Kroxylicious presents each Apache Kafka® broker to clients, where Kroxylicious will locate the Apache Kafka® cluster(s) to be proxied, and which filters Kroxylicious should use along with any configuration for those filters.
An example configuration file can be found in the config/ directory of the extracted Kroxylicious folder, which you can either modify or use as reference for creating your own configuration file.
For this quickstart we will use Kroxylicious in Port-Per-Broker configuration, and assume that both your Apache Kafka® cluster and clients are running on your local machine and using their default configuration. This means we can use the example proxy config file that comes with Kroxylicious.
If your machine uses a non-standard port configuration, or if you have used custom settings for your Apache Kafka® cluster (or if your cluster is running on a different machine) you will need to adjust your Kroxylicious configuration accordingly. More information about configuring Kroxylicious can be found in the documentation.
From within the extracted Kroxylicious folder, run the following command:
./bin/kroxylicious-start.sh --config config/example-proxy-config.yml
To use your own configuration file instead of the example, just replace the file path after --config.
To use your Kroxylicious proxy, your client(s) will need to point to the proxy (using the configured address) rather than directly at the Apache Kafka® cluster.
In each command below, substitute $KROXYLICIOUS_BOOTSTRAP for the bootstrap address of your Kroxylicious instance.
Create a topic called “my_topic” using the kafka-topics.sh command line client:
bin/kafka-topics.sh --create --topic my_topic --bootstrap-server $KROXYLICIOUS_BOOTSTRAP
Produce the string “hello world” to your new topic using the kafka-console-producer.sh command line client:
echo "hello world" | bin/kafka-console-producer.sh --topic my_topic --bootstrap-server $KROXYLICIOUS_BOOTSTRAP
Consume your string (“hello world”) from your topic (“my_topic”) using the kafka-console-consumer.sh command line client:
bin/kafka-console-consumer.sh --topic my_topic --from-beginning --bootstrap-server $KROXYLICIOUS_BOOTSTRAP
In each command below, substitute $KROXYLICIOUS_BOOTSTRAP for the bootstrap address of your Kroxylicious instance.
Create a topic called “my_topic” using Kaf:
kaf -b $KROXYLICIOUS_BOOTSTRAP topic create my_topic
Produce the string “hello world” to your new topic:
echo "hello world" | kaf -b $KROXYLICIOUS_BOOTSTRAP produce my_topic
Consume your string (“hello world”) from your topic (“my_topic”):
kaf -b $KROXYLICIOUS_BOOTSTRAP consume my_topic
Kroxylicious’ composable filter chains and pluggable API mean that you can write your own filters to apply your own rules to the Kafka protocol.
In this quickstart you will build a custom filter and use it to modify messages being sent to/consumed from Kafka, learn about filter configuration and running custom filters, and find a starting point for developing your own custom filters with your own rules and logic.
To start developing your own custom filters for Kroxylicious, you will need to install JDK 21.
You’ll also need to install the Apache Maven CLI and one of either Podman or Docker (Note that if you are using Podman, you may encounter issues with the integration tests. There are instructions here to resolve this).
The easiest way to learn how to build custom filters is with our kroxylicious-sample module, which contains some basic find-and-replace filters for you to experiment with.
Begin by downloading the latest kroxylicious-sample sources from the Kroxylicious repository.
git clone https://github.com/kroxylicious/kroxylicious.git
Building the sample project is easy! You can build the kroxylicious-sample jar either on its own or with the rest of the Kroxylicious project.
To build all of Kroxylicious, including the sample:
mvn verify
Build with the dist profile for creating executable JARs:
mvn verify -Pdist -Dquick
Build both kroxylicious-sample and kroxylicious-app with the dist profile as above, then run the following command:
KROXYLICIOUS_CLASSPATH="kroxylicious-sample/target/*" kroxylicious-app/target/kroxylicious-app-*-bin/kroxylicious-app-*/bin/kroxylicious-start.sh --config kroxylicious-sample/sample-proxy-config.yml
Filters can be added and removed by altering the filters list in the sample-proxy-config.yml file. You can also reconfigure the sample filters by changing the configuration values in this file.
The SampleFetchResponseFilter and SampleProduceRequestFilter each have two configuration values that must be specified for them to work:
findValue - the string the filter will search for in the produce/fetch datareplacementValue - the string the filter will replace the value above withThe default configuration for SampleProduceRequestFilter is:
filters:
- type: SampleProduceRequestFilterFactory
config:
findValue: foo
replacementValue: bar
This means that it will search for the string foo in the produce data and replace all occurrences with the string bar. For example, if a Kafka Producer sent a produce request with data {"myValue":"foo"}, the filter would transform this into {"myValue":"bar"} and Kroxylicious would send that to the Kafka Broker instead.
The default configuration for SampleFetchResponseFilter is:
filters:
- type: SampleFetchResponseFilterFactory
config:
findValue: bar
replacementValue: baz
This means that it will search for the string bar in the fetch data and replace all occurrences with the string baz. For example, if a Kafka Broker sent a fetch response with data {"myValue":"bar"}, the filter would transform this into {"myValue":"baz"} and Kroxylicious would send that to the Kafka Consumer instead.
Now that you know how the sample filters work, you can start modifying them! Replace the SampleFilterTransformer logic with your own code, change which messages they apply to, or whatever else you like!