Boosting your Kafka integration tests using Redpanda with Go 🚀

With the example of testcontainers and dockertest

Published in

Level Up Coding

4 min readNov 17, 2022

Integration tests are choreography tests. They do not test business rules. Rather, they test how well the assembly of components dances together. They are plumbing tests that make sure that the components are properly connected and can clearly communicate with each other. [1]

Hello, all! In this article, I want to talk about Redpanda and how we will implement it in our integration tests with the example of two popular libraries; testcontainers and dockertest.

TL;DR: https://github.com/Abdulsametileri/integration-tests-with-redpanda

Figure: Gophers want to learn how to implement Redpanda

What is Redpanda 🤔

Redpanda is a Kafka®-compatible streaming data platform that is up to 10x faster and 6x more hardware-efficient. It is also JVM-free, ZooKeeper®-free, Jepsen-tested, and source available. [2]

Why should I use Redpanda 🤔

With these excellent properties of Redpanda, we can easily use it in our tests without running Kafka and Zookeeper containers. (Better startup time and lack of failure). Also, thanks to its compatibility, there is no need to change any production code in which Kafka resides.

As we already know, writing and maintaining integration tests is tough. We want to get the output of these tests as fast as possible. After replacing our integration test with Redpanda, our tests worked extremely fast compared to old Kafka tests. So I want to share with you 🔥

Before going into the implementation details, I want to inform you; I will use a beautiful testing library testify (the suite and assert packages). So if you don't have any experience, I strongly recommend you to check out.

Note: I will create a wrapper struct for each (TestContainerWrapper, DockerTestWrapper)and write mainly three methods we need: RunContainer , CleanUp , GetBrokerAddresses .

Note 2: I want to test two functionality to ensure that my application can produce a Kafka message and consume a Kafka message successfully.

Let's get started!

👉 Implementation with testcontainers

First, before our integration tests start to work, we must ensure that our setup process is done and ready to move on.

Our test suite is shown below. We created TestContainerWrapper struct and used it with Suite lifecycle methods.

Firstly, we can focus on RunContainer implementation.

After providing the Redpanda image and version, we expose the 9092 port. Be careful; this exposed port number is from the container's perspective! From the host's perspective, Testcontainers exposes this on a random free port. This is designed to avoid port collisions that may arise with locally running software or in between parallel test runs. [3]. So we need to get the mapped port 9092 to connect the brokers from our host machine.

We use default redpanda start command. There are a couple of different parameters; it depends on what you are testing, etc. Redpanda container starts with — dev-container. With this mode, a couple of other parameters are passed by default.

I passed true to the AutoRemove flag, so the container will be removed from the host when stopped.

Lastly, I used to add additional context to my errors (%w) to understand which flow raises an error.

There is no specific logic above. After our integration tests have finished, we should terminate our container. I always try to use context.WithTimeout for these kinds of tasks. Terminate method should make it return in at most 5 seconds.

👉 Implementation with Dockertest

Because the suite lifecycle is the same as the testcontainer section, I skipped that part.

At first glance, you realize that we implement our free port implementation and integrate it with our container. Because there is no exposed random host port as testcontainer, we should do it on our own.

ExposedPorts for the container's perspective.

PortBinding for providing a container and host port mapping binding.

We also need to set advertise-kafka-addr explicitly to provide the connection from our host machine to the container. Why we didn't set it at the testcontainer section 🤔. It is because the container 9092 port is exposed to the free random host port internally. But here, if we don't provide this, we must connect as the host's 9092 port. So we don't want to use a static host port for parallel runs.

We also provide a retryFunc to understand whether our container is ready. RetryFunc tries to connect Kafka brokers, if the connection is successfully established, we can move on or try to reconnect with the adjustable backoff time.

On the cleanUp side, we purge our container, removing the container and linked volumes from docker.

Our tests 🚀

I always try to separate my tests with Given-When-Then. It means in some context, (When) some action is carried out, (Then) a set of consequences should happen.

I also try to name my tests with an underscore. In doing so, it is easy to find failed functions by searching go test output.

Source Code: https://github.com/Abdulsametileri/integration-tests-with-redpanda

Thank you for reading 💛; you can also check my other articles.