Slow Kafka loading and high CPU utilization after upgrading librdkafka from 0.11.6 to 2.4.0

[lazern]

Our app was using 0.11.6 librdkafka earlier to load data from kafka, recently we have upgraded the version of librdkafka to 2.4.0, after that we can see there is a significant degradation in performance.And also seeing high CPU utilization.

As a comparison, in 0.11.6 almost 5000 messages were consumed in 40 seconds where as 2.4.0 loading the same number of messages took 1 minutes 40 seconds.

We are using the below config, and not setting anything else.

queued.min.messages=10000
fetch.message.max.bytes=(24 * 1024 * 1024)
message.max.bytes=(24 * 1024 * 1024)
socket.max.fails=1
enable.partition.eof=true

same configs are in our app across version 0.11.6 and 2.4.0. Do you think if any change in config can help to resolve this situation ?

[emasab]

Please upgrade to latest 2.10.1

[lazern]

Thanks @emasab for the response and the suggestion to upgrade to version 2.10.1.

Could you please clarify if there were specific performance-related improvements or bug fixes introduced between 2.4.0 and 2.10.1 that might address the issue we're facing (i.e., higher latency and CPU usage compared to 0.11.6)?

Also, apart from the configuration parameters we listed earlier:
queued.min.messages=10000
fetch.message.max.bytes=25165824
message.max.bytes=25165824
socket.max.fails=1
enable.partition.eof=true

are there any additional or recommended configuration changes we should consider when using newer versions of librdkafka, especially to improve performance and reduce CPU load?

Appreciate your guidance. Thanks again!