Hi Loren,
This error occurs when your application fails to load the libzstd-jni library file from the temporary library. - zstd-jni v1.3.5-4 (included in Kafka 2.1.0): https://github.com/luben/zstd-jni/blob/v1.3.5-4/src/main/java/com/github/luben/zstd/util/Native.java#L101 - zstd-jni lastest: https://github.com/luben/zstd-jni/blob/master/src/main/java/com/github/luben/zstd/util/Native.java#L137 In short, zstd-jni works like the following: 1. It includes all supported platform's shared library in its jar. 2. When initializing, it copies the appropriate library into the temp directory. 3. Load the extracted library into the memory. So, It would be good to check: 1. What is the platform you are using? 2. Do you have enough space or permission for the temp directory? 3. The error occurs repeatedly? Please have a check and give me a reply. (disclaimer: I added zstd support to Apache Kafka.) Thanks, Dongjin On Thu, Mar 18, 2021 at 12:17 AM Loren Abigail Sion <[email protected]> wrote: > Good day, > > > We're currently in the process of implementing our application with kafka > compression type ZStandard (zstd). > > However during the testing process the consumer encountered this error: > > [ERROR] (consumer-1) > > org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer > - Container exception > org.apache.kafka.common.KafkaException: Received exception when fetching > the next record from dp.-------------------. If needed, please seek past > the record to continue consumption. > at > > org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1228) > at > > org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1400(Fetcher.java:1096) > at > > org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:544) > at > > org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:505) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1225) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1188) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1123) > at > > org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:532) > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.base/java.lang.Thread.run(Thread.java:834) > Caused by: org.apache.kafka.common.KafkaException: > java.lang.ExceptionInInitializerError: Cannot unpack libzstd-jni > > *Here's the version for the producer and consumer:* > > Producer Kafka Client Version (using ZStandard compression): 2.5.1 > Consumer Kafka Client Version: 2.1.0 > > Could you help us identify what caused this error? Do we need to upgrade > the version on the consumer side? > > > Best Regards, > > Loren Sion > -- *Dongjin Lee* *A hitchhiker in the mathematical world.* *github: <http://goog_969573159/>github.com/dongjinleekr <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin <https://speakerdeck.com/dongjin>*
