[ https://issues.apache.org/jira/browse/KAFKA-17227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870293#comment-17870293 ]
Rob Young commented on KAFKA-17227: ----------------------------------- And a quick check of the prior version 3.7.1 shows that it has the same issue with zstd/snappy compressed data if /tmp is noexec. The kafka-storage script runs without error, the broker starts without error. But it will fail to decrypt messages received that use snappy or zstd compression. {code:java} podman run -it --tmpfs /tmp:rw,size=787448k,mode=1777,noexec apache/kafka:3.7.1 /bin/bash 9be0542d2aaf:/$ cd /opt/kafka && KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" && bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty, dirs={/tmp/kraft-combined-logs: EMPTY}) Formatting /tmp/kraft-combined-logs with metadata.version 3.7-IV4. 9be0542d2aaf:/opt/kafka$ bin/kafka-server-start.sh config/kraft/server.properties ... server starts {code} and in another terminal exec in: {code:java} 9be0542d2aaf:/opt/kafka$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic 1 --compression-codec snappy >a [2024-08-01 21:35:37,967] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 6 : {1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) org.apache.kafka.common.KafkaException: java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.10-2c2e89fc-1c41-44f0-9cac-b98a64794476-libsnappyjava.so: Error loading shared library /tmp/snappy-1.1.10-2c2e89fc-1c41-44f0-9cac-b98a64794476-libsnappyjava.so: Operation not permitted at org.apache.kafka.common.compress.SnappyFactory.wrapForOutput(SnappyFactory.java:38) {code} so a client side exception, if you fix it with the system prop: {code:java} export KAFKA_OPTS="-Dorg.xerial.snappy.tempdir=/opt/kafka/tmp" 9be0542d2aaf:/opt/kafka$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic 1 --compression-codec snappy >a >b[2024-08-01 21:37:01,833] ERROR Error when sending message to topic 1 with >key: null, value: 1 bytes with error: >(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request. > >c >[2024-08-01 21:37:02,880] ERROR Error when sending message to topic 1 with >key: null, value: 1 bytes with error: >(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request. [2024-08-01 21:37:02,881] ERROR Error when sending message to topic 1 with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request. {code} you get broker side exceptions at decrypt time {code:java} [2024-08-01 21:35:38,012] INFO [Partition 1-0 broker=1] Log loaded for partition 1-0 with initial high watermark 0 (kafka.cluster.Partition) [2024-08-01 21:37:01,822] ERROR [ReplicaManager broker=1] Error processing append operation on partition 1-0 (kafka.server.ReplicaManager) java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.10-d22e3d69-18fd-4a0f-b9fa-40aab1d9cbd4-libsnappyjava.so: Error loading shared library /tmp/snappy-1.1.10-d22e3d69-18fd-4a0f-b9fa-40aab1d9cbd4-libsnappyjava.so: Operation not permitted at java.base/jdk.internal.loader.NativeLibraries.load(Native Method) {code} > Apache Kafka 3.8.0 /tmp exec permission > --------------------------------------- > > Key: KAFKA-17227 > URL: https://issues.apache.org/jira/browse/KAFKA-17227 > Project: Kafka > Issue Type: Bug > Affects Versions: 3.8.0 > Reporter: Francisco Martinez > Assignee: Josep Prat > Priority: Major > Attachments: kafka_issue.png > > > I have just downloaded and installed new Apache Kafka version 3.8.0. > It does not work for me (version 3.7.1 works fine). > In SLES 15 SP5, i have configured /etc/fstab to do not have exec permission > for the /tmp partition (noexec) (as suggested by the CIS benchmark). > Then the Kafka service does not start. Even the kafka-storage.sh script > cannot be executed to create the Kafka storage in /tmp/kraft-combined-logs. > The error reported (in all cases) is exception > java.lang.UnsatisfiedLinkError: /tmp/lib/libzstd-jni-1.5.6-3.....so: failed > to map segment from shared object. The error does not appear if i enable > again the exec permission in /tmp (i.e. mount -o remount,exec /tmp). > It seems that the zstd-jni-1.5.6-3.jar library is tried to be loaded (even in > the case the compression is disabled by default in producer.properties: > compression.type=none). Inside the jar file there is for example > lizstd-jni-1.5.6-3.so for amd64 architecture that is used by the jar, and for > that purpose it is copied to /tmp. But if the /tmp does not have execution > permissions, the Apache Kafka processes don't start. > Maybe the issue is in zstd-jni and has to be solved in zstd-jni, or maybe the > library could be imported only when necessary (only in case the compression > is used) to minimize the issue with zstd-jni. > Thanks and regards. -- This message was sent by Atlassian Jira (v8.20.10#820010)