[jira] [Updated] (KAFKA-7625) Kafka Broker node JVM crash - kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction

2018-12-13 Thread JIRA 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sebastian Puzoń updated KAFKA-7625:
---
Attachment: hs_err_pid20124.log
hs_err_pid19405.log
hs_err_pid27156.log

> Kafka Broker node JVM crash - 
> kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction
> 
>
> Key: KAFKA-7625
> URL: https://issues.apache.org/jira/browse/KAFKA-7625
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.0.0
> Environment:  environment:os.version=2.6.32-754.2.1.el6.x86_64 
> java.version=1.8.0_92 
> environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03,
>  built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
> Kafka commitId : 3402a8361b734732 
>Reporter: Sebastian Puzoń
>Priority: Critical
> Attachments: hs_err_pid10238.log, hs_err_pid15119.log, 
> hs_err_pid19131.log, hs_err_pid19405.log, hs_err_pid20124.log, 
> hs_err_pid22373.log, hs_err_pid22386.log, hs_err_pid22633.log, 
> hs_err_pid24681.log, hs_err_pid25513.log, hs_err_pid25701.log, 
> hs_err_pid26844.log, hs_err_pid27156.log, hs_err_pid27290.log, 
> hs_err_pid4194.log, hs_err_pid4299.log
>
>
> I observe broker node JVM crashes with same problematic frame:
> {code:java}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7ff4a2588261, pid=24681, tid=0x7ff3b9bb1700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 
> 1.8.0_92-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 9736 C1 
> kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Either;
>  (518 bytes) @ 0x7ff4a2588261 [0x7ff4a25871a0+0x10c1]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> #
> ---  T H R E A D  ---
> Current thread (0x7ff4b356f800):  JavaThread "kafka-request-handler-3" 
> daemon [_thread_in_Java, id=24781, 
> stack(0x7ff3b9ab1000,0x7ff3b9bb2000)]
> {code}
> {code:java}
> Stack: [0x7ff3b9ab1000,0x7ff3b9bb2000],  sp=0x7ff3b9bafca0,  free 
> space=1019k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native 
> code)
> J 9736 C1 
> kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Either;
>  (518 bytes) @ 0x7ff4a2588261 [0x7ff4a25871a0+0x10c1]
> J 10456 C2 
> kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$5(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;ILscala/Option;)Lscala/util/Either;
>  (192 bytes) @ 0x7ff4a1d413f0 [0x7ff4a1d41240+0x1b0]
> J 9303 C1 
> kafka.coordinator.transaction.TransactionCoordinator$$Lambda$1107.apply(Ljava/lang/Object;)Ljava/lang/Object;
>  (32 bytes) @ 0x7ff4a245f55c [0x7ff4a245f3c0+0x19c]
> J 10018 C2 
> scala.util.Either$RightProjection.flatMap(Lscala/Function1;)Lscala/util/Either;
>  (43 bytes) @ 0x7ff4a1f242c4 [0x7ff4a1f24260+0x64]
> J 9644 C1 
> kafka.coordinator.transaction.TransactionCoordinator.sendTxnMarkersCallback$1(Lorg/apache/kafka/common/protocol/Errors;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lscala/Function1;ILkafka/coordinator/transaction/TxnTransitMetadata;)V
>  (251 bytes) @ 0x7ff4a1ef6254 [0x7ff4a1ef5120+0x1134]
> J 9302 C1 
> kafka.coordinator.transaction.TransactionCoordinator$$Lambda$1106.apply(Ljava/lang/Object;)Ljava/lang/Object;
>  (40 bytes) @ 0x7ff4a24747ec [0x7ff4a24745a0+0x24c]
> J 10125 C2 
> kafka.coordinator.transaction.TransactionStateManager.updateCacheCallback$1(Lscala/collection/Map;Ljava/lang/String;ILkafka/coordinator/transaction/TxnTransitMetadata;Lscala/Function1;Lscala/Function1;Lorg/apache/kafka/common/TopicPartition;)V
>  (892 bytes) @ 0x7ff4a27045ec [0x7ff4a2703c60+0x98c]
> J 10051 C2 
> kafka.coordinator.transaction.TransactionStateManager$$Lambda$814.apply(Ljava/lang/Object;)Ljava/lang/Object;
>

[jira] [Updated] (KAFKA-7625) Kafka Broker node JVM crash - kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction

2018-11-14 Thread JIRA 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sebastian Puzoń updated KAFKA-7625:
---
 Attachment: hs_err_pid4194.log
 hs_err_pid4299.log
 hs_err_pid10238.log
 hs_err_pid15119.log
 hs_err_pid19131.log
 hs_err_pid22373.log
 hs_err_pid22386.log
 hs_err_pid22633.log
 hs_err_pid24681.log
 hs_err_pid25513.log
 hs_err_pid25701.log
 hs_err_pid26844.log
 hs_err_pid27290.log
Description: 
I observe broker node JVM crashes with same problematic frame:
{code:java}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7ff4a2588261, pid=24681, tid=0x7ff3b9bb1700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# J 9736 C1 
kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Either;
 (518 bytes) @ 0x7ff4a2588261 [0x7ff4a25871a0+0x10c1]
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---  T H R E A D  ---

Current thread (0x7ff4b356f800):  JavaThread "kafka-request-handler-3" 
daemon [_thread_in_Java, id=24781, stack(0x7ff3b9ab1000,0x7ff3b9bb2000)]
{code}
{code:java}
Stack: [0x7ff3b9ab1000,0x7ff3b9bb2000],  sp=0x7ff3b9bafca0,  free 
space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 9736 C1 
kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Either;
 (518 bytes) @ 0x7ff4a2588261 [0x7ff4a25871a0+0x10c1]
J 10456 C2 
kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction$5(Lkafka/coordinator/transaction/TransactionCoordinator;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;ILscala/Option;)Lscala/util/Either;
 (192 bytes) @ 0x7ff4a1d413f0 [0x7ff4a1d41240+0x1b0]
J 9303 C1 
kafka.coordinator.transaction.TransactionCoordinator$$Lambda$1107.apply(Ljava/lang/Object;)Ljava/lang/Object;
 (32 bytes) @ 0x7ff4a245f55c [0x7ff4a245f3c0+0x19c]
J 10018 C2 
scala.util.Either$RightProjection.flatMap(Lscala/Function1;)Lscala/util/Either; 
(43 bytes) @ 0x7ff4a1f242c4 [0x7ff4a1f24260+0x64]
J 9644 C1 
kafka.coordinator.transaction.TransactionCoordinator.sendTxnMarkersCallback$1(Lorg/apache/kafka/common/protocol/Errors;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lscala/Function1;ILkafka/coordinator/transaction/TxnTransitMetadata;)V
 (251 bytes) @ 0x7ff4a1ef6254 [0x7ff4a1ef5120+0x1134]
J 9302 C1 
kafka.coordinator.transaction.TransactionCoordinator$$Lambda$1106.apply(Ljava/lang/Object;)Ljava/lang/Object;
 (40 bytes) @ 0x7ff4a24747ec [0x7ff4a24745a0+0x24c]
J 10125 C2 
kafka.coordinator.transaction.TransactionStateManager.updateCacheCallback$1(Lscala/collection/Map;Ljava/lang/String;ILkafka/coordinator/transaction/TxnTransitMetadata;Lscala/Function1;Lscala/Function1;Lorg/apache/kafka/common/TopicPartition;)V
 (892 bytes) @ 0x7ff4a27045ec [0x7ff4a2703c60+0x98c]
J 10051 C2 
kafka.coordinator.transaction.TransactionStateManager$$Lambda$814.apply(Ljava/lang/Object;)Ljava/lang/Object;
 (36 bytes) @ 0x7ff4a1a9cd08 [0x7ff4a1a9cc80+0x88]
J 9349 C2 kafka.server.DelayedProduce.tryComplete()Z (52 bytes) @ 
0x7ff4a1e46e5c [0x7ff4a1e46980+0x4dc]
J 10111 C2 
kafka.server.DelayedOperationPurgatory.tryCompleteElseWatch(Lkafka/server/DelayedOperation;Lscala/collection/Seq;)Z
 (147 bytes) @ 0x7ff4a1c6e000 [0x7ff4a1c6df20+0xe0]
J 10448 C2 
kafka.server.ReplicaManager.appendRecords(JSZZLscala/collection/Map;Lscala/Function1;Lscala/Option;Lscala/Function1;)V
 (237 bytes) @ 0x7ff4a2340b6c [0x7ff4a233f3e0+0x178c]
J 10050 C2 
kafka.coordinator.transaction.TransactionStateManager.$anonfun$appendTransactionToLog$13(Lkafka/coordinator/transaction/TransactionStateManager;Ljava/lang/String;ILkafka/coordinator/transaction/TxnTransitMetadata;Lscala/Function1;Lscala/Function1;Lorg/apache/kafka/common/TopicPartition;Lscala/collection/immutable/Map;)V
 (294 bytes) @