[ https://issues.apache.org/jira/browse/KAFKA-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803933#comment-16803933 ]
Sean Humbarger commented on KAFKA-8103: --------------------------------------- We are still seeing random JVM crashes. We've switched over from OpenJDK to Oracle 1.8.202 and see the same thing: {code} # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f6c9cd85100, pid=4550, tid=0x00007f6a64792700 # # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build 1.8.0_202-b08) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode linux-amd64 compressed oops) # Problematic frame: # V [libjvm.so+0x2c7100] Handle::Handle(Thread*, oopDesc*)+0x0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x00007f6c99279000): JavaThread "kafka-request-handler-3" daemon [_thread_in_vm, id=4984, stack(0x00007f6a64692000,0x00007f6a64793000)] siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007f6c9cd85100 Registers: RAX=0x00007f6c9da7b2ce, RBX=0x00007f6c99279000, RCX=0x00000005502d3250, RDX=0x00000005502d3250 RSP=0x00007f6a647913d8, RBP=0x00007f6a64791410, RSI=0x00007f6c99279000, RDI=0x00007f6a647913e8 R8 =0x00000000aa05a64a, R9 =0x0000000550a3d9b8, R10=0x00007f6c9d488af0, R11=0x0000000000000002 R12=0x00007f6a64791470, R13=0x00000005502d3250, R14=0x00007f6c9da85f8c, R15=0x00007f6c99279000 RIP=0x00007f6c9cd85100, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000015 TRAPNO=0x000000000000000e Top of Stack: (sp=0x00007f6a647913d8) 0x00007f6a647913d8: 00007f6c9d488b38 0000007000000070 0x00007f6a647913e8: 00007f6c8a4e702c 00007f67d003adaa 0x00007f6a647913f8: 0000000000000000 00000000f2e95d57 0x00007f6a64791408: 00000000a9e2f767 00000000a9e05757 0x00007f6a64791418: 00007f6c88e78c88 00000000a9e05757 0x00007f6a64791428: 00007f6c8a0d149c 00000005aa05a64a 0x00007f6a64791438: 00000005502d3250 00000007974aeab8 0x00007f6a64791448: 000000054f02bab8 000000054f17bb60 0x00007f6a64791458: 00007f6c9dbbacdd 00000007974adfe0 0x00007f6a64791468: 00007f6c9d3cc53f 0000000000000003 0x00007f6a64791478: 000000000d70aa32 000000054f037260 0x00007f6a64791488: 00007f6c8ab3cc88 aa147bbef4590578 0x00007f6a64791498: 00000007a2c82bc0 00000007974ae068 0x00007f6a647914a8: 00000007974adcb0 00000007974adfe0 0x00007f6a647914b8: 0000000550a3ddf0 00000007974adcf8 0x00007f6a647914c8: 00000007974adc48 00000007a2c83420 0x00007f6a647914d8: 00007f6cf2e95b69 00000000a9e269d8 0x00007f6a647914e8: 00007f6c8a2c0934 00000007974adba0 0x00007f6a647914f8: 00007f6c8a000138 00007f6a64791550 0x00007f6a64791508: 00007ffed59c2c60 00007f6a64791580 0x00007f6a64791518: 00000002f4590578 000000079d149f38 0x00007f6a64791528: 0000000550a3ddf0 00007f6a64791570 0x00007f6a64791538: 0000000000000000 00000000f4590684 0x00007f6a64791548: 000000079d1b3490 00007f6a64791590 0x00007f6a64791558: 00007f6c9dbbacdd 00000007f3a366fd 0x00007f6a64791568: 00007f6c9dbbacdd 00000007974adb60 0x00007f6a64791578: 00007f6c9d3cc53f 0000000000011172 0x00007f6a64791588: 000000000d7091ab 00000000f4590567 0x00007f6a64791598: 00007f6c8a887a24 00000007a2c82bc0 0x00007f6a647915a8: 00000007a2c832d0 aa147bbe000015bd 0x00007f6a647915b8: f2e95b56974adb48 00000005f2e95b67 0x00007f6a647915c8: 00000007974adb38 00000007974adab0 Instructions: (pc=0x00007f6c9cd85100) 0x00007f6c9cd850e0: e8 0b 4c 77 00 48 83 c4 30 5b 41 5c 5d c3 66 90 0x00007f6c9cd850f0: f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0x00007f6c9cd85100: 48 85 d2 74 63 55 48 89 e5 41 55 41 54 53 49 89 0x00007f6c9cd85110: fc 48 89 d3 48 83 ec 08 4c 8b ae 38 01 00 00 49 Register to memory mapping: RAX=0x00007f6c9da7b2ce: <offset 0xfbd2ce> in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so at 0x00007f6c9cabe000 RBX=0x00007f6c99279000 is a thread RCX=0x00000005502d3250 is an oop java.lang.Object - klass: 'java/lang/Object' RDX=0x00000005502d3250 is an oop java.lang.Object - klass: 'java/lang/Object' RSP=0x00007f6a647913d8 is pointing into the stack for thread: 0x00007f6c99279000 RBP=0x00007f6a64791410 is pointing into the stack for thread: 0x00007f6c99279000 RSI=0x00007f6c99279000 is a thread RDI=0x00007f6a647913e8 is pointing into the stack for thread: 0x00007f6c99279000 R8 =0x00000000aa05a64a is an unknown value R9 =0x0000000550a3d9b8 is an oop org.apache.kafka.common.utils.KafkaThread - klass: 'org/apache/kafka/common/utils/KafkaThread' R10=0x00007f6c9d488af0: <offset 0x9caaf0> in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so at 0x00007f6c9cabe000 R11=0x0000000000000002 is an unknown value R12=0x00007f6a64791470 is pointing into the stack for thread: 0x00007f6c99279000 R13=0x00000005502d3250 is an oop java.lang.Object - klass: 'java/lang/Object' R14=0x00007f6c9da85f8c: <offset 0xfc7f8c> in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so at 0x00007f6c9cabe000 {code} > Kafka SIGSEGV on kafka-network-thread > ------------------------------------- > > Key: KAFKA-8103 > URL: https://issues.apache.org/jira/browse/KAFKA-8103 > Project: Kafka > Issue Type: Bug > Affects Versions: 1.1.1 > Environment: OS > Amazon Linux > Kernel > 4.14.97-74.72.amzn1.x86_64 #1 SMP Tue Feb 5 20:59:30 UTC 2019 x86_64 x86_64 > x86_64 GNU/Linux > Java > openjdk version "1.8.0_191" > OpenJDK Runtime Environment (build 1.8.0_191-b12) > OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) > AWS Instance Type > c5.4xlarge > Reporter: Sean Humbarger > Priority: Major > Attachments: hs_err_pid4345.log > > > We have a 4 node cluster (6 topics, 6 consumer groups) that is processing > 65,000 messages per second and are seeing SIGSEGV crashes at least once a day > (see attachment). Each broker has six disks attached to it to support the > kafka logs. When the crash occurs, we simply restart kafka and everything > seems fine. We don't see anything out of the ordinary in /var/log/messages > or dmesg when the crashes occur. Thus far, we are unable to predict during > the day when the crash will occur or which node it will occur on. > > The problematic frame is as follows: > {code:java} > # Problematic frame: > # J 8628 C2 > org.apache.kafka.common.metrics.stats.Max.update(Lorg/apache/kafka/common/metrics/stats/SampledStat$Sample;Lorg/apache/kafka/common/metrics/MetricConfig;DJ)V > (13 bytes) @ 0x00007ff779f9fca0 [0x00007ff779f9fc80+0x20] > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)