https://bz.apache.org/bugzilla/show_bug.cgi?id=62626
--- Comment #16 from Christopher Schultz <ch...@christopherschultz.net> --- (In reply to jan.pfeifer from comment #13) > Two last Thread reports: > > --------------- T H R E A D --------------- > > Current thread (0x000000002f30d800): JavaThread > "https-openssl-apr-443-exec-69" daemon [_thread_in_native, id=36360, > stack(0x000000003f5e0000,0x000000003f6e0000)] > > Stack: [0x000000003f5e0000,0x000000003f6e0000], sp=0x000000003f6ddd80, > free space=1015k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [tcnative-1.dll+0xe0a8f] > No more information about the native frame. That must be a direct-native call... > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > J 10218 org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I (0 > bytes) @ 0x0000000014d3c59f [0x0000000014d3c540+0x000000000000005f] > J 15904 c2 org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking()V (90 > bytes) @ 0x000000001545cbcc [0x000000001545c880+0x000000000000034c] > J 16170 c2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z (20 bytes) > @ 0x0000000014860a78 [0x0000000014860a40+0x0000000000000038] > J 15983 c2 org.apache.catalina.connector.CoyoteOutputStream.write([BII)V (26 > bytes) @ 0x00000000154a8f2c [0x00000000154a51c0+0x0000000000003d6c] > J 16090 c2 java.io.BufferedOutputStream.write([BII)V java.base@10.0.2 (67 > bytes) @ 0x00000000154e5660 [0x00000000154e5560+0x0000000000000100] > J 18370 c2 > com.m2000.shop.controllers.DefaultController.image(Ljava/lang/String;Ljava/ > lang/String;Ljava/lang/String;Ljavax/servlet/http/HttpServletResponse;Ljavax/ > servlet/http/HttpServletRequest;)V (1129 bytes) @ 0x00000000159f12f4 > [0x00000000159ec920+0x00000000000049d4] Okay, this is good information: 1. The crash is occurring in jni.Socket.sendbb (where many crashes have historically been found) 2. Your code is intentionally invoking a write() on the socket/stream That means that we can take a look at your code to see if there is any misuse of the streams. That is the #1 reason for crashes like these: the application misuses a stream (e.g. after it should have been recycled) and it's in an intermediate state. tcnative should not crash, but we may be able to mitigate it by fixing your code. Can you post as much of your code as possible? Feel free to do so privately if you don't want your code published in BZ. > J 19092 c2 > jdk.internal.reflect.GeneratedMethodAccessor159.invoke(Ljava/lang/Object; > [Ljava/lang/Object;)Ljava/lang/Object; (98 bytes) @ 0x0000000015ba9be8 > [0x0000000015ba9b00+0x00000000000000e8] > J 19001 c2 > org.springframework.web.method.support.InvocableHandlerMethod. > invokeForRequest(Lorg/springframework/web/context/request/NativeWebRequest; > Lorg/springframework/web/method/support/ModelAndViewContainer;[Ljava/lang/ > Object;)Ljava/lang/Object; (148 bytes) @ 0x0000000015b26018 > [0x0000000015b25ca0+0x0000000000000378] Coming through Spring. > J 18530 c2 > org.springframework.web.servlet.mvc.method.annotation. > ServletInvocableHandlerMethod.invokeAndHandle(Lorg/springframework/web/ > context/request/ServletWebRequest;Lorg/springframework/web/method/support/ > ModelAndViewContainer;[Ljava/lang/Object;)V (142 bytes) @ 0x0000000015a6e884 > [0x0000000015a6e840+0x0000000000000044] > J 18568 c2 > org.springframework.web.servlet.mvc.method.annotation. > RequestMappingHandlerAdapter.invokeHandlerMethod(Ljavax/servlet/http/ > HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/ > springframework/web/method/HandlerMethod;)Lorg/springframework/web/servlet/ > ModelAndView; (328 bytes) @ 0x0000000015a92a48 > [0x0000000015a90d60+0x0000000000001ce8] > J 19592 c2 > org.springframework.web.servlet.DispatcherServlet.doDispatch(Ljavax/servlet/ > http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (594 > bytes) @ 0x0000000015ca8c20 [0x0000000015ca8140+0x0000000000000ae0] > J 18099 c2 > org.springframework.web.servlet.DispatcherServlet.doService(Ljavax/servlet/ > http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (360 > bytes) @ 0x0000000015948490 [0x0000000015947ee0+0x00000000000005b0] > J 18074 c2 > org.springframework.web.servlet.FrameworkServlet.processRequest(Ljavax/ > servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V > (298 bytes) @ 0x0000000015919044 [0x00000000159171c0+0x0000000000001e84] > J 19591 c2 > javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/ > HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (269 bytes) @ > 0x0000000015ca6d70 [0x0000000015ca6bc0+0x00000000000001b0] >From normal Tomcat dispatch. Great. At least there is no odd threading or anything going on, here. That makes it less likely to be a problem in your code. > siginfo: EXCEPTION_ACCESS_VIOLATION (0xc0000005), writing address > 0x00000000000001a0 Interesting. Most of the crashes in the past have been when reading data, not writing it. Therefore, most of the protections we have added have been e.g. null-checking before dereference, etc. 0x01a0 (416 decimal) is likely the byte-offset into a structure that tcnative is trying to write. That might help narrow-down what structure is being written (416 bytes is somewhat large), and even which field. > Register to memory mapping: > > RIP=0x00000001800e0a8f tcnative-1.dll > RAX=0x0000000000000000 is an unknown value > RBX=0x000000003c892b10 is an unknown value > RCX=0x00000001801ccca0 tcnative-1.dll > RDX=0x00000000fffffff0 is an unknown value > RSP=0x000000003f6ddd80 is pointing into the stack for thread: > 0x000000002f30d800 > RBP=0x0000000000000009 is an unknown value > RSI=0x0000000000000017 is an unknown value > RDI=0x00000000ffffffff is an unknown value > R8 =0x0000000000000040 is an unknown value > R9 =0x0000000000000051 is an unknown value > R10=0x00000000000000a4 is an unknown value > R11=0x00000000ffffffff is an unknown value > R12=0x000000003c893a36 is an unknown value > R13=0x000000003c870680 is an unknown value > R14=0x0000000000000000 is an unknown value > R15=0x0000000000000000 is an unknown value > > > Registers: > RAX=0x0000000000000000, RBX=0x000000003c892b10, RCX=0x00000001801ccca0, > RDX=0x00000000fffffff0 > RSP=0x000000003f6ddd80, RBP=0x0000000000000009, RSI=0x0000000000000017, > RDI=0x00000000ffffffff > R8 =0x0000000000000040, R9 =0x0000000000000051, R10=0x00000000000000a4, > R11=0x00000000ffffffff > R12=0x000000003c893a36, R13=0x000000003c870680, R14=0x0000000000000000, > R15=0x0000000000000000 > RIP=0x00000001800e0a8f, EFLAGS=0x0000000000010286 > > Top of Stack: (sp=0x000000003f6ddd80) > 0x000000003f6ddd80: 000000003c892b10 0000000000000008 > 0x000000003f6ddd90: 000000003c892b10 000000003c870680 > 0x000000003f6ddda0: 00000000000003e8 0000000070675957 > 0x000000003f6dddb0: 0000000000000000 0000000070687c26 > 0x000000003f6dddc0: 0000001700000009 0000000000000009 > 0x000000003f6dddd0: 000000002d170e00 0000000000000000 > 0x000000003f6ddde0: 0000000000000000 0000000000000000 > 0x000000003f6dddf0: 0000000000000000 000000002d170e00 > 0x000000003f6dde00: 0000000000000000 0000000000000000 > 0x000000003f6dde10: 000021823ec6d43c 00000001800e0075 > 0x000000003f6dde20: 000000003f6ddf60 000000002d170e00 > 0x000000003f6dde30: 000000003f6ddf08 0000000000000009 > 0x000000003f6dde40: 000000002d170e00 0000000000000009 > 0x000000003f6dde50: 000000003c892b10 00000001800e5c2e > 0x000000003f6dde60: 0000000000000000 0000000000000000 > 0x000000003f6dde70: 000000002a992ba0 000000002f30d800 > > Instructions: (pc=0x00000001800e0a8f) > 0x00000001800e0a6f: f6 41 70 08 75 08 48 8b cb e8 d3 64 00 00 42 8d > 0x00000001800e0a7f: 04 37 e9 de f7 ff ff 33 ff 48 8b 83 80 00 00 00 > 0x00000001800e0a8f: 44 89 b0 a0 01 00 00 8b c7 e9 c7 f7 ff ff ba 9e > 0x00000001800e0a9f: 00 00 00 4c 8d 0d a7 c7 0e 00 b9 14 00 00 00 44 Disassembly (x86-64): 0: f6 41 70 08 test BYTE PTR [rcx+0x70],0x8 4: 75 08 jne 0xe 6: 48 8b cb mov rcx,rbx 9: e8 d3 64 00 00 call 0x64e1 e: 42 8d 04 37 lea eax,[rdi+r14*1] 12: e9 de f7 ff ff jmp 0xfffffffffffff7f5 17: 33 ff xor edi,edi 19: 48 8b 83 80 00 00 00 mov rax,QWORD PTR [rbx+0x80] 20: 44 89 b0 a0 01 00 00 mov DWORD PTR [rax+0x1a0],r14d 27: 8b c7 mov eax,edi 29: e9 c7 f7 ff ff jmp 0xfffffffffffff7f5 2e: ba 9e 00 00 00 mov edx,0x9e 33: 4c 8d 0d a7 c7 0e 00 lea r9,[rip+0xec7a7] # 0xec7e1 3a: b9 14 00 00 00 mov ecx,0x14 3f: 44 rex.R Byte offset 20 has an instruction that references RAX + 0x01a0, and the value of RAX is 0x0000, so we try to copy the contents of R14 into an illegal address. Simple, right? > C [tcnative-1.dll+0xe0a8f] Same. > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > J 8894 org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I (0 > bytes) @ 0x000000001540cf1f [0x000000001540cec0+0x000000000000005f] Same. > J 14207 c2 > org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjava/nio/ > ByteBuffer;)V (242 bytes) @ 0x000000001591c658 > [0x000000001591c480+0x00000000000001d8] > J 16174 c2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z (20 bytes) > @ 0x0000000015c49db0 [0x0000000015c49d00+0x00000000000000b0] > j org.apache.coyote.http2.Http2UpgradeHandler.writeGoAwayFrame(IJ[B)V+125 > j > org.apache.coyote.http2.Http2UpgradeHandler.closeConnection(Lorg/apache/ > coyote/http2/Http2Exception;)V+22 > j > org.apache.coyote.http2.Stream.close(Lorg/apache/coyote/http2/Http2Exception; > )V+124 > J 18503 c2 org.apache.coyote.http2.StreamRunnable.run()V (12 bytes) @ > 0x00000000156be630 [0x00000000156bdfa0+0x0000000000000690] > j > java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ > ThreadPoolExecutor$Worker;)V+92 java.base@10.0.2 > j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@10.0.2 > j org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+4 > j java.lang.Thread.run()V+11 java.base@10.0.2 > v ~StubRoutines::call_stub This one is different: Tomcat is trying to close the stream after your code is done and is failing. > siginfo: EXCEPTION_ACCESS_VIOLATION (0xc0000005), writing address > 0x00000000000001a0 But this looks familiar. > Instructions: (pc=0x00000001800e0a8f) > 0x00000001800e0a6f: f6 41 70 08 75 08 48 8b cb e8 d3 64 00 00 42 8d > 0x00000001800e0a7f: 04 37 e9 de f7 ff ff 33 ff 48 8b 83 80 00 00 00 > 0x00000001800e0a8f: 44 89 b0 a0 01 00 00 8b c7 e9 c7 f7 ff ff ba 9e > 0x00000001800e0a9f: 00 00 00 4c 8d 0d a7 c7 0e 00 b9 14 00 00 00 44 Yep, same native code. Obviously, something is wrong with the socket-state management. My guess is that this is really "client has disconnected" and the state isn't being correctly detected by tcnative. But why is it not crashing on Java 8? -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
