[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=57521

Mark Thomas  changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution|--- |DUPLICATE

--- Comment #10 from Mark Thomas  ---
I'm marking this as a duplicate as that looks to be the most likely cause based
on the information in this thread.

Regarding comment #9, that may be the same issue or it could be unrelated. If
you still the the issue with the latest Tomcat Native 1.2.x, latest stable
release of a supported Tomcat version and the latest Atmosphere release then
please open a new issue and provide the simplest possible test case that
reproduces the issue. Please also make sure you are using the JSR 356 WebSocket
API, not the deprecated Tomcat WebSocket API.

*** This bug has been marked as a duplicate of bug 51813 ***

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=57521

--- Comment #9 from timofee...@gmail.com ---
We have the same issue but with newer Tomcat and Native library. JVM randomly
crashes with the following dump. Unfortunately it's hard for us to track this
issue down in our environment. So, it's just a comment that the issue is
reproducible with the newer versions.

Tomcat 7.0.59
Websockets and HTTP streaming over Atmopshere Framework
Tomcat Native library 1.1.32
APR version 1.3.9
Java version 1.7.0_21
OS version Red Hat Enterprise Linux 6.6 （64bit）

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7f2376b61b8b, pid=14828, tid=139778958763776
#
# JRE version: 7.0_21-b11
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode linux-amd64
compressed oops)
# Problematic frame:
# C  [libtcnative-1.so+0x13b8b]  Java_org_apache_tomcat_jni_Socket_send+0x7b
#
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try ulimit -c unlimited before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---  T H R E A D  ---

Current thread (0x7f1fd802):  JavaThread Atmosphere-Shared-AsyncOp-13
daemon [_thread_in_native, id=31127,
stack(0x7f20d31f2000,0x7f20d32f3000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
si_addr=0x0040

Registers:
RAX=0x, RBX=0x7f20a82a0c28, RCX=0x0081,
RDX=0x7f20d32ef188
RSP=0x7f20d32ef160, RBP=0x7f1fd80201d8, RSI=0x7f20d32ef190,
RDI=0x
R8 =0x7f20d32ef190, R9 =0x0001, R10=0x7f239d9c7cfb,
R11=0x0001
R12=0x7f20d32ef190, R13=0x004c, R14=0x7f20d32f11f0,
R15=0x7f1fd802
RIP=0x7f2376b61b8b, EFLAGS=0x00010246, CSGSFS=0x0033,
ERR=0x0004
  TRAPNO=0x000e

Top of Stack: (sp=0x7f20d32ef160)
0x7f20d32ef160:   2c4e4f4954415249 69646e6f
0x7f20d32ef170:   4952503d 7f20
0x7f20d32ef180:   6f6d612c44455341 0001
0x7f20d32ef190:   707974227b7c3681 56524553223a2265
0x7f20d32ef1a0:   224f464e 3a2279646f6c
0x7f20d32ef1b0:   726576726573227b 34313a22656d6954
0x7f20d32ef1c0:   3936393736343133 2274222c7d333235
0x7f20d32ef1d0:   373634313334313a 7b7d363630313739
0x7f20d32ef1e0:   3a22644974656222 3438303733313532
0x7f20d32ef1f0:   74756f79756c 363a2265756c6156
0x7f20d32ef200:   34373039392e 313a2274222c5d7d
0x7f20d32ef210:   3739373634313334 2e393a7d34363830
0x7f20d32ef220:   74656c323634 323135323a226449
0x7f20d32ef230:   222c5d7d38373936 34313334313a2274
0x7f20d32ef240:   3436383037393736 3a226449747d7d7d
0x7f20d32ef250:   3238363734313532 313a2274222c5d7d
0x7f20d32ef260:   3639373634313334 3a22647d39393637
0x7f20d32ef270:   3630343535313532 313a2274222c5d7d
0x7f20d32ef280:   3639373634313334 7d33327d35363337
0x7f20d32ef290:   6552746e656d6574 76726553
0x7f20d32ef2a0:   737365733f74656c 317b3d64496e6f69
0x7f20d32ef2b0:   6e756f636361267d 267d327b3d644974
0x7f20d32ef2c0:   7b3d646f69726570 6c61636f6c267d33
0x7f20d32ef2d0:   222c227d347b3d65 22746e756f636361
0x7f20d32ef2e0:   756f636361227b3a 3a2265646f43746e
0x7f20d32ef2f0:   305f615f73617122 6c2230333932
0x7f20d32ef300:   6572727543657361 4247223a2279636e
0x7f20d32ef310:   6f636361222c2250 2265707954746e75
0x7f20d32ef320:   544e45494c43223a 756f636361222c22
0x7f20d32ef330:   697461657243746e 3a22656d69546e6f
0x7f20d32ef340:   3537393334393331 6c3432313435
0x7f20d32ef350:   6572727543657361 696365725079636e 

Instructions: (pc=0x7f2376b61b8b)
0x7f2376b61b6b:   30 48 89 ef 44 89 c9 4d 89 e0 ff 90 40 06 00 00
0x7f2376b61b7b:   48 8b 43 30 48 8d 54 24 28 48 8b 7b 18 4c 89 e6
0x7f2376b61b8b:   ff 50 40 85 c0 0f 84 8a 00 00 00 83 f8 0b 0f 94
0x7f2376b61b9b:   c2 3d c2 d4 01 00 75 65 48 8b 4c 24 28 48 85 c9 

Register to memory mapping:

RAX=0x is an unknown value
RBX=0x7f20a82a0c28 is an unknown value
RCX=0x0081 is an unknown value
RDX=0x7f20d32ef188 is pointing into the stack for thread:
0x7f1fd802
RSP=0x7f20d32ef160 is pointing into the stack for thread:
0x7f1fd802
RBP=0x7f1fd80201d8 is an unknown value
RSI=0x7f20d32ef190 is pointing into the stack for thread:
0x7f1fd802
RDI=0x is an unknown value
R8 =0x7f20d32ef190 is pointing into the stack for thread:
0x7f1fd802
R9 =0x0001 is an unknown value
R10=0x7f239d9c7b10 [CodeBlob (0x7f239d9c7b10)]
Framesize: 10
R11=0x0001 is an unknown value

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://bz.apache.org/bugzilla/show_bug.cgi?id=57521

timofee...@gmail.com changed:

   What|Removed |Added

 CC||timofee...@gmail.com

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

--- Comment #8 from Shunsuke Tanaka tanaka.shuns...@simplex.ne.jp ---
Christopher, Konstantin
Thank you for your comment.


(In reply to Christopher Schultz from comment #6)
 please tell me exactly which package you downloaded.

I downloaded tomcat-native-1.1.30-src.tar.gz.
I builded and upgraded tcnative from 1.1.27 to 1.1.30 in 2014/04/26.


 I'm surprised that the checks already in there aren't protecting you... 
 something deeper must be going on.

If this problem would be caused by tomcat code(if the tcnative receives the
wrong value from Tomcat),
can this symptom occur?
What do you think the following double check?

if(!s-net || s-net != apr_socket_layer){
  tcn_ThrowAPRException(e, APR_EINVALSOCK);
  return -(jint)APR_EINVALSOCK;
}

If the tcnative receives the wrong value from Tomcat, I think this check works
effectively.
And if condition is false and this symptom occur, it seems to me that either of
s-opaque, s-jsbbuff value is broken.
I'm sorry, if I have to say irrelevant things.


 If you built tcnative yourself, re-building it with debug symbols should not 
 be a problem... 

I have to care performance problem to change debug level of tcnative.


 if Tomcat crashes with some regularity, replacing the native library 
 shouldn't represent too much of a burden...

The symptom occured twice a year in production environment.



(In reply to Konstantin Kolinko from comment #7)
 From Tomcat side there have been the following fixes in the Coyote 
 (connector) category:
 7.0.46:
 7.0.54:

If this problem would be caused by tomcat thread-safe or recycle problem, as
you say,
I have a feeling that this symptom never occur by upgrading Tomcat.


 Are you sure that you aren't trying to write to a response that have already 
 been completed and recycled?

How can I check that you say?


 how does it happen that you are directly calling 
 org.apache.catalina.connector.Response method flushBuffer()?

I have no idea.
The stack trace has been writen into the core dump file when the symptom
occured.
When I set breakpoint and step-run, I confirmed that
org.apache.catalina.connector.CoyoteWriter is called.


 org.apache.catalina.connector.RECYCLE_FACADES=true

Thank you for your information.
Also in this case, I have to care performance problem to change this parameter.


I will consider that upgrade Tomcat and/or change RECYCLE_FACADES value to
true.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

--- Comment #7 from Konstantin Kolinko knst.koli...@gmail.com ---
(In reply to Mark Thomas from comment #1)
 You need to upgrade both Tomcat and your version of tc-native to the latest
 versions and re-test. There have been multiple fixes in this area since the
 versions you are using.

From Tomcat side there have been the following fixes in the Coyote
(connector) category:

7.0.46:
55602: Ensure that sockets removed from the Poller and then closed in the
APR/native connector are removed and then closed in a thread-safe manner.

7.0.54:
56399: Assert that both Coyote and Catalina request objects have been properly
recycled.

Maybe there are other changes as well.

It would be better to test with an up-to-date version rather than with 7.0.40. 
(7.0.59 is currently being voted on and is likely to be released in a few
days).

But with comet the cause may actually be in your own web application. Are you
sure that you aren't trying to write to a response that have already been
completed and recycled?

 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
 J  org.apache.tomcat.jni.Socket.sendbb(JII)I
 J  org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V
 J 
 org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/
 ActionCode;Ljava/lang/Object;)V
 J  org.apache.catalina.connector.OutputBuffer.doFlush(Z)V
 J  org.apache.catalina.connector.Response.flushBuffer()V
 J 
 jp.co.xxx.raise.app.fw.socketio.transport.XHRStreaming.flushResponse()Z
 J  scala.concurrent.impl.Future$PromiseCompletingRunnable.run()V
 J  akka.dispatch.TaskInvocation.run()V
 J 
 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/
 ThreadPoolExecutor$Worker;)V
 j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
 j  java.lang.Thread.run()V+11
 v  ~StubRoutines::call_stub

In the above stack trace - how does it happen that you are directly calling
org.apache.catalina.connector.Response method flushBuffer()?

A web application shall not access that connector Response class directly. A
application shall use CometEvent.getHttpServletResponse() which returns a
facade, not the internal object.


If you set the following system property, the facades are recycled when request
processing cycle finishes, protecting Tomcat internals. Using this property
will not help you if you are accessing the internals directly.

org.apache.catalina.connector.RECYCLE_FACADES=true

System properties reference:
http://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html#Security

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

--- Comment #6 from Christopher Schultz ch...@christopherschultz.net ---
(In reply to Shunsuke Tanaka from comment #5)
 Can I ask you  whether Bug #51813 is the same as the problem I have?

The problem is that tcnative crashes in the sendbb function. The cause is
elsewhere, this is just the symptom. So yes, you have the same symptom as
reported in bug #51813.

 I can't judge that, because there are some differences of Java,OS,Tomcat
 version.
 If same problem, I will mark this as a duplicate of bug #51813.
 
 Are you able to re-build tcnative with debugging symbols and do some 
 back-tracing for me?
 
 Do you need back-trace of the time when the problem occured?

If I could get that, it would be best. Also please tell me exactly which
package you downloaded. You have 1.1.30 shown, but 1.1.32 is the latest. It
would be best to build the latest (1.1.30) with debug symbols but if you have
to stick with 1.1.30, that's okay too. Just enough to see which of the
possibilities this could be.

I'm surprised that the checks already in there aren't protecting you...
something deeper must be going on.

 If yes, it is difficult for me that I change the production environment.

If you built tcnative yourself, re-building it with debug symbols should not be
a problem... if Tomcat crashes with some regularity, replacing the native
library shouldn't represent too much of a burden...

 Now I try reproducing the problem in test environment.

Fair enough. Get me whatever data you can.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

--- Comment #5 from Shunsuke Tanaka tanaka.shuns...@simplex.ne.jp ---
Sorry. You're right.
I looked for fixed report of this problem in Tomcat Changelog expectantly, but
I can't find that.
However I got the suggestion Tomcat/Tomcat Native upgrade in comment#1.
I think I overlooked the fixed report, so I asked what version the problem is
fixed.
Can I ask you  whether Bug #51813 is the same as the problem I have?
I can't judge that, because there are some differences of Java,OS,Tomcat
version.
If same problem, I will mark this as a duplicate of bug #51813.

Are you able to re-build tcnative with debugging symbols and do some 
back-tracing for me?

Do you need back-trace of the time when the problem occured?
If yes, it is difficult for me that I change the production environment.
Now I try reproducing the problem in test environment.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

Shunsuke Tanaka tanaka.shuns...@simplex.ne.jp changed:

   What|Removed |Added

 Status|NEEDINFO|NEW

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

--- Comment #2 from tanaka.shuns...@simplex.ne.jp ---
Please tell me the tomcat and tc-native version it had been fixed.
The system run in production environment, so if I upgrade
the tomcat and tc-native, I have to prove to my customer 
this problem never appear.

https://issues.apache.org/bugzilla/show_bug.cgi?id=51813

In the above ticket, it has been commented that Please re-open if it appears.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

Mark Thomas ma...@apache.org changed:

   What|Removed |Added

 Status|NEW |NEEDINFO

--- Comment #1 from Mark Thomas ma...@apache.org ---
You need to upgrade both Tomcat and your version of tc-native to the latest
versions and re-test. There have been multiple fixes in this area since the
versions you are using.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

--- Comment #3 from Christopher Schultz ch...@christopherschultz.net ---
(In reply to Shunsuke Tanaka from comment #2)
 Please tell me the tomcat and tc-native version it had been fixed.
 The system run in production environment, so if I upgrade
 the tomcat and tc-native, I have to prove to my customer 
 this problem never appear.

If you need to prove to your customer that the problem [will] never appear,
then that's your job, since they are paying you to get the job done.

 https://issues.apache.org/bugzilla/show_bug.cgi?id=51813
 
 In the above ticket, it has been commented that Please re-open if it
 appears.

So why didn't you re-open that ticket if you are having the same problem?

Let me save you some time: the patch went into 1.1.28 but it only tries to
prevent the JVM from crashing when the application is doing something it should
not be doing (like keeping references to request and response objects). That
patch fixes nothing other than the symptom (the JVM crash)... the core problem
is misuse of resources.

In reviewing the code (again) for sendbb, there are very few opportunities for
a null-pointer de-reference. Are you able to re-build tcnative with debugging
symbols and do some back-tracing for me? sendbb+0x75 can mean different
things depending upon the environment.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

[Bug 57521] Tomcat randomly crashes with [libtcnative-1.so.0.1.30+0xe965] Java_org_apache_tomcat_jni_Socket_sendbb+0x75.

[bugzilla]

https://issues.apache.org/bugzilla/show_bug.cgi?id=57521

--- Comment #4 from Christopher Schultz ch...@christopherschultz.net ---
Seriously considering marking this as a duplicate of bug #51813 and re-opening
that one, just because it's got better information and a longer history.

-- 
You are receiving this mail because:
You are the assignee for the bug.

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org