[
https://issues.apache.org/jira/browse/THRIFT-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844933#comment-13844933
]
hiyoucai edited comment on THRIFT-2284 at 12/11/13 1:44 AM:
------------------------------------------------------------
thrift 0.9.1 also crash down:
2013-12-10 23:08:39,307 ERROR [,Thread-46] -
[org.apache.thrift.server.TThreadedSelectorServer] - run() exiting due to
uncaught error
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:371)
at
org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
at
org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:580)
at
org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:540)
was (Author: hiyoucai):
thrift 0.9.41 also crash down:
2013-12-10 23:08:39,307 ERROR [,Thread-46] -
[org.apache.thrift.server.TThreadedSelectorServer] - run() exiting due to
uncaught error
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:371)
at
org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
at
org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:580)
at
org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:540)
> java.lang.OutOfMemoryError: Java heap space
> -------------------------------------------
>
> Key: THRIFT-2284
> URL: https://issues.apache.org/jira/browse/THRIFT-2284
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.7
> Environment: Thrift Server start up by Resin Server at Linux system.
> Reporter: hiyoucai
> Priority: Critical
> Fix For: 0.9.1
>
>
> 【1】The first version application Environment:
> # cat /proc/version
> Linux version 3.5.0-23-generic (buildd@komainu) (gcc version 4.6.3
> (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26
> UTC 2013
> # java -version
> java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
> #thrift version is 0.7
> Java Server code like following:use TNonblockingServer type server.
> public void start() {
> try {
> logger.info("TNonblockingServer start ....");
> TProcessor tprocessor = new
> IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
> TNonblockingServerSocket tnbSocketTransport = new
> TNonblockingServerSocket(19875);
> TNonblockingServer.Args tnbArgs = new
> TNonblockingServer.Args(tnbSocketTransport);
> tnbArgs.processor(tprocessor);
> tnbArgs.transportFactory(new TFramedTransport.Factory());
> tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
> TServer server = new TNonblockingServer(tnbArgs);
> logger.info("TNonblockingServer start OK1....");
> server.serve();
> logger.info("TNonblockingServer start OK2....");
> } catch (Exception e) {
> logger.info("TNonblockingServer start ERROR!!!");
> e.printStackTrace();
> }
> }
> Problem description:
> the thrift server interface invoked per day is much higher.as following:
> # wc -l thrift.log.20131209
> 55598276 thrift.log
> In general, the thrift server will be crashed down about 16 pm.
> I much restart thrift server again an again,
> it is very terrible,my boss will kill me(oh.oh).
> so i use tool to monitor the thrift server.
> linux top command result is that the RES item is 9.2G
> Why use so many memory?
> so crazy.
> So I use Jmap command to print the jvm info that as following:
> Object Histogram:
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 2493990 191419432 char[]
> 2: 1371792 43897344 java.lang.String
> 3: 525011 42306448 int[]
> 4: 356604 17116992 java.nio.HeapByteBuffer
> 5: 299290 14365920 java.nio.HeapCharBuffer
> 6: 70742 10203032 * ConstMethodKlass
> 7: 70742 9629584 * MethodKlass
> 8: 342788 8226912 java.lang.StringBuffer
> 9: 341661 8199864 java.lang.StringBuilder
> 10: 6644 7149072 * ConstantPoolKlass
> 11: 133484 6407232
> org.springframework.aop.framework.ReflectiveMethodInvocation
> 12: 101052 6096736 * SymbolKlass
> 13: 144401 5776040 java.util.HashMap$KeyIterator
> 14: 6644 5039192 * InstanceKlassKlass
> 15: 160734 4881928 java.lang.Object[]
> 16: 149160 4773120 java.util.AbstractList$Itr
> 17: 59294 4269168 org.apache.log4j.spi.LoggingEvent
> 18: 5359 4171640 * ConstantPoolCacheKlass
> 19: 164812 3955488 java.util.LinkedList$Entry
> 20: 109117 3491744 java.util.HashMap$Entry
> 21: 59069 3307864 org.apache.thrift.protocol.TBinaryProtocol
> Just a moment, thrift server throw an exception as following:
> 2013-12-09 16:01:32,181 ERROR [,Thread-44] -
> [org.apache.thrift.server.TNonblockingServer] - run() exiting due to uncaught
> error
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at
> org.apache.thrift.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:491)
> at
> org.apache.thrift.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)
> at
> org.apache.thrift.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)
> at
> org.apache.thrift.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)
> 【2】The seconde version application Environment:
> because The first version application throw an
> java.lang.OutOfMemoryError: Java heap space that i don't know how to do.
> so the first step i want to do is that upgrade my thrift version..
> the version 0.7 upgrade to the version 0.9.1
> of course, I have changed my java code as following:
> use the TThreadedSelectorServer type server.
> try {
> logger.info("TNonblockingServer start ....");
> TProcessor tprocessor = new
> IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
> TNonblockingServerSocket tnbSocketTransport = new
> TNonblockingServerSocket(9876);
> TThreadedSelectorServer.Args tnbArgs = new
> TThreadedSelectorServer.Args(tnbSocketTransport);
> tnbArgs.processor(tprocessor);
> tnbArgs.transportFactory(new TFramedTransport.Factory(256));
> tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
> TServer server = new TThreadedSelectorServer(tnbArgs);
> logger.info("TNonblockingServer start OK1....");
> server.serve();
> logger.info("TNonblockingServer start OK2....");
> } catch (Exception e) {
> logger.info("TNonblockingServer start ERROR!!!");
> e.printStackTrace();
> }
> now it work well one day ,but i use linux top command find that the java
> process use much memory, i worry about that the thrift server will run out
> of memory again.what should i do ?
> Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 97.1%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 16374956k total, 16205088k used, 169868k free, 70912k buffers
> Swap: 3999740k total, 4660k used, 3995080k free, 10265608k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 60104 www-data 20 0 12.4g 4.5g 11m S 71 29.0 154:50.91 java
>
> 52938 root 20 0 264m 37m 1168 S 2 0.2 43:10.60 rsyslogd
>
> 903 root 20 0 17468 1384 944 R 1 0.0 0:00.02 top
>
> 53674 root 20 0 191m 23m 4740 S 1 0.1 71:11.18 admin-agent-msg
>
> 487 root 20 0 0 0 0 S 0 0.0 1:05.70 kworker/13:1
>
> 547 root 20 0 0 0 0 S 0 0.0 34:44.21 kworker/0:2
> jmap info as following:
> Object Histogram:
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 140522 1205495664 byte[]
> 2: 707542 58720520 char[]
> 3: 393972 12607104 java.lang.String
> 4: 143339 12535960 int[]
> 5: 71035 10236744 * ConstMethodKlass
> 6: 71035 9669432 * MethodKlass
> 7: 6674 7173400 * ConstantPoolKlass
> 8: 101411 6121416 * SymbolKlass
> if you have any good idean,please share it to me,thanks.
> address:zhuhai city of china
> QQ:1010859834
> mail:hiyou...@126.com
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
