[ https://issues.apache.org/jira/browse/HBASE-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929708#action_12929708 ]
Kannan Muthukkaruppan commented on HBASE-3199: ---------------------------------------------- Seeing this in testing: {code} 2010-11-08 11:42:10,380 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 62469 caught: java.lang.NullPointerException at org.apache.hadoop.hbase.io.HbaseObjectWritable.getWritableSize(HbaseObjectWritable.java:301) at org.apache.hadoop.hbase.io.HbaseObjectWritable.getWritableSize(HbaseObjectWritable.java:235) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:933) {code} Seems like "instance" inside HBaseObjectWritable can be null, and the code is currently not safely handling that case. > large response handling: some fixups and cleanups > ------------------------------------------------- > > Key: HBASE-3199 > URL: https://issues.apache.org/jira/browse/HBASE-3199 > Project: HBase > Issue Type: Bug > Reporter: Kannan Muthukkaruppan > Assignee: ryan rawson > Attachments: HBASE-3199-2.txt, HBASE-3199-3.txt, HBASE-3199.txt, > HBASE-3199_prelim.txt > > > This may not be common for many use cases, but it might be good to put a > couple of safety nets as well as logging to protect against large responses. > (i) Aravind and I were trying to track down why JVM memory usage was > oscillating so much when dealing with very large buffers rather than OOM'ing > or hitting some Index out of bound type exception, and this is what we found. > java.io.ByteArrayOutputStream graduates its internal buffers by doubling > them. Also, it is supposed to be able to handle "int" sized buffers (2G). The > code which handles "write" (in jdk 1.6) is along the lines of: > {code} > public synchronized void write(byte b[], int off, int len) { > if ((off < 0) || (off > b.length) || (len < 0) || > ((off + len) > b.length) || ((off + len) < 0)) { > throw new IndexOutOfBoundsException(); > } else if (len == 0) { > return; > } > int newcount = count + len; > if (newcount > buf.length) { > buf = Arrays.copyOf(buf, Math.max(buf.length << 1, newcount)); > } > System.arraycopy(b, off, buf, count, len); > count = newcount; > } > {code} > The "buf.length << 1" will start producing -ve values when buf.length reaches > 1G, and "newcount" will instead dictate the size of the buffer allocated. At > this point, all attempts to write to the buffer will grow linearly, and the > buffer will be resized by only the required amount on each write. > Effectively, each write will allocate a new 1G buffer + reqd size buffer, > copy the contents, and so on. This will put the process in heavy GC mode > (with jvm heap oscillating by several GBs rapidly), and render it practically > unusable. > (ii) When serializing a Result, the writeArray method doesn't assert that the > resultant size does not overflow an "int". > {code} > int bufLen = 0; > for(Result result : results) { > bufLen += Bytes.SIZEOF_INT; > if(result == null || result.isEmpty()) { > continue; > } > for(KeyValue key : result.raw()) { > bufLen += key.getLength() + Bytes.SIZEOF_INT; > } > } > {code} > We should do the math in "long" and assert on bufLen values > > Integer.MAX_VALUE. > (iii) In HBaseServer.java on RPC responses, we could add some logging on > responses above a certain thresholds. > (iv) Increase buffer size threshold for buffers that are reused by RPC > handlers. And make this configurable. Currently, any response buffer about > 16k is not reused on next response. (HBaseServer.java). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.