[ https://issues.apache.org/jira/browse/OLINGO-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devansh Soni updated OLINGO-1504: --------------------------------- Attachment: JprofilerHeapWalkerGraph.png HeapDumpLargestObjects.png > JVM crashes due to OutOfMemory encountered: Java heap space > ------------------------------------------------------------ > > Key: OLINGO-1504 > URL: https://issues.apache.org/jira/browse/OLINGO-1504 > Project: Olingo > Issue Type: Bug > Components: odata4-client > Affects Versions: (Java) V4 4.7.0 > Reporter: Devansh Soni > Priority: Major > Attachments: HeapDumpLargestObjects.png, JprofilerHeapWalkerGraph.png > > > Hi > The issue occurs for non-paginated OData feeds. The feed I had tested had > 100,000 rows and 9 columns. The JVM crashes due to insufficient heap size and > I can find the stack trace from the hs_err_pidPID log file. > {code:java} > ID Value > 26 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > 27 j java.util.Arrays.copyOf([BI)[B+1 > 28 j java.io.ByteArrayOutputStream.grow(I)V+36 > 29 j java.io.ByteArrayOutputStream.ensureCapacity(I)V+12 > 30 j java.io.ByteArrayOutputStream.write([BII)V+38 > 31 j > org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;[B)J+19 > 32 j > org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;I)J+5 > 33 j > org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J+5 > 34 j > org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I+2 > 35 j > org.apache.olingo.client.core.communication.response.AbstractODataResponse.getRawResponse()Ljava/io/InputStream;+136 > 36 j > org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Lorg/apache/olingo/client/api/domain/ClientEntitySetIterator;+23 > 37 j > org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Ljava/lang/Object;+1 > 38 j > com.tableausoftware.odata.ODataProtocolImpl.fetchV4(Ljava/net/URI;Z)Lcom/tableausoftware/odata/ODataProtocolImpl$ODataResults;+50 > 39 j > com.tableausoftware.odata.ODataResultSetV4.nextBlockImpl()Lcom/tableausoftware/data/generated/DataStream$Block;+23 > 40 j > com.tableausoftware.data.ProtobufResultSet.nextBlock()Lcom/tableausoftware/data/generated/DataStream$Block;+1 > 41 j com.tableau.connect.service.QueryTask.readData()V+46 > 42 j com.tableau.connect.service.QueryTask.call()Ljava/lang/Void;+9 > 43 j com.tableau.connect.service.QueryTask.call()Ljava/lang/Object;+1 > 44 j java.util.concurrent.FutureTask.run()V+42 > 45 j > java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95 > 46 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 > 47 j java.lang.Thread.run()V+11 > 48 v ~StubRoutines::call_stub > {code} > > The issue happens because there are multiple copies of streams being created > in the > AbstractODataResponse.[getRawResponse|https://github.com/apache/olingo-odata4/blob/master/lib/client-core/src/main/java/org/apache/olingo/client/core/communication/response/AbstractODataResponse.java#L300] > method, specifically in the org.apache.commons.io.IOUtils.copy method. To me > it seems like it fails to expand the internal byte buffer when it reaches > capacity. > However, I do not understand why is the response payload being copied in a > ByteArrayOutputStream > {noformat} > org.apache.commons.io.IOUtils.copy(payload, byteArrayOutputStream); > {noformat} > and then again converted into a ByteArrayInputStream. This copying of streams > causes creation of multiple byte buffers which fills up the heap memory. > The > ODataEntitySetIteratorResponseImpl.[getBody|https://github.com/apache/olingo-odata4/blob/master/lib/client-core/src/main/java/org/apache/olingo/client/core/communication/request/retrieve/ODataEntitySetIteratorRequestImpl.java#L78] > call the getRawResponse() in the constructor call of > {noformat} > entitySetIterator = new ClientEntitySetIterator<>( > odataClient, getRawResponse(), > ContentType.parse(getContentType())); > } > {noformat} > > However the > [constructor|https://github.com/apache/olingo-odata4/blob/master/lib/client-api/src/main/java/org/apache/olingo/client/api/domain/ClientEntitySetIterator.java#L79] > of ClientEntitySetIterator accepts InputStream. > So I do not understand the reason behind conversion of the payload into > ByteArrayInputStream in the AbstractODataResponse. I am trying to figure the > reason why this was done versus returning the payload InputStream as-is. > For fixing the problem, we have tried increasing the Java heap size but it is > just a temporary solution since once the OData feed size increases further > beyond a limit, it will fail again. > I also got heap dump for the Java crash and was able to visualize the largest > object byte[] in a jprofiler to reach the same conclusion as above. > -- This message was sent by Atlassian Jira (v8.3.4#803005)