[
https://issues.apache.org/jira/browse/OLINGO-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devansh Soni updated OLINGO-1504:
---------------------------------
Attachment: 0001-OLINGO-1504-override-getRawResponse-method-in-ODataE.patch
> JVM crashes due to OutOfMemory encountered: Java heap space
> ------------------------------------------------------------
>
> Key: OLINGO-1504
> URL: https://issues.apache.org/jira/browse/OLINGO-1504
> Project: Olingo
> Issue Type: Bug
> Components: odata4-client
> Affects Versions: (Java) V4 4.7.0
> Reporter: Devansh Soni
> Priority: Major
> Attachments:
> 0001-OLINGO-1504-override-getRawResponse-method-in-ODataE.patch,
> HeapDumpLargestObjects.png, JprofilerHeapWalkerGraph.png
>
>
> Hi
> The issue occurs for non-paginated OData feeds. The feed I had tested had
> 100,000 rows and 9 columns. The JVM crashes due to insufficient heap size and
> I can find the stack trace from the hs_err_pidPID log file.
> {code:java}
> ID Value
> 26 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> 27 j java.util.Arrays.copyOf([BI)[B+1
> 28 j java.io.ByteArrayOutputStream.grow(I)V+36
> 29 j java.io.ByteArrayOutputStream.ensureCapacity(I)V+12
> 30 j java.io.ByteArrayOutputStream.write([BII)V+38
> 31 j
> org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;[B)J+19
> 32 j
> org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;I)J+5
> 33 j
> org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J+5
> 34 j
> org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I+2
> 35 j
> org.apache.olingo.client.core.communication.response.AbstractODataResponse.getRawResponse()Ljava/io/InputStream;+136
> 36 j
> org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Lorg/apache/olingo/client/api/domain/ClientEntitySetIterator;+23
> 37 j
> org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Ljava/lang/Object;+1
> 38 j
> com.tableausoftware.odata.ODataProtocolImpl.fetchV4(Ljava/net/URI;Z)Lcom/tableausoftware/odata/ODataProtocolImpl$ODataResults;+50
> 39 j
> com.tableausoftware.odata.ODataResultSetV4.nextBlockImpl()Lcom/tableausoftware/data/generated/DataStream$Block;+23
> 40 j
> com.tableausoftware.data.ProtobufResultSet.nextBlock()Lcom/tableausoftware/data/generated/DataStream$Block;+1
> 41 j com.tableau.connect.service.QueryTask.readData()V+46
> 42 j com.tableau.connect.service.QueryTask.call()Ljava/lang/Void;+9
> 43 j com.tableau.connect.service.QueryTask.call()Ljava/lang/Object;+1
> 44 j java.util.concurrent.FutureTask.run()V+42
> 45 j
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
> 46 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
> 47 j java.lang.Thread.run()V+11
> 48 v ~StubRoutines::call_stub
> {code}
>
> The issue happens because there are multiple copies of streams being created
> in the
> AbstractODataResponse.[getRawResponse|https://github.com/apache/olingo-odata4/blob/master/lib/client-core/src/main/java/org/apache/olingo/client/core/communication/response/AbstractODataResponse.java#L300]
> method, specifically in the org.apache.commons.io.IOUtils.copy method. To me
> it seems like it fails to expand the internal byte buffer when it reaches
> capacity.
> However, I do not understand why is the response payload being copied in a
> ByteArrayOutputStream
> {noformat}
> org.apache.commons.io.IOUtils.copy(payload, byteArrayOutputStream);
> {noformat}
> and then again converted into a ByteArrayInputStream. This copying of streams
> causes creation of multiple byte buffers which fills up the heap memory.
> The
> ODataEntitySetIteratorResponseImpl.[getBody|https://github.com/apache/olingo-odata4/blob/master/lib/client-core/src/main/java/org/apache/olingo/client/core/communication/request/retrieve/ODataEntitySetIteratorRequestImpl.java#L78]
> call the getRawResponse() in the constructor call of
> {noformat}
> entitySetIterator = new ClientEntitySetIterator<>(
> odataClient, getRawResponse(),
> ContentType.parse(getContentType()));
> }
> {noformat}
>
> However the
> [constructor|https://github.com/apache/olingo-odata4/blob/master/lib/client-api/src/main/java/org/apache/olingo/client/api/domain/ClientEntitySetIterator.java#L79]
> of ClientEntitySetIterator accepts InputStream.
> So I do not understand the reason behind conversion of the payload into
> ByteArrayInputStream in the AbstractODataResponse. I am trying to figure the
> reason why this was done versus returning the payload InputStream as-is.
> For fixing the problem, we have tried increasing the Java heap size but it is
> just a temporary solution since once the OData feed size increases further
> beyond a limit, it will fail again.
> I also got heap dump for the Java crash and was able to visualize the largest
> object byte[] in a jprofiler to reach the same conclusion as above.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
