[
https://issues.apache.org/jira/browse/PARQUET-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinli Shang updated PARQUET-1533:
---------------------------------
Description:
Parquet-1485 initialize the buffer size(inputBuffer and outputBuffer) from 0 to
128M in total. This cause the unit test TestSnappy() failed with OOM exception.
This is on my Mac laptop.
To solve the unit test failure, we can increase the size of -Xmx from 512m to
1024m like below. However, we need to evaluate whether or not the increase of
the initial size of direct memory usage for inputBuffer and outputBuffer will
cause real Parquet application OOM or not if that application is not with big
enough -Xmx size.
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
...
<argLine>-Xmx1014m</argLine>
...
For details of the exception, the pull
request([https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b|https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b0])
has the detail.
was:
Parquet-1485 initialize the buffer size(inputBuffer and outputBuffer) from 0 to
128M in total. This cause the unit test TestSnappy() failed with OOM exception.
This is on my Mac laptop.
To solve the unit test failure, we can increase the size of -Xmx from 512m to
1012m like below. However, we need to evaluate whether or not the increase of
the initial size of direct memory usage for inputBuffer and outputBuffer will
cause real Parquet application OOM or not if that application is not with big
enough -Xmx size.
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
...
<argLine>-Xmx1014m</argLine>
...
For details of the exception, the pull
request([https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b|https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b0])
has the detail.
> TestSnappy() throws OOM exception with Parquet-1485 change
> -----------------------------------------------------------
>
> Key: PARQUET-1533
> URL: https://issues.apache.org/jira/browse/PARQUET-1533
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.11.0
> Environment: Mac OS 10.14.1
> Reporter: Xinli Shang
> Priority: Minor
>
> Parquet-1485 initialize the buffer size(inputBuffer and outputBuffer) from 0
> to 128M in total. This cause the unit test TestSnappy() failed with OOM
> exception. This is on my Mac laptop.
> To solve the unit test failure, we can increase the size of -Xmx from 512m to
> 1024m like below. However, we need to evaluate whether or not the increase of
> the initial size of direct memory usage for inputBuffer and outputBuffer will
> cause real Parquet application OOM or not if that application is not with big
> enough -Xmx size.
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> ...
> <argLine>-Xmx1014m</argLine>
> ...
> For details of the exception, the pull
> request([https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b|https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b0])
> has the detail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
