[jira] [Commented] (HDFS-14308) DFSStripedInputStream curStripeBuf is not freed by unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936040#comment-16936040
]
Joe McDonnell commented on HDFS-14308:
--
[~zhaoyim] When we found this issue, it was easy to reproduce in Apache Impala,
but that is a very complicated development environment. It would be better to
reproduce it directly. I haven't reproduced this in a while, but I think the
basic steps for the scenario are:
# Open an erasure coded file (let's just assume it is 128MB or some decent
size)
# Read several MB in the middle (let's just assume that this is a Parquet file
and you read one column or it is a text file and you read some chunk in the
middle (without reaching the end of the file)). The contents shouldn't matter.
This should use some direct buffer memory as temporary memory.
# Call unbuffer(), which is supposed to free unneeded memory
After #3, if unbuffer() was correct, this would not be using direct buffer
memory. The important thing is that the file remains open. Calling close would
definitely free the memory.
You could run this in the debugger and test memory usage after #3, or you could
do this in a loop for 20,000 files, storing the file handles in an array (they
should not be garbage collected), and see if you run out of memory (you can
open one file 20,000 times, and it should behave the same as having 20,000
distinct files open).
I hope this helps.
> DFSStripedInputStream curStripeBuf is not freed by unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Assignee: Zhao Yi Ming
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects. Both Impala and HBase call unbuffer() when a
> file handle is being cached and potentially unused for significant chunks of
> time, yet this shows that the memory remains in use.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should avoid holding buffers after the unbuffer() call. See HDFS-7694.
> "unbuffer()" is intended to move an input stream to a lower memory state to
> support these caching use cases. In particular, the curStripeBuf seems to be
> allocated from the BUFFER_POOL on a resetCurStripeBuffer(true) call. It is
> not freed until close().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14308) DFSStripedInputStream curStripeBuf is not freed by unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated HDFS-14308:
-
Description:
Some users of HDFS cache opened HDFS file handles to avoid repeated roundtrips
to the NameNode. For example, Impala caches up to 20,000 HDFS file handles by
default. Recent tests on erasure coded files show that the open file handles
can consume a large amount of memory when not in use.
For example, here is output from Impala's JMX endpoint when 608 file handles
are cached
{noformat}
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"TotalCapacity": 1921048960,
"MemoryUsed": 1921048961,
"Count": 633,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},{noformat}
This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
Attached is output from Eclipse MAT showing that the direct buffers come from
DFSStripedInputStream objects. Both Impala and HBase call unbuffer() when a
file handle is being cached and potentially unused for significant chunks of
time, yet this shows that the memory remains in use.
To support caching file handles on erasure coded files, DFSStripedInputStream
should avoid holding buffers after the unbuffer() call. See HDFS-7694.
"unbuffer()" is intended to move an input stream to a lower memory state to
support these caching use cases. In particular, the curStripeBuf seems to be
allocated from the BUFFER_POOL on a resetCurStripeBuffer(true) call. It is not
freed until close().
was:
Some users of HDFS cache opened HDFS file handles to avoid repeated roundtrips
to the NameNode. For example, Impala caches up to 20,000 HDFS file handles by
default. Recent tests on erasure coded files show that the open file handles
can consume a large amount of memory when not in use.
For example, here is output from Impala's JMX endpoint when 608 file handles
are cached
{noformat}
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"TotalCapacity": 1921048960,
"MemoryUsed": 1921048961,
"Count": 633,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},{noformat}
This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
Attached is output from Eclipse MAT showing that the direct buffers come from
DFSStripedInputStream objects.
To support caching file handles on erasure coded files, DFSStripedInputStream
should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
to move an input stream to a lower memory state to support these caching use
cases. Both Impala and HBase call unbuffer() when a file handle is being cached
and potentially unused for significant chunks of time.
> DFSStripedInputStream curStripeBuf is not freed by unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects. Both Impala and HBase call unbuffer() when a
> file handle is being cached and potentially unused for significant chunks of
> time, yet this shows that the memory remains in use.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should avoid holding buffers after the unbuffer() call. See HDFS-7694.
> "unbuffer()" is intended to move an input stream to a lower memory state to
> support these caching use cases. In particular, the curStripeBuf seems to be
> allocated from the BUFFER_POOL on a resetCurStripeBuffer(true) call. It is
> not freed until close().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Commented] (HDFS-14308) DFSStripedInputStream curStripeBuf is not freed by unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778626#comment-16778626
]
Joe McDonnell commented on HDFS-14308:
--
Updated the description / title.
> DFSStripedInputStream curStripeBuf is not freed by unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects. Both Impala and HBase call unbuffer() when a
> file handle is being cached and potentially unused for significant chunks of
> time, yet this shows that the memory remains in use.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should avoid holding buffers after the unbuffer() call. See HDFS-7694.
> "unbuffer()" is intended to move an input stream to a lower memory state to
> support these caching use cases. In particular, the curStripeBuf seems to be
> allocated from the BUFFER_POOL on a resetCurStripeBuffer(true) call. It is
> not freed until close().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14308) DFSStripedInputStream curStripeBuf is not freed by unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated HDFS-14308:
-
Summary: DFSStripedInputStream curStripeBuf is not freed by unbuffer()
(was: DFSStripedInputStream should implement unbuffer())
> DFSStripedInputStream curStripeBuf is not freed by unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
> to move an input stream to a lower memory state to support these caching use
> cases. Both Impala and HBase call unbuffer() when a file handle is being
> cached and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14308) DFSStripedInputStream should implement unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778595#comment-16778595
]
Joe McDonnell commented on HDFS-14308:
--
[~knanasi] Yes, good point, DFSStripedInputStream does implement unbuffer(). I
think the issue revolves around curStripeBuf. It is allocated from the
BUFFER_POOL in resetCurStripeBuffer(true), but it doesn't get returned to the
BUFFER_POOL until close(). closeCurrentBlockReaders() calls
resetCurStripeBuffer(false), which clears curStripeBuf but does not return it
to the BUFFER_POOL.
I will update the description / title.
> DFSStripedInputStream should implement unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
> to move an input stream to a lower memory state to support these caching use
> cases. Both Impala and HBase call unbuffer() when a file handle is being
> cached and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14308) DFSStripedInputStream should implement unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated HDFS-14308:
-
Attachment: ec_heap_dump.png
> DFSStripedInputStream should implement unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
> to move an input stream to a lower memory state to support these caching use
> cases. Both Impala and HBase call unbuffer() when a file handle is being
> cached and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14308) DFSStripedInputStream should implement unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774576#comment-16774576
]
Joe McDonnell commented on HDFS-14308:
--
Adding a link to the original IMPALA issue
> DFSStripedInputStream should implement unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
> to move an input stream to a lower memory state to support these caching use
> cases. Both Impala and HBase call unbuffer() when a file handle is being
> cached and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14308) DFSStripedInputStream should implement unbuffer()
Joe McDonnell created HDFS-14308:
Summary: DFSStripedInputStream should implement unbuffer()
Key: HDFS-14308
URL: https://issues.apache.org/jira/browse/HDFS-14308
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Joe McDonnell
Attachments: ec_heap_dump.png
Some users of HDFS cache opened HDFS file handles to avoid repeated roundtrips
to the NameNode. For example, Impala caches up to 20,000 HDFS file handles by
default. Recent tests on erasure coded files show that the open file handles
can consume a large amount of memory when not in use.
For example, here is output from Impala's JMX endpoint when 608 file handles
are cached
{noformat}
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"TotalCapacity": 1921048960,
"MemoryUsed": 1921048961,
"Count": 633,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},{noformat}
This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
Attached is output from Eclipse MAT showing that the direct buffers come from
DFSStripedInputStream objects.
To support caching file handles on erasure coded files, DFSStripedInputStream
should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
to move an input stream to a lower memory state to support these caching use
cases. Both Impala and HBase call unbuffer() when a file handle is being cached
and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12714) Hadoop 3 missing fix for HDFS-5169
[ https://issues.apache.org/jira/browse/HDFS-12714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233549#comment-16233549 ] Joe McDonnell commented on HDFS-12714: -- [~jzhuge] The test case that spurred the creation of this JIRA is an Impala test case for Impala's fallback code when a zero copy HDFS read fails. It does this by mixing zero copy reads with encryption zones, which is a known error case (see IMPALA-3679). That error causes libhdfs to call translateZCRException, which without this patch was failing and crashing Impala. I tested this patch by building libhdfs and running the same test case. It succeeds where it was previously crashing. > Hadoop 3 missing fix for HDFS-5169 > -- > > Key: HDFS-12714 > URL: https://issues.apache.org/jira/browse/HDFS-12714 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2, 3.0.0-alpha4, > 3.0.0-alpha3 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Major > Attachments: HDFS-12714.001.patch > > > HDFS-5169 is a fix for a null pointer dereference in translateZCRException. > This line in hdfs.c: > ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > should be: > ret = printExceptionAndFree(env, exc, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > Plainly, translateZCRException should print the exception (exc) passed in to > the function rather than the uninitialized local jthr. > The fix for HDFS-5169 (part of HDFS-4949) exists on hadoop 2.* branches, but > it is missing on hadoop 3 branches including trunk. > Hadoop 2.8: > https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2514 > Hadoop 3.0: > https://github.com/apache/hadoop/blob/branch-3.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2691 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-12714) Hadoop 3 missing fix for HDFS-5169
[ https://issues.apache.org/jira/browse/HDFS-12714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-12714 started by Joe McDonnell. > Hadoop 3 missing fix for HDFS-5169 > -- > > Key: HDFS-12714 > URL: https://issues.apache.org/jira/browse/HDFS-12714 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2, 3.0.0-alpha4, > 3.0.0-alpha3 >Reporter: Joe McDonnell >Assignee: Joe McDonnell > Attachments: HDFS-12714.001.patch > > > HDFS-5169 is a fix for a null pointer dereference in translateZCRException. > This line in hdfs.c: > ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > should be: > ret = printExceptionAndFree(env, exc, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > Plainly, translateZCRException should print the exception (exc) passed in to > the function rather than the uninitialized local jthr. > The fix for HDFS-5169 (part of HDFS-4949) exists on hadoop 2.* branches, but > it is missing on hadoop 3 branches including trunk. > Hadoop 2.8: > https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2514 > Hadoop 3.0: > https://github.com/apache/hadoop/blob/branch-3.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2691 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12714) Hadoop 3 missing fix for HDFS-5169
[ https://issues.apache.org/jira/browse/HDFS-12714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222774#comment-16222774 ] Joe McDonnell commented on HDFS-12714: -- Uploaded a patch for review. [~jzhuge] FYI > Hadoop 3 missing fix for HDFS-5169 > -- > > Key: HDFS-12714 > URL: https://issues.apache.org/jira/browse/HDFS-12714 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2, 3.0.0-alpha4, > 3.0.0-alpha3 >Reporter: Joe McDonnell >Assignee: Joe McDonnell > Attachments: HDFS-12714.001.patch > > > HDFS-5169 is a fix for a null pointer dereference in translateZCRException. > This line in hdfs.c: > ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > should be: > ret = printExceptionAndFree(env, exc, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > Plainly, translateZCRException should print the exception (exc) passed in to > the function rather than the uninitialized local jthr. > The fix for HDFS-5169 (part of HDFS-4949) exists on hadoop 2.* branches, but > it is missing on hadoop 3 branches including trunk. > Hadoop 2.8: > https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2514 > Hadoop 3.0: > https://github.com/apache/hadoop/blob/branch-3.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2691 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12714) Hadoop 3 missing fix for HDFS-5169
[ https://issues.apache.org/jira/browse/HDFS-12714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell updated HDFS-12714: - Attachment: HDFS-12714.001.patch > Hadoop 3 missing fix for HDFS-5169 > -- > > Key: HDFS-12714 > URL: https://issues.apache.org/jira/browse/HDFS-12714 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2, 3.0.0-alpha4, > 3.0.0-alpha3 >Reporter: Joe McDonnell >Assignee: Joe McDonnell > Attachments: HDFS-12714.001.patch > > > HDFS-5169 is a fix for a null pointer dereference in translateZCRException. > This line in hdfs.c: > ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > should be: > ret = printExceptionAndFree(env, exc, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > Plainly, translateZCRException should print the exception (exc) passed in to > the function rather than the uninitialized local jthr. > The fix for HDFS-5169 (part of HDFS-4949) exists on hadoop 2.* branches, but > it is missing on hadoop 3 branches including trunk. > Hadoop 2.8: > https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2514 > Hadoop 3.0: > https://github.com/apache/hadoop/blob/branch-3.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2691 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12714) Hadoop 3 missing fix for HDFS-5169
[ https://issues.apache.org/jira/browse/HDFS-12714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell reassigned HDFS-12714: Assignee: Joe McDonnell > Hadoop 3 missing fix for HDFS-5169 > -- > > Key: HDFS-12714 > URL: https://issues.apache.org/jira/browse/HDFS-12714 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2, 3.0.0-alpha4, > 3.0.0-alpha3 >Reporter: Joe McDonnell >Assignee: Joe McDonnell > > HDFS-5169 is a fix for a null pointer dereference in translateZCRException. > This line in hdfs.c: > ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > should be: > ret = printExceptionAndFree(env, exc, PRINT_EXC_ALL, "hadoopZeroCopyRead: > ZeroCopyCursor#read failed"); > Plainly, translateZCRException should print the exception (exc) passed in to > the function rather than the uninitialized local jthr. > The fix for HDFS-5169 (part of HDFS-4949) exists on hadoop 2.* branches, but > it is missing on hadoop 3 branches including trunk. > Hadoop 2.8: > https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2514 > Hadoop 3.0: > https://github.com/apache/hadoop/blob/branch-3.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2691 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12714) Hadoop 3 missing fix for HDFS-5169
Joe McDonnell created HDFS-12714: Summary: Hadoop 3 missing fix for HDFS-5169 Key: HDFS-12714 URL: https://issues.apache.org/jira/browse/HDFS-12714 Project: Hadoop HDFS Issue Type: Bug Components: native Affects Versions: 3.0.0-alpha3, 3.0.0-alpha4, 3.0.0-alpha2, 3.0.0-beta1, 3.0.0-alpha1 Reporter: Joe McDonnell HDFS-5169 is a fix for a null pointer dereference in translateZCRException. This line in hdfs.c: ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "hadoopZeroCopyRead: ZeroCopyCursor#read failed"); should be: ret = printExceptionAndFree(env, exc, PRINT_EXC_ALL, "hadoopZeroCopyRead: ZeroCopyCursor#read failed"); Plainly, translateZCRException should print the exception (exc) passed in to the function rather than the uninitialized local jthr. The fix for HDFS-5169 (part of HDFS-4949) exists on hadoop 2.* branches, but it is missing on hadoop 3 branches including trunk. Hadoop 2.8: https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2514 Hadoop 3.0: https://github.com/apache/hadoop/blob/branch-3.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L2691 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12628) libhdfs crashes on thread exit for JNI+libhdfs applications
[ https://issues.apache.org/jira/browse/HDFS-12628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell updated HDFS-12628: - Attachment: jni-util-test2.cc > libhdfs crashes on thread exit for JNI+libhdfs applications > --- > > Key: HDFS-12628 > URL: https://issues.apache.org/jira/browse/HDFS-12628 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.0.0-alpha3 >Reporter: Joe McDonnell >Priority: Critical > Attachments: jni-util-test2.cc > > > Impala uses libhdfs to access HDFS while also using JNI to run other Java > code. Impala currently relies on HDFS's getJNIEnv to get a JNIEnv to interact > with the process JVM (which is created by HDFS code). It uses this JNIEnv > even for code that is not related to HDFS. > In recent versions of HDFS, getJNIEnv is no longer visible in libhdfs due to > HDFS-7879. In HDFS-8474, the proposed solution was for Impala to write its > own equivalent (tracked by IMPALA-2029). After implementing an equivalent of > getJNIEnv (heavily based on HDFS code, but with distinct names), we are > seeing crashes in hdfsThreadDestructor() in threads that use both HDFS and > JNI codepaths. The crash shows up under concurrency and does not reproduce in > serial execution. > I have distilled it down to a simple testcase that reproduces the issue. It > creates a JVM in the main thread (which Impala does at startup), then spawns > multiple threads that do basic HDFS and JNI work. I have removed all but the > essential steps. > This blocks running Impala on any hadoop version past 2.7 (when HDFS-7879 was > merged). Note that exposing getJNIEnv should unblock Impala development if a > fix is not forthcoming. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12628) libhdfs crashes on thread exit for JNI+libhdfs applications
Joe McDonnell created HDFS-12628: Summary: libhdfs crashes on thread exit for JNI+libhdfs applications Key: HDFS-12628 URL: https://issues.apache.org/jira/browse/HDFS-12628 Project: Hadoop HDFS Issue Type: Bug Components: native Affects Versions: 3.0.0-alpha3 Reporter: Joe McDonnell Priority: Critical Impala uses libhdfs to access HDFS while also using JNI to run other Java code. Impala currently relies on HDFS's getJNIEnv to get a JNIEnv to interact with the process JVM (which is created by HDFS code). It uses this JNIEnv even for code that is not related to HDFS. In recent versions of HDFS, getJNIEnv is no longer visible in libhdfs due to HDFS-7879. In HDFS-8474, the proposed solution was for Impala to write its own equivalent (tracked by IMPALA-2029). After implementing an equivalent of getJNIEnv (heavily based on HDFS code, but with distinct names), we are seeing crashes in hdfsThreadDestructor() in threads that use both HDFS and JNI codepaths. The crash shows up under concurrency and does not reproduce in serial execution. I have distilled it down to a simple testcase that reproduces the issue. It creates a JVM in the main thread (which Impala does at startup), then spawns multiple threads that do basic HDFS and JNI work. I have removed all but the essential steps. This blocks running Impala on any hadoop version past 2.7 (when HDFS-7879 was merged). Note that exposing getJNIEnv should unblock Impala development if a fix is not forthcoming. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
