[jira] [Updated] (HDFS-5433) When reloading fsimage during checkpointing, we should clear existing snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-5433: - Attachment: HDFS-5433.patch Here's a patch which addresses the issue by clearing the map of snapshottable directories when we reload the fsimage from disk during a checkpoint. > When reloading fsimage during checkpointing, we should clear existing > snapshottable directories > --- > > Key: HDFS-5433 > URL: https://issues.apache.org/jira/browse/HDFS-5433 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.2.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Critical > Attachments: HDFS-5433.patch > > > The complete set of snapshottable directories are referenced both via the > file system tree and in the SnapshotManager class. It's possible that when > the 2NN performs a checkpoint, it will reload its in-memory state based on a > new fsimage from the NN, but will not clear the set of snapshottable > directories referenced by the SnapshotManager. In this case, the 2NN will > write out an fsimage that cannot be loaded, since the integer written to the > fsimage indicating the number of snapshottable directories will be out of > sync with the actual number of snapshottable directories serialized to the > fsimage. > This is basically the same as HDFS-3835, but for snapshottable directories > instead of delegation tokens. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5433) When reloading fsimage during checkpointing, we should clear existing snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-5433: - Status: Patch Available (was: Open) > When reloading fsimage during checkpointing, we should clear existing > snapshottable directories > --- > > Key: HDFS-5433 > URL: https://issues.apache.org/jira/browse/HDFS-5433 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.2.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Critical > Attachments: HDFS-5433.patch > > > The complete set of snapshottable directories are referenced both via the > file system tree and in the SnapshotManager class. It's possible that when > the 2NN performs a checkpoint, it will reload its in-memory state based on a > new fsimage from the NN, but will not clear the set of snapshottable > directories referenced by the SnapshotManager. In this case, the 2NN will > write out an fsimage that cannot be loaded, since the integer written to the > fsimage indicating the number of snapshottable directories will be out of > sync with the actual number of snapshottable directories serialized to the > fsimage. > This is basically the same as HDFS-3835, but for snapshottable directories > instead of delegation tokens. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5433) When reloading fsimage during checkpointing, we should clear existing snapshottable directories
Aaron T. Myers created HDFS-5433: Summary: When reloading fsimage during checkpointing, we should clear existing snapshottable directories Key: HDFS-5433 URL: https://issues.apache.org/jira/browse/HDFS-5433 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 2.2.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Priority: Critical The complete set of snapshottable directories are referenced both via the file system tree and in the SnapshotManager class. It's possible that when the 2NN performs a checkpoint, it will reload its in-memory state based on a new fsimage from the NN, but will not clear the set of snapshottable directories referenced by the SnapshotManager. In this case, the 2NN will write out an fsimage that cannot be loaded, since the integer written to the fsimage indicating the number of snapshottable directories will be out of sync with the actual number of snapshottable directories serialized to the fsimage. This is basically the same as HDFS-3835, but for snapshottable directories instead of delegation tokens. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5432) TestDatanodeJsp fails on Windows due to assumption that loopback address resolves to host name localhost.
[
https://issues.apache.org/jira/browse/HDFS-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5432:
Status: Patch Available (was: Open)
> TestDatanodeJsp fails on Windows due to assumption that loopback address
> resolves to host name localhost.
> -
>
> Key: HDFS-5432
> URL: https://issues.apache.org/jira/browse/HDFS-5432
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Trivial
> Attachments: HDFS-5432.1.patch
>
>
> As discussed in many previous issues, Windows differs from Unixes in that it
> does not resolve the loopback address to hostname "localhost". Instead, the
> host name remains unresolved as "127.0.0.1". {{TestDatanodeJsp}} fails on
> Windows, because it attempts to assert a string match containing "localhost"
> as the host name.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5432) TestDatanodeJsp fails on Windows due to assumption that loopback address resolves to host name localhost.
[
https://issues.apache.org/jira/browse/HDFS-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5432:
Attachment: HDFS-5432.1.patch
This patch changes the assertion so that it checks for "127.0.0.1" on Windows
or "localhost" on other platforms.
> TestDatanodeJsp fails on Windows due to assumption that loopback address
> resolves to host name localhost.
> -
>
> Key: HDFS-5432
> URL: https://issues.apache.org/jira/browse/HDFS-5432
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Trivial
> Attachments: HDFS-5432.1.patch
>
>
> As discussed in many previous issues, Windows differs from Unixes in that it
> does not resolve the loopback address to hostname "localhost". Instead, the
> host name remains unresolved as "127.0.0.1". {{TestDatanodeJsp}} fails on
> Windows, because it attempts to assert a string match containing "localhost"
> as the host name.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5432) TestDatanodeJsp fails on Windows due to assumption that loopback address resolves to host name localhost.
Chris Nauroth created HDFS-5432:
---
Summary: TestDatanodeJsp fails on Windows due to assumption that
loopback address resolves to host name localhost.
Key: HDFS-5432
URL: https://issues.apache.org/jira/browse/HDFS-5432
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, test
Affects Versions: 2.2.0, 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Trivial
As discussed in many previous issues, Windows differs from Unixes in that it
does not resolve the loopback address to hostname "localhost". Instead, the
host name remains unresolved as "127.0.0.1". {{TestDatanodeJsp}} fails on
Windows, because it attempts to assert a string match containing "localhost" as
the host name.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-3987) Support webhdfs over HTTPS
[
https://issues.apache.org/jira/browse/HDFS-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805976#comment-13805976
]
Hadoop QA commented on HDFS-3987:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610420/HDFS-3987.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestHftpDelegationToken
org.apache.hadoop.hdfs.web.TestWebHdfsUrl
org.apache.hadoop.hdfs.TestHftpURLTimeouts
org.apache.hadoop.hdfs.TestHftpFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5287//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5287//console
This message is automatically generated.
> Support webhdfs over HTTPS
> --
>
> Key: HDFS-3987
> URL: https://issues.apache.org/jira/browse/HDFS-3987
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.2-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Haohui Mai
> Fix For: 2.3.0
>
> Attachments: HDFS-3987.000.patch, HDFS-3987.001.patch,
> HDFS-3987.002.patch, HDFS-3987.003.patch
>
>
> This is a follow up of HDFS-3983.
> We should have a new filesystem client impl/binding for encrypted WebHDFS,
> i.e. *webhdfss://*
> On the server side, webhdfs and httpfs we should only need to start the
> service on a secured (HTTPS) endpoint.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5393) Serve bootstrap and jQuery locally
[
https://issues.apache.org/jira/browse/HDFS-5393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805967#comment-13805967
]
Hadoop QA commented on HDFS-5393:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610416/HDFS-5393.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5286//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5286//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5286//console
This message is automatically generated.
> Serve bootstrap and jQuery locally
> --
>
> Key: HDFS-5393
> URL: https://issues.apache.org/jira/browse/HDFS-5393
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-5393.000.patch, HDFS-5393.001.patch,
> HDFS-5393-static.tar.bz2
>
>
> Currently the webui depends upon bootstrap and jQuery on the CDN. These
> libraries should be served locally so that the web ui can work when the
> cluster does not connect to the Internet.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5368) Namenode deadlock during safemode extention
[ https://issues.apache.org/jira/browse/HDFS-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5368: Resolution: Duplicate Status: Resolved (was: Patch Available) > Namenode deadlock during safemode extention > --- > > Key: HDFS-5368 > URL: https://issues.apache.org/jira/browse/HDFS-5368 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Vinay >Assignee: Vinay >Priority: Blocker > Attachments: HDFS-5368.patch, NN-deadlock.zip > > > Namenode entered to safemode during restart > 1. After restart NN entered to safemode extention. > 2. During this time deadlock happened between datanode heartbeat and > SafemodeMonitor() thread. > Found one Java-level deadlock: > = > "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeMonitor@9fe953": > waiting to lock monitor 0x18c3b42c (object 0x0439c6f8, a java.util.TreeMap), > which is held by "IPC Server handler 2 on 62212" > "IPC Server handler 2 on 62212": > waiting to lock monitor 0x18c3987c (object 0x043849a0, a > org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo), > which is held by > "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeMonitor@9fe953" > Check attached jstack for complete stack -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5368) Namenode deadlock during safemode extention
[ https://issues.apache.org/jira/browse/HDFS-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805931#comment-13805931 ] Vinay commented on HDFS-5368: - HDFS-5132 already fixed this issue. Closing this as duplicate > Namenode deadlock during safemode extention > --- > > Key: HDFS-5368 > URL: https://issues.apache.org/jira/browse/HDFS-5368 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Vinay >Assignee: Vinay >Priority: Blocker > Attachments: HDFS-5368.patch, NN-deadlock.zip > > > Namenode entered to safemode during restart > 1. After restart NN entered to safemode extention. > 2. During this time deadlock happened between datanode heartbeat and > SafemodeMonitor() thread. > Found one Java-level deadlock: > = > "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeMonitor@9fe953": > waiting to lock monitor 0x18c3b42c (object 0x0439c6f8, a java.util.TreeMap), > which is held by "IPC Server handler 2 on 62212" > "IPC Server handler 2 on 62212": > waiting to lock monitor 0x18c3987c (object 0x043849a0, a > org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo), > which is held by > "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeMonitor@9fe953" > Check attached jstack for complete stack -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5364) Add OpenFileCtx cache
[
https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805879#comment-13805879
]
Hadoop QA commented on HDFS-5364:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610417/HDFS-5364.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5285//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5285//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5285//console
This message is automatically generated.
> Add OpenFileCtx cache
> -
>
> Key: HDFS-5364
> URL: https://issues.apache.org/jira/browse/HDFS-5364
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch,
> HDFS-5364.003.patch
>
>
> NFS gateway can run out of memory when the stream timeout is set to a
> relatively long period(e.g., >1 minute) and user uploads thousands of files
> in parallel. Each stream DFSClient creates a DataStreamer thread, and will
> eventually run out of memory by creating too many threads.
> NFS gateway should have a OpenFileCtx cache to limit the total opened files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5429) Deprecate FsDatasetSpi#getBlockReport
[
https://issues.apache.org/jira/browse/HDFS-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805868#comment-13805868
]
Tsz Wo (Nicholas), SZE commented on HDFS-5429:
--
We may just remove it since FsDatasetSpi is only a private API.
> Deprecate FsDatasetSpi#getBlockReport
> -
>
> Key: HDFS-5429
> URL: https://issues.apache.org/jira/browse/HDFS-5429
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: Heterogeneous Storage (HDFS-2832)
> Environment: {{FsDatasetSpi#getBlockReport}} has been superseded by
> {{FsDatasetSpi#getBlockReports}} which gets one block report per storage. The
> old routine is only used in tests right now, deprecate it once we fix the
> tests.
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-3987) Support webhdfs over HTTPS
[ https://issues.apache.org/jira/browse/HDFS-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-3987: - Attachment: HDFS-3987.003.patch > Support webhdfs over HTTPS > -- > > Key: HDFS-3987 > URL: https://issues.apache.org/jira/browse/HDFS-3987 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Assignee: Haohui Mai > Fix For: 2.3.0 > > Attachments: HDFS-3987.000.patch, HDFS-3987.001.patch, > HDFS-3987.002.patch, HDFS-3987.003.patch > > > This is a follow up of HDFS-3983. > We should have a new filesystem client impl/binding for encrypted WebHDFS, > i.e. *webhdfss://* > On the server side, webhdfs and httpfs we should only need to start the > service on a secured (HTTPS) endpoint. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5364: - Attachment: HDFS-5364.003.patch > Add OpenFileCtx cache > - > > Key: HDFS-5364 > URL: https://issues.apache.org/jira/browse/HDFS-5364 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch, > HDFS-5364.003.patch > > > NFS gateway can run out of memory when the stream timeout is set to a > relatively long period(e.g., >1 minute) and user uploads thousands of files > in parallel. Each stream DFSClient creates a DataStreamer thread, and will > eventually run out of memory by creating too many threads. > NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5393) Serve bootstrap and jQuery locally
[ https://issues.apache.org/jira/browse/HDFS-5393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5393: - Attachment: HDFS-5393.001.patch > Serve bootstrap and jQuery locally > -- > > Key: HDFS-5393 > URL: https://issues.apache.org/jira/browse/HDFS-5393 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5393.000.patch, HDFS-5393.001.patch, > HDFS-5393-static.tar.bz2 > > > Currently the webui depends upon bootstrap and jQuery on the CDN. These > libraries should be served locally so that the web ui can work when the > cluster does not connect to the Internet. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5393) Serve bootstrap and jQuery locally
[
https://issues.apache.org/jira/browse/HDFS-5393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805855#comment-13805855
]
Hadoop QA commented on HDFS-5393:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610412/HDFS-5393-static.tar.bz2
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5284//console
This message is automatically generated.
> Serve bootstrap and jQuery locally
> --
>
> Key: HDFS-5393
> URL: https://issues.apache.org/jira/browse/HDFS-5393
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-5393.000.patch, HDFS-5393-static.tar.bz2
>
>
> Currently the webui depends upon bootstrap and jQuery on the CDN. These
> libraries should be served locally so that the web ui can work when the
> cluster does not connect to the Internet.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-3987) Support webhdfs over HTTPS
[
https://issues.apache.org/jira/browse/HDFS-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805845#comment-13805845
]
Hadoop QA commented on HDFS-3987:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610407/HDFS-3987.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5283//console
This message is automatically generated.
> Support webhdfs over HTTPS
> --
>
> Key: HDFS-3987
> URL: https://issues.apache.org/jira/browse/HDFS-3987
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.2-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Haohui Mai
> Fix For: 2.3.0
>
> Attachments: HDFS-3987.000.patch, HDFS-3987.001.patch,
> HDFS-3987.002.patch
>
>
> This is a follow up of HDFS-3983.
> We should have a new filesystem client impl/binding for encrypted WebHDFS,
> i.e. *webhdfss://*
> On the server side, webhdfs and httpfs we should only need to start the
> service on a secured (HTTPS) endpoint.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5393) Serve bootstrap and jQuery locally
[ https://issues.apache.org/jira/browse/HDFS-5393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5393: - Attachment: HDFS-5393-static.tar.bz2 > Serve bootstrap and jQuery locally > -- > > Key: HDFS-5393 > URL: https://issues.apache.org/jira/browse/HDFS-5393 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5393.000.patch, HDFS-5393-static.tar.bz2 > > > Currently the webui depends upon bootstrap and jQuery on the CDN. These > libraries should be served locally so that the web ui can work when the > cluster does not connect to the Internet. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5393) Serve bootstrap and jQuery locally
[ https://issues.apache.org/jira/browse/HDFS-5393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5393: - Attachment: (was: HDFS-5393-static.tar.bz2) > Serve bootstrap and jQuery locally > -- > > Key: HDFS-5393 > URL: https://issues.apache.org/jira/browse/HDFS-5393 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5393.000.patch, HDFS-5393-static.tar.bz2 > > > Currently the webui depends upon bootstrap and jQuery on the CDN. These > libraries should be served locally so that the web ui can work when the > cluster does not connect to the Internet. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5364: - Attachment: (was: HDFS-5364.003.patch) > Add OpenFileCtx cache > - > > Key: HDFS-5364 > URL: https://issues.apache.org/jira/browse/HDFS-5364 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch > > > NFS gateway can run out of memory when the stream timeout is set to a > relatively long period(e.g., >1 minute) and user uploads thousands of files > in parallel. Each stream DFSClient creates a DataStreamer thread, and will > eventually run out of memory by creating too many threads. > NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5364) Add OpenFileCtx cache
[ https://issues.apache.org/jira/browse/HDFS-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5364: - Attachment: HDFS-5364.003.patch > Add OpenFileCtx cache > - > > Key: HDFS-5364 > URL: https://issues.apache.org/jira/browse/HDFS-5364 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5364.001.patch, HDFS-5364.002.patch, > HDFS-5364.003.patch > > > NFS gateway can run out of memory when the stream timeout is set to a > relatively long period(e.g., >1 minute) and user uploads thousands of files > in parallel. Each stream DFSClient creates a DataStreamer thread, and will > eventually run out of memory by creating too many threads. > NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-3987) Support webhdfs over HTTPS
[ https://issues.apache.org/jira/browse/HDFS-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-3987: - Attachment: HDFS-3987.002.patch > Support webhdfs over HTTPS > -- > > Key: HDFS-3987 > URL: https://issues.apache.org/jira/browse/HDFS-3987 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Assignee: Haohui Mai > Fix For: 2.3.0 > > Attachments: HDFS-3987.000.patch, HDFS-3987.001.patch, > HDFS-3987.002.patch > > > This is a follow up of HDFS-3983. > We should have a new filesystem client impl/binding for encrypted WebHDFS, > i.e. *webhdfss://* > On the server side, webhdfs and httpfs we should only need to start the > service on a secured (HTTPS) endpoint. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4949) Centralized cache management in HDFS
[ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805791#comment-13805791 ] Colin Patrick McCabe commented on HDFS-4949: quick note: The reason why TestOfflineEditsViewer failed in jenkins is that the consolidated patch didn't change the binary edit log file used by that test. It succeeds on the branch. bq. incremental cache reports - HDFS-5092 This is listed as a maybe in the design doc-- it's something that we want to evaluate before doing quotas, metrics, and TTL are definitely post-merge, I think. > Centralized cache management in HDFS > > > Key: HDFS-4949 > URL: https://issues.apache.org/jira/browse/HDFS-4949 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.0.0, 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: caching-design-doc-2013-07-02.pdf, > caching-design-doc-2013-08-09.pdf, caching-design-doc-2013-10-24.pdf, > caching-testplan.pdf, HDFS-4949-consolidated.patch > > > HDFS currently has no support for managing or exposing in-memory caches at > datanodes. This makes it harder for higher level application frameworks like > Hive, Pig, and Impala to effectively use cluster memory, because they cannot > explicitly cache important datasets or place their tasks for memory locality. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5431) support cachepool-based quota management in path-based caching
Colin Patrick McCabe created HDFS-5431: -- Summary: support cachepool-based quota management in path-based caching Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5430) Support TTL on CacheBasedPathDirectives
Colin Patrick McCabe created HDFS-5430: -- Summary: Support TTL on CacheBasedPathDirectives Key: HDFS-5430 URL: https://issues.apache.org/jira/browse/HDFS-5430 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Colin Patrick McCabe Priority: Minor It would be nice if CacheBasedPathDirectives would support an expiration time, after which they would be automatically removed by the NameNode. This time would probably be in wall-block time for the convenience of system administrators. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5333) Improvement of current HDFS Web UI
[ https://issues.apache.org/jira/browse/HDFS-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805766#comment-13805766 ] Haohui Mai commented on HDFS-5333: -- bq. this client-side js only approach, which is less secure than a progressively enhanced hybrid approach used by YARN. The recent gmail XSS fiasco highlights the issue. I'm presenting an informal security analysis to compare the security of the old and the new web UIs. An attacker launches an XSS attack by injecting malicious code which are usually HTML or JavaScript fragments into the web page, so that the malicious code can have the same privileges of the web page. First, in the scope of XSS attacks, note that the threat models of launching XSS attacks on Internet sites Gmail/Linkedin and the one of the Hadoop UIs are different. They have fundamental different sets of external inputs that the attackers have control to. Internet sites have little control of these inputs. In the case of Gmail / Linkedin, an attack can send you a crafted e-mail, or put malicious description in his / her Linkedin profile. The sets of external inputs are *restricted* in Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The attacker has to launch a XSS attack by: * Compromise the jars so that the output of JMX / WebHDFS have the malicious code. * Replace the web UIs completely to include the malicious code. In either case *the attacker has to compromise the hadoop core or the namenode*. That means the new web UIs are at least as secure as the hadoop core, and the namenode machine. Second, I argue that using client-side templates are more secure than the current JSP-based server-side templates. To defend against XSS attacks, both techniques have to filter the external inputs at *every* possible execution paths. Several facts much be taken into plays when evaluating the security of both approaches in real-world environments: * The JavaScript libraries used in the new web UIs have survived in extremely large-scale production tests. jQuery is used by Google and Microsoft, bootstrap is used by Twitter, and dust.js is used by Linkedin. All libraries survived from hundreds of thousands of attack attempts on a daily basis. I agree that the libraries might still be imperfect, but there's no way that we can test the JSP web UIs to achieve the same level of assurances given the amount of resources the community has. * Client-side templates consolidate all filtering logic in one central place. Recall that the goal is to filter all external inputs at every execution paths, this is a much more systematic approach compared to the server-side templates we have today. It is difficult (if not impossible) to do it in a JSP/ASP/PHP application, since such filtering can be only achieved via ad-hoc approaches ([1] shows some empirical data). Also, HDFS-4901 recently describes a XSS vulnerability in browseDirectory.jsp. bq. You'd require proper SSL (not self signed) setup to avoid JS injection Commodity browsers enforce Same-Origin Policy to defend against code injections. It has nothing to do with what kinds of SSL certificates you hold. bq. I also have concerns that we commit these changes without matching unit tests The JavaScript code can be automatically tested. The same code can be run by node.js and the test can compared with pre-defined results. It is also possible to write an adapter to use Rhino to accomplish the same task. We can discuss how to integrate them into the maven test routines in a different thread. bq. Client side rendering completely breaks the workflows for ops who rely on text based terminal/emacs/vim browsers (no js support) to monitor component UI. links / elinks (http://elinks.or.cz/) are text-based web browsers that support JavaScript. bq. The priority/requirements for UI in core Hadoop should be security and correctness, which client side templating cannot address properly so far. I agree that we should focus on security and correctness. The paragraphs above explain that how the architecture of the new UIs makes the UIs more secure in real-world settings compared to the UI we have today. References: 1. A. Yip et al. Improving Application Security with Data Flow Assertions. In SOSP'2009. > Improvement of current HDFS Web UI > -- > > Key: HDFS-5333 > URL: https://issues.apache.org/jira/browse/HDFS-5333 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Haohui Mai > > This is an umbrella jira for improving the current JSP-based HDFS Web UI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5295) hsftp throws an exception in the end on secure cluster with https enabled
[ https://issues.apache.org/jira/browse/HDFS-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805768#comment-13805768 ] Haohui Mai commented on HDFS-5295: -- I believe that this is due to when canceling the token, the tokenrenewer gets the wrong address. > hsftp throws an exception in the end on secure cluster with https enabled > - > > Key: HDFS-5295 > URL: https://issues.apache.org/jira/browse/HDFS-5295 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.1-beta > Environment: Secure >Reporter: Yesha Vora >Assignee: Arpit Agarwal > > The hsftp command throws "java.net.SocketException: Unexpected end of file > from server" exception in the end in secure environment and with https > enabled. > Using https port defined by dfs.https.port=50701 > /usr/bin/hdfs dfs -fs hdfs://hostname:8020 -ls -R > "hsftp://hostname:50701/user/abc/1380829410/.abc.crc"; > -rw-r--r-- 3 abc abc12 2013-10-03 19:44 > hsftp://hostname:50701/user/abc/1380829410/.abc.crc > 13/10/03 19:50:48 INFO tools.DelegationTokenFetcher: error in cancel over HTTP > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:718) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:715) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1322) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.isNegotiate(KerberosAuthenticator.java:224) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:194) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:232) > at > org.apache.hadoop.security.SecurityUtil.openSecureHttpConnection(SecurityUtil.java:512) > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.cancelDelegationToken(DelegationTokenFetcher.java:354) > at > org.apache.hadoop.hdfs.HftpFileSystem$TokenManager.cancel(HftpFileSystem.java:730) > at org.apache.hadoop.security.token.Token.cancel(Token.java:384) > at > org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152) > at > org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58) > at > org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241) > at org.apache.hadoop.hdfs.HftpFileSystem.close(HftpFileSystem.java:417) > at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2524) > at > org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2541) > at > org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) > 13/10/03 19:50:48 INFO fs.FileSystem: FileSystem.Cache.closeAll() threw an > exception: > java.net.SocketException: Unexpected end of file from server -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5413) hdfs.cmd does not support passthrough to any arbitrary class.
[
https://issues.apache.org/jira/browse/HDFS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805698#comment-13805698
]
Hadoop QA commented on HDFS-5413:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610365/HDFS-5413.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5282//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5282//console
This message is automatically generated.
> hdfs.cmd does not support passthrough to any arbitrary class.
> -
>
> Key: HDFS-5413
> URL: https://issues.apache.org/jira/browse/HDFS-5413
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5413.1.patch, HDFS-5413.2.patch
>
>
> The hdfs shell script supports passthrough to calling any arbitrary class if
> the first argument is not one of the per-defined sub-commands. The
> equivalent cmd script does not implement this and instead fails trying to do
> a labeled goto to the first argument.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5413) hdfs.cmd does not support passthrough to any arbitrary class.
[ https://issues.apache.org/jira/browse/HDFS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805682#comment-13805682 ] Chuan Liu commented on HDFS-5413: - +1 The new patch looks good to me! Thanks for fixing the problem! > hdfs.cmd does not support passthrough to any arbitrary class. > - > > Key: HDFS-5413 > URL: https://issues.apache.org/jira/browse/HDFS-5413 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5413.1.patch, HDFS-5413.2.patch > > > The hdfs shell script supports passthrough to calling any arbitrary class if > the first argument is not one of the per-defined sub-commands. The > equivalent cmd script does not implement this and instead fails trying to do > a labeled goto to the first argument. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5412) Fix NPEs in BlockManager and DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-5412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5412:
Description:
The directory scanner periodically compiles a report of differences between the
datanode's on-disk and in-memory state.
The code to generate the reports is in {{DirectoryScanner#scan}} and
{{DirectoryScanner#getDiskReport}}. It looks like the volume field in
{{ScanInfo}} is not correctly initialized while compiling the diffs. This was
not an issue before but now we depend on the volume information being present.
The bug triggers the following NPE during a scan if a block is present in the
Datanode's in-memory block map but missing on disk:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(FsDatasetImpl.java:1404)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:416)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:365)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
{code}
Another NPE exposed in {{BlockManager#reportDiff}}
{code}
java.lang.NullPointerException: null
at java.util.TreeMap.getEntry(TreeMap.java:324)
at java.util.TreeMap.remove(TreeMap.java:580)
at java.util.TreeSet.remove(TreeSet.java:259)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1836)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1709)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1637)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:978)
at org.apache.hadoop.
{code}
was:
The directory scanner periodically compiles a report of differences between the
datanode's on-disk and in-memory state.
The code to generate the reports is in {{DirectoryScanner#scan}} and
{{DirectoryScanner#getDiskReport}}. It looks like the volume field in
{{ScanInfo}} is not correctly initialized while compiling the diffs. This was
not an issue before but now we depend on the volume information being present.
The bug triggers the following NPE during a scan if a block is present in the
Datanode's in-memory block map but missing on disk:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(FsDatasetImpl.java:1404)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:416)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:365)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
{code}
> Fix NPEs in BlockManager and DirectoryScanner
> -
>
> Key: HDFS-5412
> URL: https://issues.apache.org/jira/browse/HDFS-5412
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: h5412.02.patch, h5412.03.patch
>
>
> The directory scanner periodically compiles a report of differe
[jira] [Updated] (HDFS-5412) Fix NPEs in BlockManager and DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-5412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5412:
Summary: Fix NPEs in BlockManager and DirectoryScanner (was: Directory
scanner fails to propagate volume information in disk report)
> Fix NPEs in BlockManager and DirectoryScanner
> -
>
> Key: HDFS-5412
> URL: https://issues.apache.org/jira/browse/HDFS-5412
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: h5412.02.patch, h5412.03.patch
>
>
> The directory scanner periodically compiles a report of differences between
> the datanode's on-disk and in-memory state.
> The code to generate the reports is in {{DirectoryScanner#scan}} and
> {{DirectoryScanner#getDiskReport}}. It looks like the volume field in
> {{ScanInfo}} is not correctly initialized while compiling the diffs. This was
> not an issue before but now we depend on the volume information being
> present. The bug triggers the following NPE during a scan if a block is
> present in the Datanode's in-memory block map but missing on disk:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(FsDatasetImpl.java:1404)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:416)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:365)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5412) Directory scanner fails to propagate volume information in disk report
[
https://issues.apache.org/jira/browse/HDFS-5412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5412:
Attachment: h5412.03.patch
Rolling up a few more fixes for TestBlockReport in the same patch.
# Fix TestBlockReport#blockReport_03 and split into two different test cases.
# Fix NPE in {{BlockManager#reportDiff}}
> Directory scanner fails to propagate volume information in disk report
> --
>
> Key: HDFS-5412
> URL: https://issues.apache.org/jira/browse/HDFS-5412
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: h5412.02.patch, h5412.03.patch
>
>
> The directory scanner periodically compiles a report of differences between
> the datanode's on-disk and in-memory state.
> The code to generate the reports is in {{DirectoryScanner#scan}} and
> {{DirectoryScanner#getDiskReport}}. It looks like the volume field in
> {{ScanInfo}} is not correctly initialized while compiling the diffs. This was
> not an issue before but now we depend on the volume information being
> present. The bug triggers the following NPE during a scan if a block is
> present in the Datanode's in-memory block map but missing on disk:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(FsDatasetImpl.java:1404)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:416)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:365)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5413) hdfs.cmd does not support passthrough to any arbitrary class.
[
https://issues.apache.org/jira/browse/HDFS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805637#comment-13805637
]
Chris Nauroth commented on HDFS-5413:
-
{quote}
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{quote}
The patch contains only script changes, so there are no new tests. I tested
manually in a cluster running on Windows by running all hdfs.cmd sub-commands
and one direct class to cover the passthrough case.
{quote}
-1 core tests. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
{quote}
This test is known to be flaky.
> hdfs.cmd does not support passthrough to any arbitrary class.
> -
>
> Key: HDFS-5413
> URL: https://issues.apache.org/jira/browse/HDFS-5413
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5413.1.patch, HDFS-5413.2.patch
>
>
> The hdfs shell script supports passthrough to calling any arbitrary class if
> the first argument is not one of the per-defined sub-commands. The
> equivalent cmd script does not implement this and instead fails trying to do
> a labeled goto to the first argument.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5413) hdfs.cmd does not support passthrough to any arbitrary class.
[
https://issues.apache.org/jira/browse/HDFS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805629#comment-13805629
]
Hadoop QA commented on HDFS-5413:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610342/HDFS-5413.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5280//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5280//console
This message is automatically generated.
> hdfs.cmd does not support passthrough to any arbitrary class.
> -
>
> Key: HDFS-5413
> URL: https://issues.apache.org/jira/browse/HDFS-5413
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5413.1.patch, HDFS-5413.2.patch
>
>
> The hdfs shell script supports passthrough to calling any arbitrary class if
> the first argument is not one of the per-defined sub-commands. The
> equivalent cmd script does not implement this and instead fails trying to do
> a labeled goto to the first argument.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HDFS-5424) Verify initializations of LocatedBlock/RecoveringBlock
[ https://issues.apache.org/jira/browse/HDFS-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5424. - Resolution: Duplicate Release Note: Jira seems to have created a duplicate. > Verify initializations of LocatedBlock/RecoveringBlock > -- > > Key: HDFS-5424 > URL: https://issues.apache.org/jira/browse/HDFS-5424 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Tracking Jira to make sure we verify initialization of LocatedBlock and > RecoveringBlock, possibly reorg the constructors to make missing > initialization of StorageIDs less likely. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5429) Deprecate FsDatasetSpi#getBlockReport
Arpit Agarwal created HDFS-5429:
---
Summary: Deprecate FsDatasetSpi#getBlockReport
Key: HDFS-5429
URL: https://issues.apache.org/jira/browse/HDFS-5429
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Environment: {{FsDatasetSpi#getBlockReport}} has been superseded by
{{FsDatasetSpi#getBlockReports}} which gets one block report per storage. The
old routine is only used in tests right now, deprecate it once we fix the tests.
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5257) addBlock() retry should return LocatedBlock with locations else client will get AIOBE
[
https://issues.apache.org/jira/browse/HDFS-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805590#comment-13805590
]
Hudson commented on HDFS-5257:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4656 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4656/])
HDFS-5257. addBlock() retry should return LocatedBlock with locations else
client will get AIOBE. Contributed by Vinay. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535811)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java
> addBlock() retry should return LocatedBlock with locations else client will
> get AIOBE
> -
>
> Key: HDFS-5257
> URL: https://issues.apache.org/jira/browse/HDFS-5257
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Vinay
>Assignee: Vinay
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HDFS-5257.patch, HDFS-5257.patch, HDFS-5257.patch,
> HDFS-5257.patch
>
>
> {{addBlock()}} call retry should return the LocatedBlock with locations if
> the block was created in previous call and failover/restart of namenode
> happened.
> otherwise client will get {{ArrayIndexOutOfBoundsException}} while creating
> the block and write will fail.
> {noformat}java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1118)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:511){noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5413) hdfs.cmd does not support passthrough to any arbitrary class.
[ https://issues.apache.org/jira/browse/HDFS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5413: Attachment: HDFS-5413.2.patch Thanks for another good catch. I was trying to build the list of hdfscommand from looking at the usage text. Unfortunately, the usage text was out of date, so this led me astray. Here is an updated patch that updates the usage text, adds haadmin, and also adds journalnode. This one was completely missing from hdfs.cmd. > hdfs.cmd does not support passthrough to any arbitrary class. > - > > Key: HDFS-5413 > URL: https://issues.apache.org/jira/browse/HDFS-5413 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5413.1.patch, HDFS-5413.2.patch > > > The hdfs shell script supports passthrough to calling any arbitrary class if > the first argument is not one of the per-defined sub-commands. The > equivalent cmd script does not implement this and instead fails trying to do > a labeled goto to the first argument. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5257) addBlock() retry should return LocatedBlock with locations else client will get AIOBE
[
https://issues.apache.org/jira/browse/HDFS-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5257:
Resolution: Fixed
Fix Version/s: 2.3.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this to trunk and branch-2. Thanks for the fix Vinay! And thanks
for the review Uma and Colin.
> addBlock() retry should return LocatedBlock with locations else client will
> get AIOBE
> -
>
> Key: HDFS-5257
> URL: https://issues.apache.org/jira/browse/HDFS-5257
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Vinay
>Assignee: Vinay
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HDFS-5257.patch, HDFS-5257.patch, HDFS-5257.patch,
> HDFS-5257.patch
>
>
> {{addBlock()}} call retry should return the LocatedBlock with locations if
> the block was created in previous call and failover/restart of namenode
> happened.
> otherwise client will get {{ArrayIndexOutOfBoundsException}} while creating
> the block and write will fail.
> {noformat}java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1118)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:511){noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5257) addBlock() retry should return LocatedBlock with locations else client will get AIOBE
[
https://issues.apache.org/jira/browse/HDFS-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805562#comment-13805562
]
Jing Zhao commented on HDFS-5257:
-
The failed test should be unrelated. I will commit the patch shortly.
In the meanwhile, for the current fix, can we move "add new chosen targets to
already allocated block" part from getAdditionalBlock directly into
analyzeFileState, so that we do not need to wrap the LocatedBlock twice? This
is a minor issue and we can address it in a separate jira if necessary.
> addBlock() retry should return LocatedBlock with locations else client will
> get AIOBE
> -
>
> Key: HDFS-5257
> URL: https://issues.apache.org/jira/browse/HDFS-5257
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Vinay
>Assignee: Vinay
>Priority: Critical
> Attachments: HDFS-5257.patch, HDFS-5257.patch, HDFS-5257.patch,
> HDFS-5257.patch
>
>
> {{addBlock()}} call retry should return the LocatedBlock with locations if
> the block was created in previous call and failover/restart of namenode
> happened.
> otherwise client will get {{ArrayIndexOutOfBoundsException}} while creating
> the block and write will fail.
> {noformat}java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1118)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:511){noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5413) hdfs.cmd does not support passthrough to any arbitrary class.
[ https://issues.apache.org/jira/browse/HDFS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805559#comment-13805559 ] Chuan Liu commented on HDFS-5413: - It seems haadmin is missing from 'hdfscommands'. > hdfs.cmd does not support passthrough to any arbitrary class. > - > > Key: HDFS-5413 > URL: https://issues.apache.org/jira/browse/HDFS-5413 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5413.1.patch > > > The hdfs shell script supports passthrough to calling any arbitrary class if > the first argument is not one of the per-defined sub-commands. The > equivalent cmd script does not implement this and instead fails trying to do > a labeled goto to the first argument. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5412) Directory scanner fails to propagate volume information in disk report
[
https://issues.apache.org/jira/browse/HDFS-5412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5412:
Attachment: h5412.02.patch
Patch to propagate volume information. {{FsDatasetSpi#getFinalizedBlocks}} has
been changed to avoid the unnecessary upcast from FinalizedReplica to Block.
{code}
- public synchronized List getFinalizedBlocks(String bpid) {
+ public synchronized List getFinalizedBlocks(String bpid)
{code}
This patch fixes TestBlockReport#blockReport_02. Other test cases in
TestBlockReport appear to be broken by an unrelated issue I am fixing in a
separate patch.
> Directory scanner fails to propagate volume information in disk report
> --
>
> Key: HDFS-5412
> URL: https://issues.apache.org/jira/browse/HDFS-5412
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: h5412.02.patch
>
>
> The directory scanner periodically compiles a report of differences between
> the datanode's on-disk and in-memory state.
> The code to generate the reports is in {{DirectoryScanner#scan}} and
> {{DirectoryScanner#getDiskReport}}. It looks like the volume field in
> {{ScanInfo}} is not correctly initialized while compiling the diffs. This was
> not an issue before but now we depend on the volume information being
> present. The bug triggers the following NPE during a scan if a block is
> present in the Datanode's in-memory block map but missing on disk:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(FsDatasetImpl.java:1404)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:416)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:365)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805497#comment-13805497 ] Tsz Wo (Nicholas), SZE commented on HDFS-2832: -- > h5417_20131025.patch: updated with HDFS-5417. Oops, the file should be h2832_20131025.patch. > Enable support for heterogeneous storages in HDFS > - > > Key: HDFS-2832 > URL: https://issues.apache.org/jira/browse/HDFS-2832 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: 20130813-HeterogeneousStorage.pdf, > h2832_20131023b.patch, h2832_20131023.patch, h2832_20131025.patch > > > HDFS currently supports configuration where storages are a list of > directories. Typically each of these directories correspond to a volume with > its own file system. All these directories are homogeneous and therefore > identified as a single storage at the namenode. I propose, change to the > current model where Datanode * is a * storage, to Datanode * is a collection > * of strorages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2832: - Attachment: h2832_20131025.patch > Enable support for heterogeneous storages in HDFS > - > > Key: HDFS-2832 > URL: https://issues.apache.org/jira/browse/HDFS-2832 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: 20130813-HeterogeneousStorage.pdf, > h2832_20131023b.patch, h2832_20131023.patch, h2832_20131025.patch > > > HDFS currently supports configuration where storages are a list of > directories. Typically each of these directories correspond to a volume with > its own file system. All these directories are homogeneous and therefore > identified as a single storage at the namenode. I propose, change to the > current model where Datanode * is a * storage, to Datanode * is a collection > * of strorages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2832: - Attachment: (was: h5417_20131025.patch) > Enable support for heterogeneous storages in HDFS > - > > Key: HDFS-2832 > URL: https://issues.apache.org/jira/browse/HDFS-2832 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: 20130813-HeterogeneousStorage.pdf, > h2832_20131023b.patch, h2832_20131023.patch, h2832_20131025.patch > > > HDFS currently supports configuration where storages are a list of > directories. Typically each of these directories correspond to a volume with > its own file system. All these directories are homogeneous and therefore > identified as a single storage at the namenode. I propose, change to the > current model where Datanode * is a * storage, to Datanode * is a collection > * of strorages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[
https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805492#comment-13805492
]
Hadoop QA commented on HDFS-2832:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610345/h5417_20131025.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5279//console
This message is automatically generated.
> Enable support for heterogeneous storages in HDFS
> -
>
> Key: HDFS-2832
> URL: https://issues.apache.org/jira/browse/HDFS-2832
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 0.24.0
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: 20130813-HeterogeneousStorage.pdf,
> h2832_20131023b.patch, h2832_20131023.patch, h5417_20131025.patch
>
>
> HDFS currently supports configuration where storages are a list of
> directories. Typically each of these directories correspond to a volume with
> its own file system. All these directories are homogeneous and therefore
> identified as a single storage at the namenode. I propose, change to the
> current model where Datanode * is a * storage, to Datanode * is a collection
> * of strorages.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[
https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805489#comment-13805489
]
Hadoop QA commented on HDFS-2832:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610345/h5417_20131025.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5278//console
This message is automatically generated.
> Enable support for heterogeneous storages in HDFS
> -
>
> Key: HDFS-2832
> URL: https://issues.apache.org/jira/browse/HDFS-2832
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 0.24.0
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: 20130813-HeterogeneousStorage.pdf,
> h2832_20131023b.patch, h2832_20131023.patch, h5417_20131025.patch
>
>
> HDFS currently supports configuration where storages are a list of
> directories. Typically each of these directories correspond to a volume with
> its own file system. All these directories are homogeneous and therefore
> identified as a single storage at the namenode. I propose, change to the
> current model where Datanode * is a * storage, to Datanode * is a collection
> * of strorages.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2832: - Attachment: h5417_20131025.patch h5417_20131025.patch: updated with HDFS-5417. > Enable support for heterogeneous storages in HDFS > - > > Key: HDFS-2832 > URL: https://issues.apache.org/jira/browse/HDFS-2832 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: 20130813-HeterogeneousStorage.pdf, > h2832_20131023b.patch, h2832_20131023.patch, h5417_20131025.patch > > > HDFS currently supports configuration where storages are a list of > directories. Typically each of these directories correspond to a volume with > its own file system. All these directories are homogeneous and therefore > identified as a single storage at the namenode. I propose, change to the > current model where Datanode * is a * storage, to Datanode * is a collection > * of strorages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-5417) Fix storage IDs in PBHelper and UpgradeUtilities
[ https://issues.apache.org/jira/browse/HDFS-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-5417. -- Resolution: Fixed Fix Version/s: Heterogeneous Storage (HDFS-2832) Hadoop Flags: Reviewed I have committed this. > Fix storage IDs in PBHelper and UpgradeUtilities > > > Key: HDFS-5417 > URL: https://issues.apache.org/jira/browse/HDFS-5417 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Fix For: Heterogeneous Storage (HDFS-2832) > > Attachments: h5417_20131024.patch, h5417_20131025.patch > > > - PBHelper should check null for storage IDs when converting LocatedBlock. > - UpgradeUtilities should createStorageID in createDataNodeVersionFile(..). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5413) hdfs.cmd does not support passthrough to any arbitrary class.
[ https://issues.apache.org/jira/browse/HDFS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5413: Status: Patch Available (was: Open) > hdfs.cmd does not support passthrough to any arbitrary class. > - > > Key: HDFS-5413 > URL: https://issues.apache.org/jira/browse/HDFS-5413 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 2.2.0, 3.0.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5413.1.patch > > > The hdfs shell script supports passthrough to calling any arbitrary class if > the first argument is not one of the per-defined sub-commands. The > equivalent cmd script does not implement this and instead fails trying to do > a labeled goto to the first argument. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5413) hdfs.cmd does not support passthrough to any arbitrary class.
[ https://issues.apache.org/jira/browse/HDFS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5413: Attachment: HDFS-5413.1.patch Attaching patch to fix hdfs.cmd. BTW, I also reviewed hadoop.cmd for the same bug, but that one is fine. > hdfs.cmd does not support passthrough to any arbitrary class. > - > > Key: HDFS-5413 > URL: https://issues.apache.org/jira/browse/HDFS-5413 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5413.1.patch > > > The hdfs shell script supports passthrough to calling any arbitrary class if > the first argument is not one of the per-defined sub-commands. The > equivalent cmd script does not implement this and instead fails trying to do > a labeled goto to the first argument. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5417) Fix storage IDs in PBHelper and UpgradeUtilities
[ https://issues.apache.org/jira/browse/HDFS-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805479#comment-13805479 ] Junping Du commented on HDFS-5417: -- +1. Patch looks good. > Fix storage IDs in PBHelper and UpgradeUtilities > > > Key: HDFS-5417 > URL: https://issues.apache.org/jira/browse/HDFS-5417 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h5417_20131024.patch, h5417_20131025.patch > > > - PBHelper should check null for storage IDs when converting LocatedBlock. > - UpgradeUtilities should createStorageID in createDataNodeVersionFile(..). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5426) Fix NPE in PBHelper
[ https://issues.apache.org/jira/browse/HDFS-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805477#comment-13805477 ] Junping Du commented on HDFS-5426: -- Ok. Let's mark it as duplicated. Thanks! > Fix NPE in PBHelper > --- > > Key: HDFS-5426 > URL: https://issues.apache.org/jira/browse/HDFS-5426 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Junping Du >Assignee: Junping Du > Attachments: HDFS-5426.patch > > > According to > https://builds.apache.org/job/PreCommit-HDFS-Build/5268/testReport/, 6 test > failures in TestPBHelper due to NPE issues if not setting storageIDs before. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-5426) Fix NPE in PBHelper
[ https://issues.apache.org/jira/browse/HDFS-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du resolved HDFS-5426. -- Resolution: Duplicate > Fix NPE in PBHelper > --- > > Key: HDFS-5426 > URL: https://issues.apache.org/jira/browse/HDFS-5426 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Junping Du >Assignee: Junping Du > Attachments: HDFS-5426.patch > > > According to > https://builds.apache.org/job/PreCommit-HDFS-Build/5268/testReport/, 6 test > failures in TestPBHelper due to NPE issues if not setting storageIDs before. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5426) Fix NPE in PBHelper
[ https://issues.apache.org/jira/browse/HDFS-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805474#comment-13805474 ] Tsz Wo (Nicholas), SZE commented on HDFS-5426: -- Hi Junping, I have covered this in HDFS-5417. > Fix NPE in PBHelper > --- > > Key: HDFS-5426 > URL: https://issues.apache.org/jira/browse/HDFS-5426 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Junping Du >Assignee: Junping Du > Attachments: HDFS-5426.patch > > > According to > https://builds.apache.org/job/PreCommit-HDFS-Build/5268/testReport/, 6 test > failures in TestPBHelper due to NPE issues if not setting storageIDs before. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5417) Fix storage IDs in PBHelper and UpgradeUtilities
[ https://issues.apache.org/jira/browse/HDFS-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805472#comment-13805472 ] Arpit Agarwal commented on HDFS-5417: - +1 for the patch. > Fix storage IDs in PBHelper and UpgradeUtilities > > > Key: HDFS-5417 > URL: https://issues.apache.org/jira/browse/HDFS-5417 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h5417_20131024.patch, h5417_20131025.patch > > > - PBHelper should check null for storage IDs when converting LocatedBlock. > - UpgradeUtilities should createStorageID in createDataNodeVersionFile(..). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5417) Fix storage IDs in PBHelper and UpgradeUtilities
[ https://issues.apache.org/jira/browse/HDFS-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5417: - Attachment: h5417_20131025.patch Arpit, thanks for taking a look. Here is an updated patch. h5417_20131025.patch > Fix storage IDs in PBHelper and UpgradeUtilities > > > Key: HDFS-5417 > URL: https://issues.apache.org/jira/browse/HDFS-5417 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h5417_20131024.patch, h5417_20131025.patch > > > - PBHelper should check null for storage IDs when converting LocatedBlock. > - UpgradeUtilities should createStorageID in createDataNodeVersionFile(..). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4949) Centralized cache management in HDFS
[ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805463#comment-13805463 ] Chris Nauroth commented on HDFS-4949: - Here is a list of items discussed in the design doc to be completed later, with corresponding jira if it exists: * quota enforcement - need to file jira? * cache expiry based on TTL - need to file jira? * incremental cache reports - HDFS-5092 * metrics - HDFS-5320 > Centralized cache management in HDFS > > > Key: HDFS-4949 > URL: https://issues.apache.org/jira/browse/HDFS-4949 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.0.0, 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: caching-design-doc-2013-07-02.pdf, > caching-design-doc-2013-08-09.pdf, caching-design-doc-2013-10-24.pdf, > caching-testplan.pdf, HDFS-4949-consolidated.patch > > > HDFS currently has no support for managing or exposing in-memory caches at > datanodes. This makes it harder for higher level application frameworks like > Hive, Pig, and Impala to effectively use cluster memory, because they cannot > explicitly cache important datasets or place their tasks for memory locality. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5333) Improvement of current HDFS Web UI
[ https://issues.apache.org/jira/browse/HDFS-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805457#comment-13805457 ] Luke Lu commented on HDFS-5333: --- I have concerns with this client-side js only approach, which is less secure than a progressively enhanced hybrid approach used by YARN. The recent gmail XSS fiasco highlights the issue. I also have concerns that we commit these changes without matching unit tests -- the fact you cannot effectively unit test these changes should tell you something about this approach. _Requiring_ JS means that an admin cannot turn off js to (partially) use core Hadoop UI. You'd _require_ proper SSL (not self signed) setup to avoid JS injection, even if security of js libraries used is perfect, which I doubt (search gmail/linkedin XSS). Client side rendering completely breaks the workflows for ops who rely on text based terminal/emacs/vim browsers (no js support) to monitor component UI. IMO, JS-only rendering belongs to social networking sites and/or SaaS front-ends. I think eventually most users will use a self servicing UI in a SaaS front-end that uses REST/JMX API to get data from back-end components, besides their own app master/service UI. The priority/requirements for UI in core Hadoop should be security and correctness, which client side templating cannot address properly so far. > Improvement of current HDFS Web UI > -- > > Key: HDFS-5333 > URL: https://issues.apache.org/jira/browse/HDFS-5333 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Haohui Mai > > This is an umbrella jira for improving the current JSP-based HDFS Web UI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4949) Centralized cache management in HDFS
[ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805435#comment-13805435 ] Suresh Srinivas commented on HDFS-4949: --- Given that the merge vote thread has been started, can someone post details about what functionality from original design has been completed and what is pending? Looks like the current functionality does not cover quota management. Any other features pending? > Centralized cache management in HDFS > > > Key: HDFS-4949 > URL: https://issues.apache.org/jira/browse/HDFS-4949 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.0.0, 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: caching-design-doc-2013-07-02.pdf, > caching-design-doc-2013-08-09.pdf, caching-design-doc-2013-10-24.pdf, > caching-testplan.pdf, HDFS-4949-consolidated.patch > > > HDFS currently has no support for managing or exposing in-memory caches at > datanodes. This makes it harder for higher level application frameworks like > Hive, Pig, and Impala to effectively use cluster memory, because they cannot > explicitly cache important datasets or place their tasks for memory locality. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangjingyou updated HDFS-5138: --- Issue Type: Bug (was: Test) > Support HDFS upgrade in HA > -- > > Key: HDFS-5138 > URL: https://issues.apache.org/jira/browse/HDFS-5138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Kihwal Lee >Priority: Blocker > > With HA enabled, NN wo't start with "-upgrade". Since there has been a layout > version change between 2.0.x and 2.1.x, starting NN in upgrade mode was > necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way > to get around this was to disable HA and upgrade. > The NN and the cluster cannot be flipped back to HA until the upgrade is > finalized. If HA is disabled only on NN for layout upgrade and HA is turned > back on without involving DNs, things will work, but finaliizeUpgrade won't > work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade > snapshots won't get removed. > We will need a different ways of doing layout upgrade and upgrade snapshot. > I am marking this as a 2.1.1-beta blocker based on feedback from others. If > there is a reasonable workaround that does not increase maintenance window > greatly, we can lower its priority from blocker to critical. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangjingyou updated HDFS-5138: --- Issue Type: Test (was: Bug) > Support HDFS upgrade in HA > -- > > Key: HDFS-5138 > URL: https://issues.apache.org/jira/browse/HDFS-5138 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 2.1.1-beta >Reporter: Kihwal Lee >Priority: Blocker > > With HA enabled, NN wo't start with "-upgrade". Since there has been a layout > version change between 2.0.x and 2.1.x, starting NN in upgrade mode was > necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way > to get around this was to disable HA and upgrade. > The NN and the cluster cannot be flipped back to HA until the upgrade is > finalized. If HA is disabled only on NN for layout upgrade and HA is turned > back on without involving DNs, things will work, but finaliizeUpgrade won't > work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade > snapshots won't get removed. > We will need a different ways of doing layout upgrade and upgrade snapshot. > I am marking this as a 2.1.1-beta blocker based on feedback from others. If > there is a reasonable workaround that does not increase maintenance window > greatly, we can lower its priority from blocker to critical. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5428) under construction files deletion after snapshot+checkpoint+nn restart leads nn safemode
[
https://issues.apache.org/jira/browse/HDFS-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805328#comment-13805328
]
Hadoop QA commented on HDFS-5428:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610312/HDFS-5428.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5277//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5277//console
This message is automatically generated.
> under construction files deletion after snapshot+checkpoint+nn restart leads
> nn safemode
>
>
> Key: HDFS-5428
> URL: https://issues.apache.org/jira/browse/HDFS-5428
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5428.patch
>
>
> 1. allow snapshots under dir /foo
> 2. create a file /foo/test/bar and start writing to it
> 3. create a snapshot s1 under /foo after block is allocated and some data has
> been written to it
> 4. Delete the directory /foo/test
> 4. wait till checkpoint or do saveNameSpace
> 5. restart NN.
> NN enters to safemode.
> Analysis:
> Snapshot nodes loaded from fsimage are always complete and all blocks will be
> in COMPLETE state.
> So when the Datanode reports RBW blocks those will not be updated in
> blocksmap.
> Some of the FINALIZED blocks will be marked as corrupt due to length mismatch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5171) NFS should create input stream for a file and try to share it with multiple read requests
[ https://issues.apache.org/jira/browse/HDFS-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805316#comment-13805316 ] Hudson commented on HDFS-5171: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1589 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1589/]) HDFS-5171. NFS should create input stream for a file and try to share it with multiple read requests. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535586) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestDFSClientCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > NFS should create input stream for a file and try to share it with multiple > read requests > - > > Key: HDFS-5171 > URL: https://issues.apache.org/jira/browse/HDFS-5171 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Haohui Mai > Fix For: 2.2.1 > > Attachments: HDFS-5171.000.patch, HDFS-5171.001.patch > > > Currently, NFS creates an input steam for each read request and closes it > after the request is served. With lots of read request, the overhead is > significant. > Like for write request, NFS should create input stream for a file and try to > share it with multiple read requests. The stream can be closed if there is no > read request for a certain amount of time (e.g., 10 sec). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[
https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805307#comment-13805307
]
Hadoop QA commented on HDFS-5427:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610308/HDFS-5427.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5276//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5276//console
This message is automatically generated.
> not able to read deleted files from snapshot directly under snapshottable dir
> after checkpoint and NN restart
> -
>
> Key: HDFS-5427
> URL: https://issues.apache.org/jira/browse/HDFS-5427
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5427.patch, HDFS-5427.patch
>
>
> 1. allow snapshots under dir /foo
> 2. create a file /foo/bar
> 3. create a snapshot s1 under /foo
> 4. wait till checkpoint or do saveNameSpace
> 5. restart NN.
> 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar
> client will get BlockMissingException
> Reason is
> While loading the deleted file list for a snashottable dir from fsimage,
> blocks were not updated in blocksmap
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5171) NFS should create input stream for a file and try to share it with multiple read requests
[ https://issues.apache.org/jira/browse/HDFS-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805293#comment-13805293 ] Hudson commented on HDFS-5171: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1563 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1563/]) HDFS-5171. NFS should create input stream for a file and try to share it with multiple read requests. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535586) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestDFSClientCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > NFS should create input stream for a file and try to share it with multiple > read requests > - > > Key: HDFS-5171 > URL: https://issues.apache.org/jira/browse/HDFS-5171 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Haohui Mai > Fix For: 2.2.1 > > Attachments: HDFS-5171.000.patch, HDFS-5171.001.patch > > > Currently, NFS creates an input steam for each read request and closes it > after the request is served. With lots of read request, the overhead is > significant. > Like for write request, NFS should create input stream for a file and try to > share it with multiple read requests. The stream can be closed if there is no > read request for a certain amount of time (e.g., 10 sec). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5428) under construction files deletion after snapshot+checkpoint+nn restart leads nn safemode
[ https://issues.apache.org/jira/browse/HDFS-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5428: Assignee: Vinay Affects Version/s: 3.0.0 2.2.0 Status: Patch Available (was: Open) > under construction files deletion after snapshot+checkpoint+nn restart leads > nn safemode > > > Key: HDFS-5428 > URL: https://issues.apache.org/jira/browse/HDFS-5428 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0, 3.0.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5428.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/test/bar and start writing to it > 3. create a snapshot s1 under /foo after block is allocated and some data has > been written to it > 4. Delete the directory /foo/test > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > NN enters to safemode. > Analysis: > Snapshot nodes loaded from fsimage are always complete and all blocks will be > in COMPLETE state. > So when the Datanode reports RBW blocks those will not be updated in > blocksmap. > Some of the FINALIZED blocks will be marked as corrupt due to length mismatch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5428) under construction files deletion after snapshot+checkpoint+nn restart leads nn safemode
[ https://issues.apache.org/jira/browse/HDFS-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5428: Attachment: HDFS-5428.patch Attaching a patch which solves the current issue. Please review and suggest if any improvements needs to be done. Thanks > under construction files deletion after snapshot+checkpoint+nn restart leads > nn safemode > > > Key: HDFS-5428 > URL: https://issues.apache.org/jira/browse/HDFS-5428 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Vinay > Attachments: HDFS-5428.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/test/bar and start writing to it > 3. create a snapshot s1 under /foo after block is allocated and some data has > been written to it > 4. Delete the directory /foo/test > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > NN enters to safemode. > Analysis: > Snapshot nodes loaded from fsimage are always complete and all blocks will be > in COMPLETE state. > So when the Datanode reports RBW blocks those will not be updated in > blocksmap. > Some of the FINALIZED blocks will be marked as corrupt due to length mismatch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Moved] (HDFS-5428) under construction files deletion after snapshot+checkpoint+nn restart leads nn safemode
[ https://issues.apache.org/jira/browse/HDFS-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay moved HADOOP-10071 to HDFS-5428: -- Key: HDFS-5428 (was: HADOOP-10071) Project: Hadoop HDFS (was: Hadoop Common) > under construction files deletion after snapshot+checkpoint+nn restart leads > nn safemode > > > Key: HDFS-5428 > URL: https://issues.apache.org/jira/browse/HDFS-5428 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Vinay > > 1. allow snapshots under dir /foo > 2. create a file /foo/test/bar and start writing to it > 3. create a snapshot s1 under /foo after block is allocated and some data has > been written to it > 4. Delete the directory /foo/test > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > NN enters to safemode. > Analysis: > Snapshot nodes loaded from fsimage are always complete and all blocks will be > in COMPLETE state. > So when the Datanode reports RBW blocks those will not be updated in > blocksmap. > Some of the FINALIZED blocks will be marked as corrupt due to length mismatch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[ https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5427: Attachment: HDFS-5427.patch > not able to read deleted files from snapshot directly under snapshottable dir > after checkpoint and NN restart > - > > Key: HDFS-5427 > URL: https://issues.apache.org/jira/browse/HDFS-5427 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.2.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5427.patch, HDFS-5427.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/bar > 3. create a snapshot s1 under /foo > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar > client will get BlockMissingException > Reason is > While loading the deleted file list for a snashottable dir from fsimage, > blocks were not updated in blocksmap -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[ https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5427: Status: Patch Available (was: Open) > not able to read deleted files from snapshot directly under snapshottable dir > after checkpoint and NN restart > - > > Key: HDFS-5427 > URL: https://issues.apache.org/jira/browse/HDFS-5427 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.2.0, 3.0.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5427.patch, HDFS-5427.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/bar > 3. create a snapshot s1 under /foo > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar > client will get BlockMissingException > Reason is > While loading the deleted file list for a snashottable dir from fsimage, > blocks were not updated in blocksmap -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[ https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5427: Status: Open (was: Patch Available) > not able to read deleted files from snapshot directly under snapshottable dir > after checkpoint and NN restart > - > > Key: HDFS-5427 > URL: https://issues.apache.org/jira/browse/HDFS-5427 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.2.0, 3.0.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5427.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/bar > 3. create a snapshot s1 under /foo > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar > client will get BlockMissingException > Reason is > While loading the deleted file list for a snashottable dir from fsimage, > blocks were not updated in blocksmap -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[ https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5427: Attachment: HDFS-5427.patch Attaching the correct patch > not able to read deleted files from snapshot directly under snapshottable dir > after checkpoint and NN restart > - > > Key: HDFS-5427 > URL: https://issues.apache.org/jira/browse/HDFS-5427 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.2.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5427.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/bar > 3. create a snapshot s1 under /foo > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar > client will get BlockMissingException > Reason is > While loading the deleted file list for a snashottable dir from fsimage, > blocks were not updated in blocksmap -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[ https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5427: Attachment: (was: HDFS-5427.patch) > not able to read deleted files from snapshot directly under snapshottable dir > after checkpoint and NN restart > - > > Key: HDFS-5427 > URL: https://issues.apache.org/jira/browse/HDFS-5427 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.2.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5427.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/bar > 3. create a snapshot s1 under /foo > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar > client will get BlockMissingException > Reason is > While loading the deleted file list for a snashottable dir from fsimage, > blocks were not updated in blocksmap -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[
https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805229#comment-13805229
]
Hadoop QA commented on HDFS-5427:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610305/HDFS-5427.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5275//console
This message is automatically generated.
> not able to read deleted files from snapshot directly under snapshottable dir
> after checkpoint and NN restart
> -
>
> Key: HDFS-5427
> URL: https://issues.apache.org/jira/browse/HDFS-5427
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Vinay
>Assignee: Vinay
> Attachments: HDFS-5427.patch
>
>
> 1. allow snapshots under dir /foo
> 2. create a file /foo/bar
> 3. create a snapshot s1 under /foo
> 4. wait till checkpoint or do saveNameSpace
> 5. restart NN.
> 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar
> client will get BlockMissingException
> Reason is
> While loading the deleted file list for a snashottable dir from fsimage,
> blocks were not updated in blocksmap
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[ https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5427: Affects Version/s: 3.0.0 2.2.0 Status: Patch Available (was: Open) > not able to read deleted files from snapshot directly under snapshottable dir > after checkpoint and NN restart > - > > Key: HDFS-5427 > URL: https://issues.apache.org/jira/browse/HDFS-5427 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.2.0, 3.0.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5427.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/bar > 3. create a snapshot s1 under /foo > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar > client will get BlockMissingException > Reason is > While loading the deleted file list for a snashottable dir from fsimage, > blocks were not updated in blocksmap -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
[ https://issues.apache.org/jira/browse/HDFS-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5427: Attachment: HDFS-5427.patch Attaching a patch. Please review > not able to read deleted files from snapshot directly under snapshottable dir > after checkpoint and NN restart > - > > Key: HDFS-5427 > URL: https://issues.apache.org/jira/browse/HDFS-5427 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5427.patch > > > 1. allow snapshots under dir /foo > 2. create a file /foo/bar > 3. create a snapshot s1 under /foo > 4. wait till checkpoint or do saveNameSpace > 5. restart NN. > 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar > client will get BlockMissingException > Reason is > While loading the deleted file list for a snashottable dir from fsimage, > blocks were not updated in blocksmap -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5171) NFS should create input stream for a file and try to share it with multiple read requests
[ https://issues.apache.org/jira/browse/HDFS-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805222#comment-13805222 ] Hudson commented on HDFS-5171: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #373 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/373/]) HDFS-5171. NFS should create input stream for a file and try to share it with multiple read requests. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535586) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestDFSClientCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > NFS should create input stream for a file and try to share it with multiple > read requests > - > > Key: HDFS-5171 > URL: https://issues.apache.org/jira/browse/HDFS-5171 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Haohui Mai > Fix For: 2.2.1 > > Attachments: HDFS-5171.000.patch, HDFS-5171.001.patch > > > Currently, NFS creates an input steam for each read request and closes it > after the request is served. With lots of read request, the overhead is > significant. > Like for write request, NFS should create input stream for a file and try to > share it with multiple read requests. The stream can be closed if there is no > read request for a certain amount of time (e.g., 10 sec). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5427) not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart
Vinay created HDFS-5427: --- Summary: not able to read deleted files from snapshot directly under snapshottable dir after checkpoint and NN restart Key: HDFS-5427 URL: https://issues.apache.org/jira/browse/HDFS-5427 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Reporter: Vinay Assignee: Vinay 1. allow snapshots under dir /foo 2. create a file /foo/bar 3. create a snapshot s1 under /foo 4. wait till checkpoint or do saveNameSpace 5. restart NN. 6. Now try to read the file from snapshot /foo/.snapshot/s1/bar client will get BlockMissingException Reason is While loading the deleted file list for a snashottable dir from fsimage, blocks were not updated in blocksmap -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5257) addBlock() retry should return LocatedBlock with locations else client will get AIOBE
[
https://issues.apache.org/jira/browse/HDFS-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805186#comment-13805186
]
Hadoop QA commented on HDFS-5257:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610283/HDFS-5257.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5274//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5274//console
This message is automatically generated.
> addBlock() retry should return LocatedBlock with locations else client will
> get AIOBE
> -
>
> Key: HDFS-5257
> URL: https://issues.apache.org/jira/browse/HDFS-5257
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Vinay
>Assignee: Vinay
>Priority: Critical
> Attachments: HDFS-5257.patch, HDFS-5257.patch, HDFS-5257.patch,
> HDFS-5257.patch
>
>
> {{addBlock()}} call retry should return the LocatedBlock with locations if
> the block was created in previous call and failover/restart of namenode
> happened.
> otherwise client will get {{ArrayIndexOutOfBoundsException}} while creating
> the block and write will fail.
> {noformat}java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1118)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:511){noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5426) Fix NPE in PBHelper
[ https://issues.apache.org/jira/browse/HDFS-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805138#comment-13805138 ] Junping Du commented on HDFS-5426: -- Verify all failed tests in TestPBHelper are fixed by this patch. > Fix NPE in PBHelper > --- > > Key: HDFS-5426 > URL: https://issues.apache.org/jira/browse/HDFS-5426 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Junping Du >Assignee: Junping Du > Attachments: HDFS-5426.patch > > > According to > https://builds.apache.org/job/PreCommit-HDFS-Build/5268/testReport/, 6 test > failures in TestPBHelper due to NPE issues if not setting storageIDs before. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5426) Fix NPE in PBHelper
[ https://issues.apache.org/jira/browse/HDFS-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-5426: - Attachment: HDFS-5426.patch > Fix NPE in PBHelper > --- > > Key: HDFS-5426 > URL: https://issues.apache.org/jira/browse/HDFS-5426 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Junping Du >Assignee: Junping Du > Attachments: HDFS-5426.patch > > > According to > https://builds.apache.org/job/PreCommit-HDFS-Build/5268/testReport/, 6 test > failures in TestPBHelper due to NPE issues if not setting storageIDs before. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5426) Fix NPE in PBHelper
Junping Du created HDFS-5426: Summary: Fix NPE in PBHelper Key: HDFS-5426 URL: https://issues.apache.org/jira/browse/HDFS-5426 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Junping Du Assignee: Junping Du According to https://builds.apache.org/job/PreCommit-HDFS-Build/5268/testReport/, 6 test failures in TestPBHelper due to NPE issues if not setting storageIDs before. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5257) addBlock() retry should return LocatedBlock with locations else client will get AIOBE
[
https://issues.apache.org/jira/browse/HDFS-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805104#comment-13805104
]
Jing Zhao commented on HDFS-5257:
-
bq. So I feel better both should be present.
Yeah, we should fix the both ends. The server side fix guarantee the correct
retry, while the client side's fix allows ppl to have a temporary fix in case
they do not want to upgrade server immediately.
+1 for the new patch.
> addBlock() retry should return LocatedBlock with locations else client will
> get AIOBE
> -
>
> Key: HDFS-5257
> URL: https://issues.apache.org/jira/browse/HDFS-5257
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Vinay
>Assignee: Vinay
>Priority: Critical
> Attachments: HDFS-5257.patch, HDFS-5257.patch, HDFS-5257.patch,
> HDFS-5257.patch
>
>
> {{addBlock()}} call retry should return the LocatedBlock with locations if
> the block was created in previous call and failover/restart of namenode
> happened.
> otherwise client will get {{ArrayIndexOutOfBoundsException}} while creating
> the block and write will fail.
> {noformat}java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1118)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:511){noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
