[jira] [Commented] (HDFS-4101) ZKFC should implement zookeeper.recovery.retry like HBase to connect to ZooKeeper
[
https://issues.apache.org/jira/browse/HDFS-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525090#comment-14525090
]
Hadoop QA commented on HDFS-4101:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 1m 0s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12550528/HDFS-4101-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10708/console |
This message was automatically generated.
> ZKFC should implement zookeeper.recovery.retry like HBase to connect to
> ZooKeeper
> -
>
> Key: HDFS-4101
> URL: https://issues.apache.org/jira/browse/HDFS-4101
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: auto-failover, ha
>Affects Versions: 2.0.0-alpha, 3.0.0
> Environment: running CDH4.1.1
>Reporter: Damien Hardy
>Assignee: Damien Hardy
>Priority: Minor
> Labels: newbie
> Attachments: HDFS-4101-2.patch
>
>
> When zkfc start and zookeeper is not yet started ZKFC fails and stop directly.
> Maybe ZKFC should allow some retries on Zookeeper services like does HBase
> with zookeeper.recovery.retry
> This particularly appends when I start my whole cluster on VirtualBox for
> example (every components nearly at the same time) ZKFC is the only that fail
> and stop ...
> Every others can wait each-others some time independently of the start order
> like NameNode/DataNode/JournalNode/Zookeeper/HBaseMaster/HBaseRS so that the
> system can be set and stable in few seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8309) Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows
[
https://issues.apache.org/jira/browse/HDFS-8309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525074#comment-14525074
]
Hadoop QA commented on HDFS-8309:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 5s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 22s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 12s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 1s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 1m 18s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 165m 10s | Tests passed in hadoop-hdfs.
|
| | | 186m 35s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12729894/HDFS-8309.00.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10517/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10517/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10517/console |
This message was automatically generated.
> Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows
>
>
> Key: HDFS-8309
> URL: https://issues.apache.org/jira/browse/HDFS-8309
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.7.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Minor
> Attachments: HDFS-8309.00.patch
>
>
> As [~cnauroth] noted in HDFS-7917 below,
> DataNodeTestUtils.injectDataDirFailure() won't work for Windows as rename
> will fail due to open handles on data node dir. This ticket is opened to skip
> these tests for Windows.
> bq.Unfortunately, I just remembered that the rename isn't going to work on
> Windows. It typically doesn't allow you to rename a directory where there are
> open file handles anywhere in the sub-tree. We'd have to shutdown the
> DataNode before doing the rename and then start it up. By doing that, we'd be
> changing the meaning of the test from covering an online failure to covering
> a failure at DataNode startup, so I don't think we can make that change.
> Below are the two test cases that need to be fixed:
> # TestDataNodeVolumeFailure#testFailedVolumeBeingRemovedFromDataNode
> # TestDataNodeHotSwapVolumes.testDirectlyReloadAfterCheckDiskError
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5546) race condition crashes "hadoop ls -R" when directories are moved/removed
[
https://issues.apache.org/jira/browse/HDFS-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525075#comment-14525075
]
Hadoop QA commented on HDFS-5546:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 33s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 30s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 5s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 40s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | common tests | 24m 9s | Tests passed in
hadoop-common. |
| | | 60m 56s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12652070/HDFS-5546.2.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10663/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10663/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10663/console |
This message was automatically generated.
> race condition crashes "hadoop ls -R" when directories are moved/removed
>
>
> Key: HDFS-5546
> URL: https://issues.apache.org/jira/browse/HDFS-5546
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Attachments: HDFS-5546.1.patch, HDFS-5546.2.000.patch,
> HDFS-5546.2.001.patch, HDFS-5546.2.002.patch, HDFS-5546.2.003.patch,
> HDFS-5546.2.004.patch
>
>
> This seems to be a rare race condition where we have a sequence of events
> like this:
> 1. org.apache.hadoop.shell.Ls calls DFS#getFileStatus on directory D.
> 2. someone deletes or moves directory D
> 3. org.apache.hadoop.shell.Ls calls PathData#getDirectoryContents(D), which
> calls DFS#listStatus(D). This throws FileNotFoundException.
> 4. ls command terminates with FNF
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5952) Create a tool to run data analysis on the PB format fsimage
[
https://issues.apache.org/jira/browse/HDFS-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525030#comment-14525030
]
Hadoop QA commented on HDFS-5952:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12663638/HDFS-5952.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10700/console |
This message was automatically generated.
> Create a tool to run data analysis on the PB format fsimage
> ---
>
> Key: HDFS-5952
> URL: https://issues.apache.org/jira/browse/HDFS-5952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.6.0
>Reporter: Akira AJISAKA
> Attachments: HDFS-5952.patch
>
>
> Delimited processor in OfflineImageViewer is not supported after HDFS-5698
> was merged.
> The motivation of delimited processor is to run data analysis on the fsimage,
> therefore, there might be more values to create a tool for Hive or Pig that
> reads the PB format fsimage directly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6660) Use int instead of object reference to DatanodeStorageInfo in BlockInfo triplets,
[
https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525026#comment-14525026
]
Hadoop QA commented on HDFS-6660:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12666461/HDFS-6660.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10696/console |
This message was automatically generated.
> Use int instead of object reference to DatanodeStorageInfo in BlockInfo
> triplets,
> -
>
> Key: HDFS-6660
> URL: https://issues.apache.org/jira/browse/HDFS-6660
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.4.1
>Reporter: Amir Langer
>Assignee: Amir Langer
> Labels: performance
> Attachments: HDFS-6660.patch
>
>
> Map an int index to every DatanodeStorageInfo and use it instead of object
> reference in the BlockInfo triplets data structure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525041#comment-14525041
]
Hadoop QA commented on HDFS-6980:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12665841/HDFS-6980.1-2.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10704/console |
This message was automatically generated.
> TestWebHdfsFileSystemContract fails in trunk
>
>
> Key: HDFS-6980
> URL: https://issues.apache.org/jira/browse/HDFS-6980
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Tsuyoshi Ozawa
> Attachments: HDFS-6980.1-2.patch, HDFS-6980.1.patch
>
>
> Many tests in TestWebHdfsFileSystemContract fail by "too many open files"
> error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[
https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525028#comment-14525028
]
Hadoop QA commented on HDFS-6914:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12671201/HDFS-6914.v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10698/console |
This message was automatically generated.
> Resolve huge memory consumption Issue with OIV processing PB-based fsimages
> ---
>
> Key: HDFS-6914
> URL: https://issues.apache.org/jira/browse/HDFS-6914
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Hao Chen
> Labels: hdfs
> Attachments: HDFS-6914.patch, HDFS-6914.v2.patch
>
>
> For better managing and supporting a lot of large hadoop clusters in
> production, we internally need to automatically export fsimage to delimited
> text files in LSR style and then analyse with hive or pig or build system
> metrics for real-time analyzing.
> However due to the internal layout changes introduced by the protobuf-based
> fsimage, OIV processing program consumes excessive amount of memory. For
> example, in order to export the fsimage in size of 8GB, it should have taken
> about 85GB memory which is really not reasonable and impacted performance of
> other services badly in the same server.
> To resolve above problem, I submit this patch which will reduce memory
> consumption of OIV LSR processing by 50%.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7264) Tha last datanode in a pipeline should send a heartbeat when there is no traffic
[
https://issues.apache.org/jira/browse/HDFS-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525036#comment-14525036
]
Hadoop QA commented on HDFS-7264:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12675891/h7264_20141020.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10702/console |
This message was automatically generated.
> Tha last datanode in a pipeline should send a heartbeat when there is no
> traffic
>
>
> Key: HDFS-7264
> URL: https://issues.apache.org/jira/browse/HDFS-7264
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h7264_20141017.patch, h7264_20141020.patch
>
>
> When the client is writing slowly, the client will send a heartbeat to signal
> that the connection is still alive. This case works fine.
> However, when a client is writing fast but some of the datanodes in the
> pipeline are busy, a PacketResponder may get a timeout since no ack is sent
> from the upstream datanode. We suggest that the last datanode in a pipeline
> should send a heartbeat when there is no traffic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7485) Avoid string operations in FSPermissionChecker
[
https://issues.apache.org/jira/browse/HDFS-7485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525034#comment-14525034
]
Hadoop QA commented on HDFS-7485:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12685507/HDFS-7485.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10701/console |
This message was automatically generated.
> Avoid string operations in FSPermissionChecker
> --
>
> Key: HDFS-7485
> URL: https://issues.apache.org/jira/browse/HDFS-7485
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-7485.000.patch
>
>
> Currently {{FSPermissionChecker}} compares strings when testing users and
> groups. It should compare the id assigned by SerialNumberManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7401) Add block info to DFSInputStream' WARN message when it adds node to deadNodes
[
https://issues.apache.org/jira/browse/HDFS-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525043#comment-14525043
]
Hadoop QA commented on HDFS-7401:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12681956/HDFS-7401.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10706/console |
This message was automatically generated.
> Add block info to DFSInputStream' WARN message when it adds node to deadNodes
> -
>
> Key: HDFS-7401
> URL: https://issues.apache.org/jira/browse/HDFS-7401
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Keith Pak
>Priority: Minor
> Attachments: HDFS-7401.patch
>
>
> Block info is missing in the below message
> {noformat}
> 2014-11-14 03:59:00,386 WARN org.apache.hadoop.hdfs.DFSClient: Failed to
> connect to /xx.xx.xx.xxx:50010 for block, add to deadNodes and continue.
> java.io.IOException: Got error for OP_READ_BLOCK
> {noformat}
> The code
> {noformat}
> DFSInputStream.java
> DFSClient.LOG.warn("Failed to connect to " + targetAddr + " for
> block"
> + ", add to deadNodes and continue. " + ex, ex);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5757) refreshNodes with many nodes at the same time could slow down NN
[
https://issues.apache.org/jira/browse/HDFS-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525029#comment-14525029
]
Hadoop QA commented on HDFS-5757:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12685508/HDFS-5757.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10699/console |
This message was automatically generated.
> refreshNodes with many nodes at the same time could slow down NN
>
>
> Key: HDFS-5757
> URL: https://issues.apache.org/jira/browse/HDFS-5757
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-5757.patch
>
>
> Sometimes we need to decomm a whole rack of nodes at the same time. When the
> decomm is in process; NN is slow.
> The reason is when DecommissionManager checks the decomm status, it acquires
> namesystem's writer lock and iterates through all DNs; for each DN that is in
> decommissioning state, it check if replication is done for all the blocks on
> the machine via blockManager.isReplicationInProgress; for large cluster; the
> number of blocks on the machine could be big.
> The fix could be to have DecommissionManager check for several
> decomm-in-progress nodes each time it aquires namesystem's writer lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7123) Run legacy fsimage checkpoint in parallel with PB fsimage checkpoint
[
https://issues.apache.org/jira/browse/HDFS-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525027#comment-14525027
]
Hadoop QA commented on HDFS-7123:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12670479/HDFS-7123.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10697/console |
This message was automatically generated.
> Run legacy fsimage checkpoint in parallel with PB fsimage checkpoint
>
>
> Key: HDFS-7123
> URL: https://issues.apache.org/jira/browse/HDFS-7123
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-7123.patch
>
>
> HDFS-7097 will address the checkpoint and BR issue. In addition, it might
> still be useful to reduce the overall checkpoint duration, given it blocks
> edit log replay. If there is large volume of edit log to catch up and NN fail
> overs, it will impact the availability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7174) Support for more efficient large directories
[
https://issues.apache.org/jira/browse/HDFS-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525040#comment-14525040
]
Hadoop QA commented on HDFS-7174:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12672576/HDFS-7174.new.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10703/console |
This message was automatically generated.
> Support for more efficient large directories
>
>
> Key: HDFS-7174
> URL: https://issues.apache.org/jira/browse/HDFS-7174
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Attachments: HDFS-7174.new.patch, HDFS-7174.patch, HDFS-7174.patch
>
>
> When the number of children under a directory grows very large, insertion
> becomes very costly. E.g. creating 1M entries takes 10s of minutes. This is
> because the complexity of an insertion is O\(n\). As the size of a list
> grows, the overhead grows n^2. (integral of linear function). It also causes
> allocations and copies of big arrays.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7265) Use a throttler for replica write in datanode
[
https://issues.apache.org/jira/browse/HDFS-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525042#comment-14525042
]
Hadoop QA commented on HDFS-7265:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12675706/h7265_20141018.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10705/console |
This message was automatically generated.
> Use a throttler for replica write in datanode
> -
>
> Key: HDFS-7265
> URL: https://issues.apache.org/jira/browse/HDFS-7265
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h7265_20141018.patch
>
>
> BlockReceiver process packets in BlockReceiver.receivePacket() as follows
> # read from socket
> # enqueue the ack
> # write to downstream
> # write to disk
> The above steps is repeated for each packet in a single thread. When there
> are a lot of concurrent writes in a datanode, the write time in #4 becomes
> very long. As a result, it leads to SocketTimeoutException since it cannot
> read from the socket for a long time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6633) Support reading new data in a being written file until the file is closed
[
https://issues.apache.org/jira/browse/HDFS-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525018#comment-14525018
]
Hadoop QA commented on HDFS-6633:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12689437/HDFS-6633-003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10692/console |
This message was automatically generated.
> Support reading new data in a being written file until the file is closed
> -
>
> Key: HDFS-6633
> URL: https://issues.apache.org/jira/browse/HDFS-6633
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Vinayakumar B
> Attachments: HDFS-6633-001.patch, HDFS-6633-002.patch,
> HDFS-6633-003.patch, h6633_20140707.patch, h6633_20140708.patch
>
>
> When a file is being written, the file length keeps increasing. If the file
> is opened for read, the reader first gets the file length and then read only
> up to that length. The reader will not be able to read the new data written
> afterward.
> We propose adding a new feature so that readers will be able to read all the
> data until the writer closes the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7304) TestFileCreation#testOverwriteOpenForWrite hangs
[
https://issues.apache.org/jira/browse/HDFS-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525012#comment-14525012
]
Hadoop QA commented on HDFS-7304:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12680976/HDFS-7304.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10689/console |
This message was automatically generated.
> TestFileCreation#testOverwriteOpenForWrite hangs
>
>
> Key: HDFS-7304
> URL: https://issues.apache.org/jira/browse/HDFS-7304
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Akira AJISAKA
> Attachments: HDFS-7304.patch, HDFS-7304.patch
>
>
> The test case times out. It has been observed in multiple pre-commit builds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7101) Potential null dereference in DFSck#doWork()
[
https://issues.apache.org/jira/browse/HDFS-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525022#comment-14525022
]
Hadoop QA commented on HDFS-7101:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12671192/HDFS-7101_001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10694/console |
This message was automatically generated.
> Potential null dereference in DFSck#doWork()
>
>
> Key: HDFS-7101
> URL: https://issues.apache.org/jira/browse/HDFS-7101
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Ted Yu
>Assignee: skrho
>Priority: Minor
> Attachments: HDFS-7101_001.patch
>
>
> {code}
> String lastLine = null;
> int errCode = -1;
> try {
> while ((line = input.readLine()) != null) {
> ...
> if (lastLine.endsWith(NamenodeFsck.HEALTHY_STATUS)) {
> errCode = 0;
> {code}
> If readLine() throws exception, lastLine may be null, leading to NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7214) Display the time when NN became active on the webUI
[
https://issues.apache.org/jira/browse/HDFS-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525020#comment-14525020
]
Hadoop QA commented on HDFS-7214:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12673689/HDFS-7214.v1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10691/console |
This message was automatically generated.
> Display the time when NN became active on the webUI
> ---
>
> Key: HDFS-7214
> URL: https://issues.apache.org/jira/browse/HDFS-7214
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siqi Li
>Assignee: Siqi Li
> Attachments: HDFS-7214.v1.patch
>
>
> The currently NN webUI displayed JVM start up. It will be useful to show when
> NN became active.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5689) FsDatasetImpl registers mbean using uninitialized DataNode UUID
[
https://issues.apache.org/jira/browse/HDFS-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525013#comment-14525013
]
Hadoop QA commented on HDFS-5689:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12619608/HDFS-5689.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10690/console |
This message was automatically generated.
> FsDatasetImpl registers mbean using uninitialized DataNode UUID
> ---
>
> Key: HDFS-5689
> URL: https://issues.apache.org/jira/browse/HDFS-5689
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: David Powell
>Assignee: David Powell
>Priority: Minor
> Attachments: HDFS-5689.patch
>
>
> FsDatasetImpl's constructor attempts to include the datanode UUID in its
> mbean's ObjectName:
> registerMBean(datanode.getDatanodeUuid());
> Unfortunately this doesn't work because the provided DataNode's UUID isn't
> set until bpRegistrationSucceeded() is called... after the FsDatasetImpl has
> been created. The result is the mbean is always registered with a bogus
> (though valid) ObjectName:
> Hadoop:name=FSDatasetState-null,service=DataNode
> Prior to HDFS-2832 and the storageID -> datanodeUuid rename, this was
> initialized using the DataStorage:
> registerMBean(storage.getStorageID());
> With the fix for HDFS-5454 in place, doing equivalent thing (already done by
> SimulatedFSDataset):
> registerMBean(storage.getDatanodeUuid());
> ...fixes the problem:
>
> Hadoop:name=FSDatasetState-24aed86a-fee6-4b88-868e-285e09ea2766,service=DataNode
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7469) Inline permission checking done by FSPermissionChecker#checkSubAccess
[
https://issues.apache.org/jira/browse/HDFS-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525023#comment-14525023
]
Hadoop QA commented on HDFS-7469:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12686701/HDFS-7469.006.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10695/console |
This message was automatically generated.
> Inline permission checking done by FSPermissionChecker#checkSubAccess
> -
>
> Key: HDFS-7469
> URL: https://issues.apache.org/jira/browse/HDFS-7469
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-7469.000.patch, HDFS-7469.001.patch,
> HDFS-7469.002.patch, HDFS-7469.003.patch, HDFS-7469.004.patch,
> HDFS-7469.005.patch, HDFS-7469.006.patch
>
>
> {{FSPermissionChecker#checkSubAccess}} is used by {{getContentSummary()}} and
> {{computeSnapshotDiff()}} to validate the permission of the entire subtree
> through BFS.
> As these two calls traverse subtrees to compute the result, the permission
> checking should be done during the same traversal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7529) Consolidate encryption zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525019#comment-14525019
]
Hadoop QA commented on HDFS-7529:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12687849/HDFS-7529.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10693/console |
This message was automatically generated.
> Consolidate encryption zone related implementation into a single class
> --
>
> Key: HDFS-7529
> URL: https://issues.apache.org/jira/browse/HDFS-7529
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-7529.000.patch, HDFS-7529.001.patch
>
>
> This jira proposes to consolidate encryption zone related implementation to a
> single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6633) Support reading new data in a being written file until the file is closed
[
https://issues.apache.org/jira/browse/HDFS-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525001#comment-14525001
]
Hadoop QA commented on HDFS-6633:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12689437/HDFS-6633-003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10688/console |
This message was automatically generated.
> Support reading new data in a being written file until the file is closed
> -
>
> Key: HDFS-6633
> URL: https://issues.apache.org/jira/browse/HDFS-6633
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Vinayakumar B
> Attachments: HDFS-6633-001.patch, HDFS-6633-002.patch,
> HDFS-6633-003.patch, h6633_20140707.patch, h6633_20140708.patch
>
>
> When a file is being written, the file length keeps increasing. If the file
> is opened for read, the reader first gets the file length and then read only
> up to that length. The reader will not be able to read the new data written
> afterward.
> We propose adding a new feature so that readers will be able to read all the
> data until the writer closes the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7469) Inline permission checking done by FSPermissionChecker#checkSubAccess
[
https://issues.apache.org/jira/browse/HDFS-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524996#comment-14524996
]
Hadoop QA commented on HDFS-7469:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12686701/HDFS-7469.006.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10686/console |
This message was automatically generated.
> Inline permission checking done by FSPermissionChecker#checkSubAccess
> -
>
> Key: HDFS-7469
> URL: https://issues.apache.org/jira/browse/HDFS-7469
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-7469.000.patch, HDFS-7469.001.patch,
> HDFS-7469.002.patch, HDFS-7469.003.patch, HDFS-7469.004.patch,
> HDFS-7469.005.patch, HDFS-7469.006.patch
>
>
> {{FSPermissionChecker#checkSubAccess}} is used by {{getContentSummary()}} and
> {{computeSnapshotDiff()}} to validate the permission of the entire subtree
> through BFS.
> As these two calls traverse subtrees to compute the result, the permission
> checking should be done during the same traversal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7678) Erasure coding: DFSInputStream with decode functionality
[
https://issues.apache.org/jira/browse/HDFS-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524998#comment-14524998
]
Hadoop QA commented on HDFS-7678:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 38s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 53s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 54s | The patch appears to introduce
10 new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 228m 10s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 17s | Tests passed in
hadoop-hdfs-client. |
| | | 270m 44s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 90% of time
Unsynchronized access at DFSOutputStream.java:90% of time Unsynchronized
access at DFSOutputStream.java:[line 142] |
| | Switch statement found in
org.apache.hadoop.hdfs.DFSStripedInputStream.fetchBlockByteRange(long, long,
long, byte[], int, Map) where default case is missing At
DFSStripedInputStream.java:long, long, byte[], int, Map) where default case is
missing At DFSStripedInputStream.java:[lines 481-502] |
| | Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived
from an Exception, even though it is named as such At DataStreamer.java:from
an Exception, even though it is named as such At DataStreamer.java:[lines
177-201] |
| | Dead store to offSuccess in
org.apache.hadoop.hdfs.StripedDataStreamer.endBlock() At
StripedDataStreamer.java:org.apache.hadoop.hdfs.StripedDataStreamer.endBlock()
At StripedDataStreamer.java:[line 105] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 206] |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.createErasureCodingZone(String,
ECSchema):in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.createErasureCodingZone(String,
ECSchema): String.getBytes() At ErasureCodingZoneManager.java:[line 117] |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.getECZoneInfo(INodesInPath):in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.getECZoneInfo(INodesInPath):
new String(byte[]) At ErasureCodingZoneManager.java:[line 81] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:[line 85] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.planReadPortions(int, int, long,
int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.planReadPortions(int, int, long,
int, int) At StripedBlockUtil.java:[line 167] |
| FindBugs | module:hadoop-hdfs-client |
| | org.apache.hadoop.hdfs.protocol.LocatedStripedBlock.getBlockIndices() may
expose internal representation by returning LocatedStripedBlock.blockIndices
At LocatedStripedBlock.java:by returning LocatedStripedBlock.blockIndices At
LocatedStripedBlock.java:[line 57] |
| Failed unit tests | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestQuota |
| | hadoop.cli.TestHDFSCLI |
| |
[jira] [Commented] (HDFS-7529) Consolidate encryption zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524999#comment-14524999
]
Hadoop QA commented on HDFS-7529:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12687849/HDFS-7529.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10687/console |
This message was automatically generated.
> Consolidate encryption zone related implementation into a single class
> --
>
> Key: HDFS-7529
> URL: https://issues.apache.org/jira/browse/HDFS-7529
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-7529.000.patch, HDFS-7529.001.patch
>
>
> This jira proposes to consolidate encryption zone related implementation to a
> single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5757) refreshNodes with many nodes at the same time could slow down NN
[
https://issues.apache.org/jira/browse/HDFS-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524995#comment-14524995
]
Hadoop QA commented on HDFS-5757:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12685508/HDFS-5757.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10685/console |
This message was automatically generated.
> refreshNodes with many nodes at the same time could slow down NN
>
>
> Key: HDFS-5757
> URL: https://issues.apache.org/jira/browse/HDFS-5757
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-5757.patch
>
>
> Sometimes we need to decomm a whole rack of nodes at the same time. When the
> decomm is in process; NN is slow.
> The reason is when DecommissionManager checks the decomm status, it acquires
> namesystem's writer lock and iterates through all DNs; for each DN that is in
> decommissioning state, it check if replication is done for all the blocks on
> the machine via blockManager.isReplicationInProgress; for large cluster; the
> number of blocks on the machine could be big.
> The fix could be to have DecommissionManager check for several
> decomm-in-progress nodes each time it aquires namesystem's writer lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7401) Add block info to DFSInputStream' WARN message when it adds node to deadNodes
[
https://issues.apache.org/jira/browse/HDFS-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524988#comment-14524988
]
Hadoop QA commented on HDFS-7401:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12681956/HDFS-7401.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10683/console |
This message was automatically generated.
> Add block info to DFSInputStream' WARN message when it adds node to deadNodes
> -
>
> Key: HDFS-7401
> URL: https://issues.apache.org/jira/browse/HDFS-7401
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Keith Pak
>Priority: Minor
> Attachments: HDFS-7401.patch
>
>
> Block info is missing in the below message
> {noformat}
> 2014-11-14 03:59:00,386 WARN org.apache.hadoop.hdfs.DFSClient: Failed to
> connect to /xx.xx.xx.xxx:50010 for block, add to deadNodes and continue.
> java.io.IOException: Got error for OP_READ_BLOCK
> {noformat}
> The code
> {noformat}
> DFSInputStream.java
> DFSClient.LOG.warn("Failed to connect to " + targetAddr + " for
> block"
> + ", add to deadNodes and continue. " + ex, ex);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7485) Avoid string operations in FSPermissionChecker
[
https://issues.apache.org/jira/browse/HDFS-7485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524994#comment-14524994
]
Hadoop QA commented on HDFS-7485:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12685507/HDFS-7485.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10684/console |
This message was automatically generated.
> Avoid string operations in FSPermissionChecker
> --
>
> Key: HDFS-7485
> URL: https://issues.apache.org/jira/browse/HDFS-7485
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-7485.000.patch
>
>
> Currently {{FSPermissionChecker}} compares strings when testing users and
> groups. It should compare the id assigned by SerialNumberManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7304) TestFileCreation#testOverwriteOpenForWrite hangs
[
https://issues.apache.org/jira/browse/HDFS-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524986#comment-14524986
]
Hadoop QA commented on HDFS-7304:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12680976/HDFS-7304.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10682/console |
This message was automatically generated.
> TestFileCreation#testOverwriteOpenForWrite hangs
>
>
> Key: HDFS-7304
> URL: https://issues.apache.org/jira/browse/HDFS-7304
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Akira AJISAKA
> Attachments: HDFS-7304.patch, HDFS-7304.patch
>
>
> The test case times out. It has been observed in multiple pre-commit builds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6660) Use int instead of object reference to DatanodeStorageInfo in BlockInfo triplets,
[
https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524976#comment-14524976
]
Hadoop QA commented on HDFS-6660:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12666461/HDFS-6660.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10681/console |
This message was automatically generated.
> Use int instead of object reference to DatanodeStorageInfo in BlockInfo
> triplets,
> -
>
> Key: HDFS-6660
> URL: https://issues.apache.org/jira/browse/HDFS-6660
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.4.1
>Reporter: Amir Langer
>Assignee: Amir Langer
> Labels: performance
> Attachments: HDFS-6660.patch
>
>
> Map an int index to every DatanodeStorageInfo and use it instead of object
> reference in the BlockInfo triplets data structure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7264) Tha last datanode in a pipeline should send a heartbeat when there is no traffic
[
https://issues.apache.org/jira/browse/HDFS-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524974#comment-14524974
]
Hadoop QA commented on HDFS-7264:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12675891/h7264_20141020.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10679/console |
This message was automatically generated.
> Tha last datanode in a pipeline should send a heartbeat when there is no
> traffic
>
>
> Key: HDFS-7264
> URL: https://issues.apache.org/jira/browse/HDFS-7264
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h7264_20141017.patch, h7264_20141020.patch
>
>
> When the client is writing slowly, the client will send a heartbeat to signal
> that the connection is still alive. This case works fine.
> However, when a client is writing fast but some of the datanodes in the
> pipeline are busy, a PacketResponder may get a timeout since no ack is sent
> from the upstream datanode. We suggest that the last datanode in a pipeline
> should send a heartbeat when there is no traffic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7214) Display the time when NN became active on the webUI
[
https://issues.apache.org/jira/browse/HDFS-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524968#comment-14524968
]
Hadoop QA commented on HDFS-7214:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12673689/HDFS-7214.v1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10675/console |
This message was automatically generated.
> Display the time when NN became active on the webUI
> ---
>
> Key: HDFS-7214
> URL: https://issues.apache.org/jira/browse/HDFS-7214
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siqi Li
>Assignee: Siqi Li
> Attachments: HDFS-7214.v1.patch
>
>
> The currently NN webUI displayed JVM start up. It will be useful to show when
> NN became active.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7174) Support for more efficient large directories
[
https://issues.apache.org/jira/browse/HDFS-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524970#comment-14524970
]
Hadoop QA commented on HDFS-7174:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12672576/HDFS-7174.new.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10676/console |
This message was automatically generated.
> Support for more efficient large directories
>
>
> Key: HDFS-7174
> URL: https://issues.apache.org/jira/browse/HDFS-7174
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Attachments: HDFS-7174.new.patch, HDFS-7174.patch, HDFS-7174.patch
>
>
> When the number of children under a directory grows very large, insertion
> becomes very costly. E.g. creating 1M entries takes 10s of minutes. This is
> because the complexity of an insertion is O\(n\). As the size of a list
> grows, the overhead grows n^2. (integral of linear function). It also causes
> allocations and copies of big arrays.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[
https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524973#comment-14524973
]
Hadoop QA commented on HDFS-6914:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12671201/HDFS-6914.v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10678/console |
This message was automatically generated.
> Resolve huge memory consumption Issue with OIV processing PB-based fsimages
> ---
>
> Key: HDFS-6914
> URL: https://issues.apache.org/jira/browse/HDFS-6914
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Hao Chen
> Labels: hdfs
> Attachments: HDFS-6914.patch, HDFS-6914.v2.patch
>
>
> For better managing and supporting a lot of large hadoop clusters in
> production, we internally need to automatically export fsimage to delimited
> text files in LSR style and then analyse with hive or pig or build system
> metrics for real-time analyzing.
> However due to the internal layout changes introduced by the protobuf-based
> fsimage, OIV processing program consumes excessive amount of memory. For
> example, in order to export the fsimage in size of 8GB, it should have taken
> about 85GB memory which is really not reasonable and impacted performance of
> other services badly in the same server.
> To resolve above problem, I submit this patch which will reduce memory
> consumption of OIV LSR processing by 50%.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5952) Create a tool to run data analysis on the PB format fsimage
[
https://issues.apache.org/jira/browse/HDFS-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524971#comment-14524971
]
Hadoop QA commented on HDFS-5952:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12663638/HDFS-5952.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10677/console |
This message was automatically generated.
> Create a tool to run data analysis on the PB format fsimage
> ---
>
> Key: HDFS-5952
> URL: https://issues.apache.org/jira/browse/HDFS-5952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.6.0
>Reporter: Akira AJISAKA
> Attachments: HDFS-5952.patch
>
>
> Delimited processor in OfflineImageViewer is not supported after HDFS-5698
> was merged.
> The motivation of delimited processor is to run data analysis on the fsimage,
> therefore, there might be more values to create a tool for Hive or Pig that
> reads the PB format fsimage directly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7265) Use a throttler for replica write in datanode
[
https://issues.apache.org/jira/browse/HDFS-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524975#comment-14524975
]
Hadoop QA commented on HDFS-7265:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12675706/h7265_20141018.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10680/console |
This message was automatically generated.
> Use a throttler for replica write in datanode
> -
>
> Key: HDFS-7265
> URL: https://issues.apache.org/jira/browse/HDFS-7265
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h7265_20141018.patch
>
>
> BlockReceiver process packets in BlockReceiver.receivePacket() as follows
> # read from socket
> # enqueue the ack
> # write to downstream
> # write to disk
> The above steps is repeated for each packet in a single thread. When there
> are a lot of concurrent writes in a datanode, the write time in #4 becomes
> very long. As a result, it leads to SocketTimeoutException since it cannot
> read from the socket for a long time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524967#comment-14524967
]
Hadoop QA commented on HDFS-6980:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12665841/HDFS-6980.1-2.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10674/console |
This message was automatically generated.
> TestWebHdfsFileSystemContract fails in trunk
>
>
> Key: HDFS-6980
> URL: https://issues.apache.org/jira/browse/HDFS-6980
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Tsuyoshi Ozawa
> Attachments: HDFS-6980.1-2.patch, HDFS-6980.1.patch
>
>
> Many tests in TestWebHdfsFileSystemContract fail by "too many open files"
> error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5689) FsDatasetImpl registers mbean using uninitialized DataNode UUID
[
https://issues.apache.org/jira/browse/HDFS-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524963#comment-14524963
]
Hadoop QA commented on HDFS-5689:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12619608/HDFS-5689.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10671/console |
This message was automatically generated.
> FsDatasetImpl registers mbean using uninitialized DataNode UUID
> ---
>
> Key: HDFS-5689
> URL: https://issues.apache.org/jira/browse/HDFS-5689
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: David Powell
>Assignee: David Powell
>Priority: Minor
> Attachments: HDFS-5689.patch
>
>
> FsDatasetImpl's constructor attempts to include the datanode UUID in its
> mbean's ObjectName:
> registerMBean(datanode.getDatanodeUuid());
> Unfortunately this doesn't work because the provided DataNode's UUID isn't
> set until bpRegistrationSucceeded() is called... after the FsDatasetImpl has
> been created. The result is the mbean is always registered with a bogus
> (though valid) ObjectName:
> Hadoop:name=FSDatasetState-null,service=DataNode
> Prior to HDFS-2832 and the storageID -> datanodeUuid rename, this was
> initialized using the DataStorage:
> registerMBean(storage.getStorageID());
> With the fix for HDFS-5454 in place, doing equivalent thing (already done by
> SimulatedFSDataset):
> registerMBean(storage.getDatanodeUuid());
> ...fixes the problem:
>
> Hadoop:name=FSDatasetState-24aed86a-fee6-4b88-868e-285e09ea2766,service=DataNode
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6066) logGenerationStamp is not needed to reduce editlog size
[
https://issues.apache.org/jira/browse/HDFS-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524959#comment-14524959
]
Hadoop QA commented on HDFS-6066:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12633573/HDFS-6066-trunk.3.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10670/console |
This message was automatically generated.
> logGenerationStamp is not needed to reduce editlog size
> ---
>
> Key: HDFS-6066
> URL: https://issues.apache.org/jira/browse/HDFS-6066
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: chenping
>Priority: Minor
> Attachments: HDFS-6066-trunk.1.patch, HDFS-6066-trunk.2.patch,
> HDFS-6066-trunk.3.patch
>
>
> almost every logGenerationStamp has a logAddBlock followed, so we can get the
> newest gs from the logAddBlock operation indirectly.this will reduce the edit
> log size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524957#comment-14524957
]
Hadoop QA commented on HDFS-5274:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660003/HDFS-5274-17.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10669/console |
This message was automatically generated.
> Add Tracing to HDFS
> ---
>
> Key: HDFS-5274
> URL: https://issues.apache.org/jira/browse/HDFS-5274
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
> 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
> HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
> HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch,
> HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch,
> HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch,
> HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png,
> Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
>
>
> Since Google's Dapper paper has shown the benefits of tracing for a large
> distributed system, it seems like a good time to add tracing to HDFS. HBase
> has added tracing using HTrace. I propose that the same can be done within
> HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7101) Potential null dereference in DFSck#doWork()
[
https://issues.apache.org/jira/browse/HDFS-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524965#comment-14524965
]
Hadoop QA commented on HDFS-7101:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12671192/HDFS-7101_001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10673/console |
This message was automatically generated.
> Potential null dereference in DFSck#doWork()
>
>
> Key: HDFS-7101
> URL: https://issues.apache.org/jira/browse/HDFS-7101
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Ted Yu
>Assignee: skrho
>Priority: Minor
> Attachments: HDFS-7101_001.patch
>
>
> {code}
> String lastLine = null;
> int errCode = -1;
> try {
> while ((line = input.readLine()) != null) {
> ...
> if (lastLine.endsWith(NamenodeFsck.HEALTHY_STATUS)) {
> errCode = 0;
> {code}
> If readLine() throws exception, lastLine may be null, leading to NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7123) Run legacy fsimage checkpoint in parallel with PB fsimage checkpoint
[
https://issues.apache.org/jira/browse/HDFS-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524964#comment-14524964
]
Hadoop QA commented on HDFS-7123:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12670479/HDFS-7123.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10672/console |
This message was automatically generated.
> Run legacy fsimage checkpoint in parallel with PB fsimage checkpoint
>
>
> Key: HDFS-7123
> URL: https://issues.apache.org/jira/browse/HDFS-7123
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-7123.patch
>
>
> HDFS-7097 will address the checkpoint and BR issue. In addition, it might
> still be useful to reduce the overall checkpoint duration, given it blocks
> edit log replay. If there is large volume of edit log to catch up and NN fail
> overs, it will impact the availability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4631) Support customized call back method during failover automatically.
[
https://issues.apache.org/jira/browse/HDFS-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524954#comment-14524954
]
Hadoop QA commented on HDFS-4631:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10668/console |
This message was automatically generated.
> Support customized call back method during failover automatically.
> --
>
> Key: HDFS-4631
> URL: https://issues.apache.org/jira/browse/HDFS-4631
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0
>Reporter: Fengdong Yu
> Labels: features, ha, hadoop
> Attachments: HDFS-4631.patch
>
> Original Estimate: 0.5m
> Remaining Estimate: 0.5m
>
> ZKFC add HealthCallbacks by default, this can do quiteElection at least. but
> we often want to be alerted if there is fail over occurring(such as send
> email, short messages), especially for prod cluster.
> There is a configured fence script. maybe we can put all these logics in the
> script. but in reasonable, fence script does only one thing: fence :)
> So I added this patch, we can configure customized HM callback method, if
> there is no configuration, then only HealthCallbacks is added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6436) WebHdfsFileSystem execute get, renew and cancel delegationtoken operation should use spnego to authenticate
[
https://issues.apache.org/jira/browse/HDFS-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524952#comment-14524952
]
Hadoop QA commented on HDFS-6436:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12645782/HDFS-6436.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10667/console |
This message was automatically generated.
> WebHdfsFileSystem execute get, renew and cancel delegationtoken operation
> should use spnego to authenticate
> ---
>
> Key: HDFS-6436
> URL: https://issues.apache.org/jira/browse/HDFS-6436
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0, 2.4.0
> Environment: Kerberos
>Reporter: Bangtao Zhou
> Attachments: HDFS-6436.patch
>
>
> while in kerberos secure mode, when using WebHdfsFileSystem to access HDFS,
> it allways get an
> *org.apache.hadoop.security.authentication.client.AuthenticationException:
> Unauthorized*, for example, when call WebHdfsFileSystem.listStatus it will
> execute a LISTSTATUS Op, and this Op should authenticate via *delegation
> token*, so it will execute a GETDELEGATIONTOKEN Op to get a delegation
> token(actually GETDELEGATIONTOKEN authenticates via *SPNEGO*), but it still
> use delegation token to authenticate, so it allways get an Unauthorized
> Exception.
> Exception is like this:
> {code:java}
> 19:05:11.758 [main] DEBUG o.a.h.hdfs.web.URLConnectionFactory - open URL
> connection
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Unauthorized
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:287)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:82)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:538)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:406)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:434)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:430)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1058)
> 19:05:11.766 [main] DEBUG o.a.h.security.UserGroupInformation -
> PrivilegedActionException as:bang...@cyhadoop.com (auth:KERBEROS)
> cause:java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Unauthorized
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:134)
> 19:05:11.767 [main] DEBUG o.a.h.security.UserGroupInformation -
> PrivilegedActionException as:bang...@cyhadoop.com (auth:KERBEROS)
> cause:java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Unauthorized
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:213)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:371)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:392)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:602)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:533)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:406)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:434)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:430)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1037)
> at org.apache.had
[jira] [Commented] (HDFS-6022) Moving deadNodes from being thread local. Improving dead datanode handling in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524942#comment-14524942
]
Hadoop QA commented on HDFS-6022:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12631558/HADOOP-6022.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10665/console |
This message was automatically generated.
> Moving deadNodes from being thread local. Improving dead datanode handling in
> DFSClient
>
>
> Key: HDFS-6022
> URL: https://issues.apache.org/jira/browse/HDFS-6022
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0, 0.23.9, 0.23.10, 2.2.0, 2.3.0
>Reporter: Jack Levin
>Assignee: Colin Patrick McCabe
> Labels: patch
> Attachments: HADOOP-6022.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> This patch solves an issue of deadNodes list being thread local. deadNodes
> list is created by DFSClient when some problems with write/reading, or
> contacting datanode exist. The problem is that deadNodes is not visible to
> other DFSInputStream threads, hence every DFSInputStream ends up building its
> own deadNodes. This affect performance of DFSClient to a large degree
> especially when a datanode goes completely offline (there is a tcp connect
> delay experienced by all DFSInputStream threads affecting performance of the
> whole cluster).
> This patch moves deadNodes to be global in DFSClient class so that as soon as
> a single DFSInputStream thread reports a dead datanode, all other
> DFSInputStream threads are informed, negating the need to create their own
> independent lists (concurrent Map really).
> Further, a global deadNodes health check manager thread (DeadNodeVerifier) is
> created to verify all dead datanodes every 5 seconds, and remove the same
> list as soon as it is up. That thread under normal conditions (deadNodes
> empty) would be sleeping. If deadNodes is not empty, the thread will attempt
> to open tcp connection every 5 seconds to affected datanodes.
> This patch has a test (TestDFSClientDeadNodes) that is quite simple, since
> the deadNodes creation is not affected by the patch, we only test datanode
> removal from deadNodes by the health check manager thread. Test will create
> a file in dfs minicluster, read from the same file rapidly, cause datanode to
> restart, and test is the health check manager thread does the right thing,
> removing the alive datanode from the global deadNodes list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6867) For DFSOutputStream, do pipeline recovery for a single block in the background
[
https://issues.apache.org/jira/browse/HDFS-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524943#comment-14524943
]
Hadoop QA commented on HDFS-6867:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12669530/HDFS-6867-20140917-v2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10666/console |
This message was automatically generated.
> For DFSOutputStream, do pipeline recovery for a single block in the background
> --
>
> Key: HDFS-6867
> URL: https://issues.apache.org/jira/browse/HDFS-6867
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Colin Patrick McCabe
>Assignee: Zhe Zhang
> Attachments: HDFS-6867-20140827-2.patch, HDFS-6867-20140827-3.patch,
> HDFS-6867-20140827.patch, HDFS-6867-20140828-1.patch,
> HDFS-6867-20140828-2.patch, HDFS-6867-20140910.patch,
> HDFS-6867-20140911.patch, HDFS-6867-20140917-v1.patch,
> HDFS-6867-20140917-v2.patch, HDFS-6867-design-20140820.pdf,
> HDFS-6867-design-20140821.pdf, HDFS-6867-design-20140822.pdf,
> HDFS-6867-design-20140827.pdf, HDFS-6867-design-20140910.pdf
>
>
> For DFSOutputStream, we should be able to do pipeline recovery in the
> background, while the user is continuing to write to the file. This is
> especially useful for long-lived clients that write to an HDFS file slowly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6867) For DFSOutputStream, do pipeline recovery for a single block in the background
[
https://issues.apache.org/jira/browse/HDFS-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524937#comment-14524937
]
Hadoop QA commented on HDFS-6867:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12669530/HDFS-6867-20140917-v2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10664/console |
This message was automatically generated.
> For DFSOutputStream, do pipeline recovery for a single block in the background
> --
>
> Key: HDFS-6867
> URL: https://issues.apache.org/jira/browse/HDFS-6867
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Colin Patrick McCabe
>Assignee: Zhe Zhang
> Attachments: HDFS-6867-20140827-2.patch, HDFS-6867-20140827-3.patch,
> HDFS-6867-20140827.patch, HDFS-6867-20140828-1.patch,
> HDFS-6867-20140828-2.patch, HDFS-6867-20140910.patch,
> HDFS-6867-20140911.patch, HDFS-6867-20140917-v1.patch,
> HDFS-6867-20140917-v2.patch, HDFS-6867-design-20140820.pdf,
> HDFS-6867-design-20140821.pdf, HDFS-6867-design-20140822.pdf,
> HDFS-6867-design-20140827.pdf, HDFS-6867-design-20140910.pdf
>
>
> For DFSOutputStream, we should be able to do pipeline recovery in the
> background, while the user is continuing to write to the file. This is
> especially useful for long-lived clients that write to an HDFS file slowly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6436) WebHdfsFileSystem execute get, renew and cancel delegationtoken operation should use spnego to authenticate
[
https://issues.apache.org/jira/browse/HDFS-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524933#comment-14524933
]
Hadoop QA commented on HDFS-6436:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12645782/HDFS-6436.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10662/console |
This message was automatically generated.
> WebHdfsFileSystem execute get, renew and cancel delegationtoken operation
> should use spnego to authenticate
> ---
>
> Key: HDFS-6436
> URL: https://issues.apache.org/jira/browse/HDFS-6436
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0, 2.4.0
> Environment: Kerberos
>Reporter: Bangtao Zhou
> Attachments: HDFS-6436.patch
>
>
> while in kerberos secure mode, when using WebHdfsFileSystem to access HDFS,
> it allways get an
> *org.apache.hadoop.security.authentication.client.AuthenticationException:
> Unauthorized*, for example, when call WebHdfsFileSystem.listStatus it will
> execute a LISTSTATUS Op, and this Op should authenticate via *delegation
> token*, so it will execute a GETDELEGATIONTOKEN Op to get a delegation
> token(actually GETDELEGATIONTOKEN authenticates via *SPNEGO*), but it still
> use delegation token to authenticate, so it allways get an Unauthorized
> Exception.
> Exception is like this:
> {code:java}
> 19:05:11.758 [main] DEBUG o.a.h.hdfs.web.URLConnectionFactory - open URL
> connection
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Unauthorized
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:287)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:82)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:538)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:406)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:434)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:430)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1058)
> 19:05:11.766 [main] DEBUG o.a.h.security.UserGroupInformation -
> PrivilegedActionException as:bang...@cyhadoop.com (auth:KERBEROS)
> cause:java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Unauthorized
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:134)
> 19:05:11.767 [main] DEBUG o.a.h.security.UserGroupInformation -
> PrivilegedActionException as:bang...@cyhadoop.com (auth:KERBEROS)
> cause:java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Unauthorized
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:213)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:371)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:392)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:602)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:533)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:406)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:434)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:430)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1037)
> at org.apache.had
[jira] [Commented] (HDFS-4631) Support customized call back method during failover automatically.
[
https://issues.apache.org/jira/browse/HDFS-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524930#comment-14524930
]
Hadoop QA commented on HDFS-4631:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10660/console |
This message was automatically generated.
> Support customized call back method during failover automatically.
> --
>
> Key: HDFS-4631
> URL: https://issues.apache.org/jira/browse/HDFS-4631
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0
>Reporter: Fengdong Yu
> Labels: features, ha, hadoop
> Attachments: HDFS-4631.patch
>
> Original Estimate: 0.5m
> Remaining Estimate: 0.5m
>
> ZKFC add HealthCallbacks by default, this can do quiteElection at least. but
> we often want to be alerted if there is fail over occurring(such as send
> email, short messages), especially for prod cluster.
> There is a configured fence script. maybe we can put all these logics in the
> script. but in reasonable, fence script does only one thing: fence :)
> So I added this patch, we can configure customized HM callback method, if
> there is no configuration, then only HealthCallbacks is added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6066) logGenerationStamp is not needed to reduce editlog size
[
https://issues.apache.org/jira/browse/HDFS-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524931#comment-14524931
]
Hadoop QA commented on HDFS-6066:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12633573/HDFS-6066-trunk.3.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10661/console |
This message was automatically generated.
> logGenerationStamp is not needed to reduce editlog size
> ---
>
> Key: HDFS-6066
> URL: https://issues.apache.org/jira/browse/HDFS-6066
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: chenping
>Priority: Minor
> Attachments: HDFS-6066-trunk.1.patch, HDFS-6066-trunk.2.patch,
> HDFS-6066-trunk.3.patch
>
>
> almost every logGenerationStamp has a logAddBlock followed, so we can get the
> newest gs from the logAddBlock operation indirectly.this will reduce the edit
> log size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6022) Moving deadNodes from being thread local. Improving dead datanode handling in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524925#comment-14524925
]
Hadoop QA commented on HDFS-6022:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12631558/HADOOP-6022.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10659/console |
This message was automatically generated.
> Moving deadNodes from being thread local. Improving dead datanode handling in
> DFSClient
>
>
> Key: HDFS-6022
> URL: https://issues.apache.org/jira/browse/HDFS-6022
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0, 0.23.9, 0.23.10, 2.2.0, 2.3.0
>Reporter: Jack Levin
>Assignee: Colin Patrick McCabe
> Labels: patch
> Attachments: HADOOP-6022.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> This patch solves an issue of deadNodes list being thread local. deadNodes
> list is created by DFSClient when some problems with write/reading, or
> contacting datanode exist. The problem is that deadNodes is not visible to
> other DFSInputStream threads, hence every DFSInputStream ends up building its
> own deadNodes. This affect performance of DFSClient to a large degree
> especially when a datanode goes completely offline (there is a tcp connect
> delay experienced by all DFSInputStream threads affecting performance of the
> whole cluster).
> This patch moves deadNodes to be global in DFSClient class so that as soon as
> a single DFSInputStream thread reports a dead datanode, all other
> DFSInputStream threads are informed, negating the need to create their own
> independent lists (concurrent Map really).
> Further, a global deadNodes health check manager thread (DeadNodeVerifier) is
> created to verify all dead datanodes every 5 seconds, and remove the same
> list as soon as it is up. That thread under normal conditions (deadNodes
> empty) would be sleeping. If deadNodes is not empty, the thread will attempt
> to open tcp connection every 5 seconds to affected datanodes.
> This patch has a test (TestDFSClientDeadNodes) that is quite simple, since
> the deadNodes creation is not affected by the patch, we only test datanode
> removal from deadNodes by the health check manager thread. Test will create
> a file in dfs minicluster, read from the same file rapidly, cause datanode to
> restart, and test is the health check manager thread does the right thing,
> removing the alive datanode from the global deadNodes list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524924#comment-14524924
]
Hadoop QA commented on HDFS-5274:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660003/HDFS-5274-17.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10658/console |
This message was automatically generated.
> Add Tracing to HDFS
> ---
>
> Key: HDFS-5274
> URL: https://issues.apache.org/jira/browse/HDFS-5274
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
> 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
> HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
> HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch,
> HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch,
> HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch,
> HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png,
> Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
>
>
> Since Google's Dapper paper has shown the benefits of tracing for a large
> distributed system, it seems like a good time to add tracing to HDFS. HBase
> has added tracing using HTrace. I propose that the same can be done within
> HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5887) Add suffix to generated protobuf class
[
https://issues.apache.org/jira/browse/HDFS-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524918#comment-14524918
]
Hadoop QA commented on HDFS-5887:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12661725/HDFS-5887.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10657/console |
This message was automatically generated.
> Add suffix to generated protobuf class
> --
>
> Key: HDFS-5887
> URL: https://issues.apache.org/jira/browse/HDFS-5887
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-5698 (FSImage in protobuf)
>Reporter: Haohui Mai
>Assignee: Tassapol Athiapinya
>Priority: Minor
> Attachments: HDFS-5887.000.patch,
> HDFS-5887.000.proto_files-only.patch, HDFS-5887.001.patch
>
>
> As suggested by [~tlipcon], the code is more readable if we give each class
> generated by the protobuf the suffix "Proto".
> This jira proposes to rename the classes and to introduce no functionality
> changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6860) BlockStateChange logs are too noisy
[
https://issues.apache.org/jira/browse/HDFS-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524909#comment-14524909
]
Hadoop QA commented on HDFS-6860:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12665129/HDFS6860.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10655/console |
This message was automatically generated.
> BlockStateChange logs are too noisy
> ---
>
> Key: HDFS-6860
> URL: https://issues.apache.org/jira/browse/HDFS-6860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Chang Li
> Labels: newbie
> Attachments: HDFS6860.patch, HDFS6860.patch
>
>
> Block State Change logs are too noisy at the default INFO level and affect NN
> performance on busy clusters.
> Most of these state changes can be logged at debug level instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6526) Implement HDFS TtlManager
[
https://issues.apache.org/jira/browse/HDFS-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524917#comment-14524917
]
Hadoop QA commented on HDFS-6526:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12651960/HDFS-6526.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10656/console |
This message was automatically generated.
> Implement HDFS TtlManager
> -
>
> Key: HDFS-6526
> URL: https://issues.apache.org/jira/browse/HDFS-6526
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6526.1.patch
>
>
> This issue is used to track development of HDFS TtlManager, for details see
> HDFS-6382.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6262) HDFS doesn't raise FileNotFoundException if the source of a rename() is missing
[
https://issues.apache.org/jira/browse/HDFS-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524906#comment-14524906
]
Hadoop QA commented on HDFS-6262:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12641192/HDFS-6262.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10654/console |
This message was automatically generated.
> HDFS doesn't raise FileNotFoundException if the source of a rename() is
> missing
> ---
>
> Key: HDFS-6262
> URL: https://issues.apache.org/jira/browse/HDFS-6262
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>Assignee: Akira AJISAKA
> Attachments: HDFS-6262.2.patch, HDFS-6262.patch
>
>
> HDFS's {{rename(src, dest)}} returns false if src does not exist -all the
> other filesystems raise {{FileNotFoundException}}
> This behaviour is defined in {{FSDirectory.unprotectedRenameTo()}} -the
> attempt is logged, but the operation then just returns false.
> I propose changing the behaviour of {{DistributedFileSystem}} to be the same
> as that of the others -and of {{FileContext}}, which does reject renames with
> nonexistent sources
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4504) DFSOutputStream#close doesn't always release resources (such as leases)
[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524893#comment-14524893
]
Hadoop QA commented on HDFS-4504:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12599081/HDFS-4504.016.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10648/console |
This message was automatically generated.
> DFSOutputStream#close doesn't always release resources (such as leases)
> ---
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch,
> HDFS-4504.007.patch, HDFS-4504.008.patch, HDFS-4504.009.patch,
> HDFS-4504.010.patch, HDFS-4504.011.patch, HDFS-4504.014.patch,
> HDFS-4504.015.patch, HDFS-4504.016.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5464) Simplify block report diff calculation
[
https://issues.apache.org/jira/browse/HDFS-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524903#comment-14524903
]
Hadoop QA commented on HDFS-5464:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12655886/h5464_20140715b.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10653/console |
This message was automatically generated.
> Simplify block report diff calculation
> --
>
> Key: HDFS-5464
> URL: https://issues.apache.org/jira/browse/HDFS-5464
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h5464_20131105.patch, h5464_20131105b.patch,
> h5464_20131105c.patch, h5464_20140715.patch, h5464_20140715b.patch
>
>
> The current calculation in BlockManager.reportDiff(..) is unnecessarily
> complicated. We could simplify the calculation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[
https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524894#comment-14524894
]
Hadoop QA commented on HDFS-4754:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12596540/4754.v4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10647/console |
This message was automatically generated.
> Add an API in the namenode to mark a datanode as stale
> --
>
> Key: HDFS-4754
> URL: https://issues.apache.org/jira/browse/HDFS-4754
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Critical
> Attachments: 4754.v1.patch, 4754.v2.patch, 4754.v4.patch,
> 4754.v4.patch
>
>
> There is a detection of the stale datanodes in HDFS since HDFS-3703, with a
> timeout, defaulted to 30s.
> There are two reasons to add an API to mark a node as stale even if the
> timeout is not yet reached:
> 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase,
> we sometimes start the recovery before a node is marked staled. (even with
> reasonable settings as: stale: 20s; HBase ZK timeout: 30s
> 2) Some third parties could detect that a node is dead before the timeout,
> hence saving us the cost of retrying. An example or such hw is Arista,
> presented here by [~tsuna]
> http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in
> HBASE-6290.
> As usual, even if the node is dead it can comeback before the 10 minutes
> limit. So I would propose to set a timebound. The API would be
> namenode.markStale(String ipAddress, int port, long durationInMs);
> After durationInMs, the namenode would again rely only on its heartbeat to
> decide.
> Thoughts?
> If there is no objections, and if nobody in the hdfs dev team has the time to
> spend some time on it, I will give it a try for branch 2 & 3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6525) FsShell supports HDFS TTL
[
https://issues.apache.org/jira/browse/HDFS-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524902#comment-14524902
]
Hadoop QA commented on HDFS-6525:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12652123/HDFS-6525.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10652/console |
This message was automatically generated.
> FsShell supports HDFS TTL
> -
>
> Key: HDFS-6525
> URL: https://issues.apache.org/jira/browse/HDFS-6525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, tools
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6525.1.patch, HDFS-6525.2.patch
>
>
> This issue is used to track development of supporting HDFS TTL for FsShell,
> for details see HDFS-6382.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6861) Separate Balancer specific logic form Dispatcher
[
https://issues.apache.org/jira/browse/HDFS-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524900#comment-14524900
]
Hadoop QA commented on HDFS-6861:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12662699/h6861_20140819.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10650/console |
This message was automatically generated.
> Separate Balancer specific logic form Dispatcher
>
>
> Key: HDFS-6861
> URL: https://issues.apache.org/jira/browse/HDFS-6861
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h6861_20140818.patch, h6861_20140819.patch
>
>
> In order to balance datanode storage utilization of a cluster, Balancer (1)
> classifies datanodes into different groups (overUtilized, aboveAvgUtilized,
> belowAvgUtilized and underUtilized), (2) chooses source and target datanode
> pairs and (3) chooses blocks to move. Some of these logic are in Dispatcher.
> It is better to separate them out. This JIRA is a further work of HDFS-6828.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6813) WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable with thead-safe.
[
https://issues.apache.org/jira/browse/HDFS-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524889#comment-14524889
]
Hadoop QA commented on HDFS-6813:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12659533/HDFS-6813.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10645/console |
This message was automatically generated.
> WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable
> with thead-safe.
> ---
>
> Key: HDFS-6813
> URL: https://issues.apache.org/jira/browse/HDFS-6813
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6813.001.patch
>
>
> {{PositionedReadable}} definition requires the implementations for its
> interfaces should be thread-safe.
> OffsetUrlInputStream(WebHdfsFileSystem inputstream) doesn't implement these
> interfaces with tread-safe, this JIRA is to fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524891#comment-14524891
]
Hadoop QA commented on HDFS-5730:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12623677/HDFS-5730.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10646/console |
This message was automatically generated.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-5730.patch, HDFS-5730.patch
>
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4916) DataTransfer may mask the IOException during block transfering
[
https://issues.apache.org/jira/browse/HDFS-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524896#comment-14524896
]
Hadoop QA commented on HDFS-4916:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12588510/4916.v0.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10649/console |
This message was automatically generated.
> DataTransfer may mask the IOException during block transfering
> --
>
> Key: HDFS-4916
> URL: https://issues.apache.org/jira/browse/HDFS-4916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.0.4-alpha, 2.0.5-alpha
>Reporter: Zesheng Wu
>Priority: Critical
> Attachments: 4916.v0.patch
>
>
> When a new datanode is added to the pipeline, the client will trigger the
> block transfer process. In the current implementation, the src datanode calls
> the run() method of the DataTransfer to transfer the block, this method will
> mask the IOExceptions during the transfering, and will make the client not
> realize the failure during the transferring, as a result the client will
> mistake the failing transferring as successful one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6588) Investigating removing getTrueCause method in Server.java
[
https://issues.apache.org/jira/browse/HDFS-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524901#comment-14524901
]
Hadoop QA commented on HDFS-6588:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12657013/HDFS-6588.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10651/console |
This message was automatically generated.
> Investigating removing getTrueCause method in Server.java
> -
>
> Key: HDFS-6588
> URL: https://issues.apache.org/jira/browse/HDFS-6588
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, webhdfs
>Affects Versions: 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6588.001.patch, HDFS-6588.001.patch,
> HDFS-6588.001.patch, HDFS-6588.001.patch
>
>
> When addressing Daryn Sharp's comment for HDFS-6475 quoted below:
> {quote}
> What I'm saying is I think the patch adds too much unnecessary code. Filing
> an improvement to delete all but a few lines of the code changed in this
> patch seems a bit odd. I think you just need to:
> - Delete getTrueCause entirely instead of moving it elsewhere
> - In saslProcess, just throw the exception instead of running it through
> getTrueCause since it's not a "InvalidToken wrapping another exception"
> anymore.
> - Keep your 3-line change to unwrap SecurityException in toResponse
> {quote}
> There are multiple test failures, after making the suggested changes, Filing
> this jira to dedicate to the investigation of removing getTrueCause method.
> More detail will be put in the first comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6897) If DatanodeThreshold is not set up, do not show information regarding DatanodeThreshold during NN startup
[
https://issues.apache.org/jira/browse/HDFS-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524883#comment-14524883
]
Hadoop QA commented on HDFS-6897:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12665545/HDFS-6897.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10644/console |
This message was automatically generated.
> If DatanodeThreshold is not set up, do not show information regarding
> DatanodeThreshold during NN startup
> -
>
> Key: HDFS-6897
> URL: https://issues.apache.org/jira/browse/HDFS-6897
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.5.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
>Priority: Minor
> Attachments: HDFS-6897.patch
>
>
> During Namenode startup, we see the following message:
> {code}
> The number of live datanodes XXX has reached the minimum number 0. Safe mode
> will be turned off automatically once the thresholds have been reached.
> {code}
> We have not setup the datanode threshold. So this message is not necessary.
> It may be good to turn off this message if datanode threshold is not setup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6860) BlockStateChange logs are too noisy
[
https://issues.apache.org/jira/browse/HDFS-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524882#comment-14524882
]
Hadoop QA commented on HDFS-6860:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12665129/HDFS6860.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10643/console |
This message was automatically generated.
> BlockStateChange logs are too noisy
> ---
>
> Key: HDFS-6860
> URL: https://issues.apache.org/jira/browse/HDFS-6860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Chang Li
> Labels: newbie
> Attachments: HDFS6860.patch, HDFS6860.patch
>
>
> Block State Change logs are too noisy at the default INFO level and affect NN
> performance on busy clusters.
> Most of these state changes can be logged at debug level instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6861) Separate Balancer specific logic form Dispatcher
[
https://issues.apache.org/jira/browse/HDFS-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524880#comment-14524880
]
Hadoop QA commented on HDFS-6861:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 1s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12662699/h6861_20140819.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10642/console |
This message was automatically generated.
> Separate Balancer specific logic form Dispatcher
>
>
> Key: HDFS-6861
> URL: https://issues.apache.org/jira/browse/HDFS-6861
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h6861_20140818.patch, h6861_20140819.patch
>
>
> In order to balance datanode storage utilization of a cluster, Balancer (1)
> classifies datanodes into different groups (overUtilized, aboveAvgUtilized,
> belowAvgUtilized and underUtilized), (2) chooses source and target datanode
> pairs and (3) chooses blocks to move. Some of these logic are in Dispatcher.
> It is better to separate them out. This JIRA is a further work of HDFS-6828.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5887) Add suffix to generated protobuf class
[
https://issues.apache.org/jira/browse/HDFS-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524877#comment-14524877
]
Hadoop QA commented on HDFS-5887:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 1s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12661725/HDFS-5887.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10641/console |
This message was automatically generated.
> Add suffix to generated protobuf class
> --
>
> Key: HDFS-5887
> URL: https://issues.apache.org/jira/browse/HDFS-5887
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-5698 (FSImage in protobuf)
>Reporter: Haohui Mai
>Assignee: Tassapol Athiapinya
>Priority: Minor
> Attachments: HDFS-5887.000.patch,
> HDFS-5887.000.proto_files-only.patch, HDFS-5887.001.patch
>
>
> As suggested by [~tlipcon], the code is more readable if we give each class
> generated by the protobuf the suffix "Proto".
> This jira proposes to rename the classes and to introduce no functionality
> changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6813) WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable with thead-safe.
[
https://issues.apache.org/jira/browse/HDFS-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524873#comment-14524873
]
Hadoop QA commented on HDFS-6813:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12659533/HDFS-6813.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10640/console |
This message was automatically generated.
> WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable
> with thead-safe.
> ---
>
> Key: HDFS-6813
> URL: https://issues.apache.org/jira/browse/HDFS-6813
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6813.001.patch
>
>
> {{PositionedReadable}} definition requires the implementations for its
> interfaces should be thread-safe.
> OffsetUrlInputStream(WebHdfsFileSystem inputstream) doesn't implement these
> interfaces with tread-safe, this JIRA is to fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4916) DataTransfer may mask the IOException during block transfering
[
https://issues.apache.org/jira/browse/HDFS-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524870#comment-14524870
]
Hadoop QA commented on HDFS-4916:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12588510/4916.v0.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10638/console |
This message was automatically generated.
> DataTransfer may mask the IOException during block transfering
> --
>
> Key: HDFS-4916
> URL: https://issues.apache.org/jira/browse/HDFS-4916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.0.4-alpha, 2.0.5-alpha
>Reporter: Zesheng Wu
>Priority: Critical
> Attachments: 4916.v0.patch
>
>
> When a new datanode is added to the pipeline, the client will trigger the
> block transfer process. In the current implementation, the src datanode calls
> the run() method of the DataTransfer to transfer the block, this method will
> mask the IOExceptions during the transfering, and will make the client not
> realize the failure during the transferring, as a result the client will
> mistake the failing transferring as successful one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[
https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524872#comment-14524872
]
Hadoop QA commented on HDFS-4754:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12596540/4754.v4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10639/console |
This message was automatically generated.
> Add an API in the namenode to mark a datanode as stale
> --
>
> Key: HDFS-4754
> URL: https://issues.apache.org/jira/browse/HDFS-4754
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Critical
> Attachments: 4754.v1.patch, 4754.v2.patch, 4754.v4.patch,
> 4754.v4.patch
>
>
> There is a detection of the stale datanodes in HDFS since HDFS-3703, with a
> timeout, defaulted to 30s.
> There are two reasons to add an API to mark a node as stale even if the
> timeout is not yet reached:
> 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase,
> we sometimes start the recovery before a node is marked staled. (even with
> reasonable settings as: stale: 20s; HBase ZK timeout: 30s
> 2) Some third parties could detect that a node is dead before the timeout,
> hence saving us the cost of retrying. An example or such hw is Arista,
> presented here by [~tsuna]
> http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in
> HBASE-6290.
> As usual, even if the node is dead it can comeback before the 10 minutes
> limit. So I would propose to set a timebound. The API would be
> namenode.markStale(String ipAddress, int port, long durationInMs);
> After durationInMs, the namenode would again rely only on its heartbeat to
> decide.
> Thoughts?
> If there is no objections, and if nobody in the hdfs dev team has the time to
> spend some time on it, I will give it a try for branch 2 & 3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5464) Simplify block report diff calculation
[
https://issues.apache.org/jira/browse/HDFS-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524860#comment-14524860
]
Hadoop QA commented on HDFS-5464:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12655886/h5464_20140715b.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10634/console |
This message was automatically generated.
> Simplify block report diff calculation
> --
>
> Key: HDFS-5464
> URL: https://issues.apache.org/jira/browse/HDFS-5464
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h5464_20131105.patch, h5464_20131105b.patch,
> h5464_20131105c.patch, h5464_20140715.patch, h5464_20140715b.patch
>
>
> The current calculation in BlockManager.reportDiff(..) is unnecessarily
> complicated. We could simplify the calculation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524865#comment-14524865
]
Hadoop QA commented on HDFS-5730:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12623677/HDFS-5730.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10635/console |
This message was automatically generated.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-5730.patch, HDFS-5730.patch
>
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6588) Investigating removing getTrueCause method in Server.java
[
https://issues.apache.org/jira/browse/HDFS-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524866#comment-14524866
]
Hadoop QA commented on HDFS-6588:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12657013/HDFS-6588.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10636/console |
This message was automatically generated.
> Investigating removing getTrueCause method in Server.java
> -
>
> Key: HDFS-6588
> URL: https://issues.apache.org/jira/browse/HDFS-6588
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, webhdfs
>Affects Versions: 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6588.001.patch, HDFS-6588.001.patch,
> HDFS-6588.001.patch, HDFS-6588.001.patch
>
>
> When addressing Daryn Sharp's comment for HDFS-6475 quoted below:
> {quote}
> What I'm saying is I think the patch adds too much unnecessary code. Filing
> an improvement to delete all but a few lines of the code changed in this
> patch seems a bit odd. I think you just need to:
> - Delete getTrueCause entirely instead of moving it elsewhere
> - In saslProcess, just throw the exception instead of running it through
> getTrueCause since it's not a "InvalidToken wrapping another exception"
> anymore.
> - Keep your 3-line change to unwrap SecurityException in toResponse
> {quote}
> There are multiple test failures, after making the suggested changes, Filing
> this jira to dedicate to the investigation of removing getTrueCause method.
> More detail will be put in the first comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6526) Implement HDFS TtlManager
[
https://issues.apache.org/jira/browse/HDFS-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524868#comment-14524868
]
Hadoop QA commented on HDFS-6526:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12651960/HDFS-6526.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10637/console |
This message was automatically generated.
> Implement HDFS TtlManager
> -
>
> Key: HDFS-6526
> URL: https://issues.apache.org/jira/browse/HDFS-6526
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6526.1.patch
>
>
> This issue is used to track development of HDFS TtlManager, for details see
> HDFS-6382.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6262) HDFS doesn't raise FileNotFoundException if the source of a rename() is missing
[
https://issues.apache.org/jira/browse/HDFS-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524857#comment-14524857
]
Hadoop QA commented on HDFS-6262:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12641192/HDFS-6262.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10632/console |
This message was automatically generated.
> HDFS doesn't raise FileNotFoundException if the source of a rename() is
> missing
> ---
>
> Key: HDFS-6262
> URL: https://issues.apache.org/jira/browse/HDFS-6262
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>Assignee: Akira AJISAKA
> Attachments: HDFS-6262.2.patch, HDFS-6262.patch
>
>
> HDFS's {{rename(src, dest)}} returns false if src does not exist -all the
> other filesystems raise {{FileNotFoundException}}
> This behaviour is defined in {{FSDirectory.unprotectedRenameTo()}} -the
> attempt is logged, but the operation then just returns false.
> I propose changing the behaviour of {{DistributedFileSystem}} to be the same
> as that of the others -and of {{FileContext}}, which does reject renames with
> nonexistent sources
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4504) DFSOutputStream#close doesn't always release resources (such as leases)
[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524859#comment-14524859
]
Hadoop QA commented on HDFS-4504:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12599081/HDFS-4504.016.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10633/console |
This message was automatically generated.
> DFSOutputStream#close doesn't always release resources (such as leases)
> ---
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch,
> HDFS-4504.007.patch, HDFS-4504.008.patch, HDFS-4504.009.patch,
> HDFS-4504.010.patch, HDFS-4504.011.patch, HDFS-4504.014.patch,
> HDFS-4504.015.patch, HDFS-4504.016.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4861) BlockPlacementPolicyDefault does not consider decommissioning racks
[
https://issues.apache.org/jira/browse/HDFS-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524841#comment-14524841
]
Hadoop QA commented on HDFS-4861:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12638791/HDFS-4861-v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10629/console |
This message was automatically generated.
> BlockPlacementPolicyDefault does not consider decommissioning racks
> ---
>
> Key: HDFS-4861
> URL: https://issues.apache.org/jira/browse/HDFS-4861
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 0.23.7, 2.1.0-beta
>Reporter: Kihwal Lee
>Assignee: Rushabh S Shah
> Attachments: HDFS-4861-v2.patch, HDFS-4861.patch
>
>
> getMaxNodesPerRack() calculates the max replicas/rack like this:
> {code}
> int maxNodesPerRack = (totalNumOfReplicas-1)/clusterMap.getNumOfRacks()+2;
> {code}
> Since this does not consider the racks that are being decommissioned and the
> decommissioning state is only checked later in isGoodTarget(), certain blocks
> are not replicated even when there are many racks and nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-3503) Move LengthInputStream and PositionTrackingInputStream to common
[
https://issues.apache.org/jira/browse/HDFS-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524837#comment-14524837
]
Hadoop QA commented on HDFS-3503:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12639073/h3503_20140407.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10626/console |
This message was automatically generated.
> Move LengthInputStream and PositionTrackingInputStream to common
>
>
> Key: HDFS-3503
> URL: https://issues.apache.org/jira/browse/HDFS-3503
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h3503_20140328.patch, h3503_20140407.patch
>
>
> We have LengthInputStream in org.apache.hadoop.hdfs.server.datanode.fsdataset
> and PositionTrackingInputStream in
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader. These two classes
> are generally useful. Let's move them to org.apache.hadoop.io.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6525) FsShell supports HDFS TTL
[
https://issues.apache.org/jira/browse/HDFS-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524849#comment-14524849
]
Hadoop QA commented on HDFS-6525:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12652123/HDFS-6525.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10631/console |
This message was automatically generated.
> FsShell supports HDFS TTL
> -
>
> Key: HDFS-6525
> URL: https://issues.apache.org/jira/browse/HDFS-6525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, tools
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6525.1.patch, HDFS-6525.2.patch
>
>
> This issue is used to track development of supporting HDFS TTL for FsShell,
> for details see HDFS-6382.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4273) Fix some issue in DFSInputstream
[
https://issues.apache.org/jira/browse/HDFS-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524845#comment-14524845
]
Hadoop QA commented on HDFS-4273:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12621932/HDFS-4273.v8.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10630/console |
This message was automatically generated.
> Fix some issue in DFSInputstream
>
>
> Key: HDFS-4273
> URL: https://issues.apache.org/jira/browse/HDFS-4273
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.2-alpha
>Reporter: Binglin Chang
>Assignee: Binglin Chang
>Priority: Minor
> Attachments: HDFS-4273-v2.patch, HDFS-4273.patch, HDFS-4273.v3.patch,
> HDFS-4273.v4.patch, HDFS-4273.v5.patch, HDFS-4273.v6.patch,
> HDFS-4273.v7.patch, HDFS-4273.v8.patch, TestDFSInputStream.java
>
>
> Following issues in DFSInputStream are addressed in this jira:
> 1. read may not retry enough in some cases cause early failure
> Assume the following call logic
> {noformat}
> readWithStrategy()
> -> blockSeekTo()
> -> readBuffer()
> -> reader.doRead()
> -> seekToNewSource() add currentNode to deadnode, wish to get a
> different datanode
> -> blockSeekTo()
>-> chooseDataNode()
> -> block missing, clear deadNodes and pick the currentNode again
> seekToNewSource() return false
> readBuffer() re-throw the exception quit loop
> readWithStrategy() got the exception, and may fail the read call before
> tried MaxBlockAcquireFailures.
> {noformat}
> 2. In multi-threaded scenario(like hbase), DFSInputStream.failures has race
> condition, it is cleared to 0 when it is still used by other thread. So it is
> possible that some read thread may never quit. Change failures to local
> variable solve this issue.
> 3. If local datanode is added to deadNodes, it will not be removed from
> deadNodes if DN is back alive. We need a way to remove local datanode from
> deadNodes when the local datanode is become live.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5745) Unnecessary disk check triggered when socket operation has problem.
[
https://issues.apache.org/jira/browse/HDFS-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524839#comment-14524839
]
Hadoop QA commented on HDFS-5745:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12622369/HDFS-5745.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10628/console |
This message was automatically generated.
> Unnecessary disk check triggered when socket operation has problem.
> ---
>
> Key: HDFS-5745
> URL: https://issues.apache.org/jira/browse/HDFS-5745
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.2.1
>Reporter: MaoYuan Xian
> Attachments: HDFS-5745.patch
>
>
> When BlockReceiver transfer data fails, it can be found SocketOutputStream
> translates the exception as IOException with the message "The stream is
> closed":
> 2014-01-06 11:48:04,716 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver.run():
> java.io.IOException: The stream is closed
> at org.apache.hadoop.net.SocketOutputStream.write
> at java.io.BufferedOutputStream.flushBuffer
> at java.io.BufferedOutputStream.flush
> at java.io.DataOutputStream.flush
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run
> at java.lang.Thread.run
> Which makes the checkDiskError method of DataNode called and triggers the
> disk scan.
> Can we make the modifications like below in checkDiskError to avoiding this
> unneccessary disk scan operations?:
> {code}
> --- a/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> +++ b/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> @@ -938,7 +938,8 @@ public class DataNode extends Configured
> || e.getMessage().startsWith("An established connection was
> aborted")
> || e.getMessage().startsWith("Broken pipe")
> || e.getMessage().startsWith("Connection reset")
> - || e.getMessage().contains("java.nio.channels.SocketChannel")) {
> + || e.getMessage().contains("java.nio.channels.SocketChannel")
> + || e.getMessage().startsWith("The stream is closed")) {
>LOG.info("Not checking disk as checkDiskError was called on a network"
> +
> " related exception");
>return;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5296) DFS usage gets doubled in the WebUI of federated namenode
[
https://issues.apache.org/jira/browse/HDFS-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524832#comment-14524832
]
Hadoop QA commented on HDFS-5296:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12606661/HDFS-5296-v1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10625/console |
This message was automatically generated.
> DFS usage gets doubled in the WebUI of federated namenode
> -
>
> Key: HDFS-5296
> URL: https://issues.apache.org/jira/browse/HDFS-5296
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Siqi Li
>Assignee: Siqi Li
>Priority: Minor
> Attachments: BBF12817-B83E-4CC5-B0B8-4FA322E87FB7.png,
> HDFS-5296-v1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4019) FSShell should support creating symlinks
[
https://issues.apache.org/jira/browse/HDFS-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524838#comment-14524838
]
Hadoop QA commented on HDFS-4019:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12596248/HDFS-4019.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10627/console |
This message was automatically generated.
> FSShell should support creating symlinks
>
>
> Key: HDFS-4019
> URL: https://issues.apache.org/jira/browse/HDFS-4019
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-4019.001.patch, HDFS-4019.002.patch,
> HDFS-4019.003.patch
>
>
> FSShell should support creating symlinks. This would allow users to create
> symlinks from the shell without having to write a Java program.
> One thing that makes this complicated is that FSShell currently uses
> FileSystem internally, and symlinks are currently only supported by the
> FileContext API. So either FSShell would have to be ported to FileContext,
> or symlinks would have to be added to FileSystem. Or perhaps we could open a
> FileContext only when symlinks were necessary, but that seems messy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6002) DFSOutputStream.DataStreamer.bytesSent not updated correctly
[
https://issues.apache.org/jira/browse/HDFS-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524820#comment-14524820
]
Hadoop QA commented on HDFS-6002:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12630620/HDFS-6002.v1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10617/console |
This message was automatically generated.
> DFSOutputStream.DataStreamer.bytesSent not updated correctly
>
>
> Key: HDFS-6002
> URL: https://issues.apache.org/jira/browse/HDFS-6002
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Binglin Chang
>Assignee: Binglin Chang
>Priority: Minor
> Attachments: HDFS-6002.v1.patch
>
>
> DFSOutputStream.DataStreamer.bytesSent record bytes sent in current block, a
> simple search for all the references would discover that it is only updated
> larger and larger, but never got reset when a new block is allocated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5410) Remake org.apache.hadoop.hdfs.server.datanode.TestStartSecureDataNode on MiniKdc
[
https://issues.apache.org/jira/browse/HDFS-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524828#comment-14524828
]
Hadoop QA commented on HDFS-5410:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12609883/HDFS-5410-trunk--N1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10621/console |
This message was automatically generated.
> Remake org.apache.hadoop.hdfs.server.datanode.TestStartSecureDataNode on
> MiniKdc
>
>
> Key: HDFS-5410
> URL: https://issues.apache.org/jira/browse/HDFS-5410
> Project: Hadoop HDFS
> Issue Type: Test
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Ivan A. Veselovsky
>Assignee: Ivan A. Veselovsky
>Priority: Minor
> Attachments: HDFS-5410-branch-2--N1.patch, HDFS-5410-trunk--N1.patch
>
>
> Reimplement test
> org.apache.hadoop.hdfs.server.datanode.TestStartSecureDataNode on MiniKdc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5875) Add iterator support to INodesInPath
[
https://issues.apache.org/jira/browse/HDFS-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524830#comment-14524830
]
Hadoop QA commented on HDFS-5875:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12626789/HDFS-5875.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10623/console |
This message was automatically generated.
> Add iterator support to INodesInPath
>
>
> Key: HDFS-5875
> URL: https://issues.apache.org/jira/browse/HDFS-5875
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-5875.patch
>
>
> "Resolve as you go" inode iteration will help with the implementation of
> alternative locking schemes. It will also be the pre-cursor for resolving
> paths once and only once per operation, as opposed to ~3 times/call.
> This is an incremental and compatible change for IIP that does not break
> existing callers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6318) refreshServiceAcl cannot affect both active NN and standby NN
[
https://issues.apache.org/jira/browse/HDFS-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524826#comment-14524826
]
Hadoop QA commented on HDFS-6318:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12642804/HDFS-6318.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10619/console |
This message was automatically generated.
> refreshServiceAcl cannot affect both active NN and standby NN
> -
>
> Key: HDFS-6318
> URL: https://issues.apache.org/jira/browse/HDFS-6318
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>Assignee: Fengdong Yu
> Attachments: HDFS-6318.patch, HDFS-6318.patch
>
>
> refreshServiceAcl cannot affect both active NN and standby NN, it only select
> one NN to reload the ACL configuration. but we should reload Acl on both
> active NN and standby NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4312) fix test TestSecureNameNode and improve test TestSecureNameNodeWithExternalKdc
[
https://issues.apache.org/jira/browse/HDFS-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524822#comment-14524822
]
Hadoop QA commented on HDFS-4312:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12609871/HDFS-4312-trunk--N2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10618/console |
This message was automatically generated.
> fix test TestSecureNameNode and improve test TestSecureNameNodeWithExternalKdc
> --
>
> Key: HDFS-4312
> URL: https://issues.apache.org/jira/browse/HDFS-4312
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan A. Veselovsky
>Assignee: Ivan A. Veselovsky
> Attachments: HDFS-4312-trunk--N2.patch, HDFS-4312.patch
>
>
> TestSecureNameNode does not work on Java6 without
> "dfs.web.authentication.kerberos.principal" config property set.
> Also the following improved:
> 1) keytab files are checked for existence and readability to provide
> fast-fail on config error.
> 2) added comment to TestSecureNameNode describing the required sys props.
> 3) string literals replaced with config constants.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space
[
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524827#comment-14524827
]
Hadoop QA commented on HDFS-3570:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12627311/HDFS-3570.2.patch |
| Optional Tests | site javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10620/console |
This message was automatically generated.
> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used
> space
>
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Akira AJISAKA
>Priority: Minor
> Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch
>
>
> Report from a user here:
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
> post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among
> dfs.data.dirs, and very little DFS usage (which is computed against total
> possible capacity).
> Balancer apparently only looks at the usage, and ignores to consider that
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a
> DFS Usage report from DN is 8% only, its got a lot of free space to write
> more blocks, when that isn't true as shown by the case of this user. It went
> on scheduling writes to the DN to balance it out, but the DN simply can't
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space
> being used for non DN data, but it does expose a valid bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[
https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524829#comment-14524829
]
Hadoop QA commented on HDFS-5242:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12623196/HDFS-5242.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10622/console |
This message was automatically generated.
> Reduce contention on DatanodeInfo instances
> ---
>
> Key: HDFS-5242
> URL: https://issues.apache.org/jira/browse/HDFS-5242
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-5242.patch, HDFS-5242.patch
>
>
> Synchronization in {{DatanodeInfo}} instances causes unnecessary contention
> between call handlers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5177) blocksScheduled count should be decremented for abandoned blocks
[
https://issues.apache.org/jira/browse/HDFS-5177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524815#comment-14524815
]
Hadoop QA commented on HDFS-5177:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12616329/HDFS-5177.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10616/console |
This message was automatically generated.
> blocksScheduled count should be decremented for abandoned blocks
> -
>
> Key: HDFS-5177
> URL: https://issues.apache.org/jira/browse/HDFS-5177
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-5177.patch, HDFS-5177.patch, HDFS-5177.patch
>
>
> DatanodeDescriptor#incBlocksScheduled() will be called for all datanodes of
> the block on each allocation. But same should be decremented for abandoned
> blocks.
> When one of the datanodes is down and same is allocated for the block along
> with other live datanodes, then this block will be abandoned, but the
> scheduled count on other datanodes will consider live datanodes as loaded,
> but in reality these datanodes may not be loaded.
> Anyway this scheduled count will be rolled every 20 mins.
> Problem will come if the rate of creation of files is more. Due to increase
> in the scheduled count, there might be chances of missing local datanode to
> write to. and some times writes also can fail in small clusters.
> So we need to decrement the unnecessary count on abandon block call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5699) NPE is thrown when DN is restarted while job execution
[
https://issues.apache.org/jira/browse/HDFS-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524831#comment-14524831
]
Hadoop QA commented on HDFS-5699:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12620331/HDFS-5699-0001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10624/console |
This message was automatically generated.
> NPE is thrown when DN is restarted while job execution
> --
>
> Key: HDFS-5699
> URL: https://issues.apache.org/jira/browse/HDFS-5699
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: sathish
>Assignee: sathish
>Priority: Minor
> Attachments: HDFS-5699-0001.patch
>
>
> 1.Run jobs
> 2.Restart one DN
> 3.After DN come up it should not throw NPE in DN logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6596) Improve InputStream when read spans two blocks
[
https://issues.apache.org/jira/browse/HDFS-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524795#comment-14524795
]
Hadoop QA commented on HDFS-6596:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12653150/HDFS-6596.3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10608/console |
This message was automatically generated.
> Improve InputStream when read spans two blocks
> --
>
> Key: HDFS-6596
> URL: https://issues.apache.org/jira/browse/HDFS-6596
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6596.1.patch, HDFS-6596.2.patch, HDFS-6596.2.patch,
> HDFS-6596.2.patch, HDFS-6596.3.patch, HDFS-6596.3.patch
>
>
> In the current implementation of DFSInputStream, read(buffer, offset, length)
> is implemented as following:
> {code}
> int realLen = (int) Math.min(len, (blockEnd - pos + 1L));
> if (locatedBlocks.isLastBlockComplete()) {
> realLen = (int) Math.min(realLen, locatedBlocks.getFileLength());
> }
> int result = readBuffer(strategy, off, realLen, corruptedBlockMap);
> {code}
> From the above code, we can conclude that the read will return at most
> (blockEnd - pos + 1) bytes. As a result, when read spans two blocks, the
> caller must call read() second time to complete the request, and must wait
> second time to acquire the DFSInputStream lock(read() is synchronized for
> DFSInputStream). For latency sensitive applications, such as hbase, this will
> result in latency pain point when they under massive race conditions. So here
> we propose that we should loop internally in read() to do best effort read.
> In the current implementation of pread(read(position, buffer, offset,
> lenght)), it does loop internally to do best effort read. So we can refactor
> to support this on normal read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6093) Expose more caching information for debugging by users
[
https://issues.apache.org/jira/browse/HDFS-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524812#comment-14524812
]
Hadoop QA commented on HDFS-6093:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12635714/hdfs-6093-4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10615/console |
This message was automatically generated.
> Expose more caching information for debugging by users
> --
>
> Key: HDFS-6093
> URL: https://issues.apache.org/jira/browse/HDFS-6093
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: caching
>Affects Versions: 2.4.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-6093-1.patch, hdfs-6093-2.patch, hdfs-6093-3.patch,
> hdfs-6093-4.patch
>
>
> When users submit a new cache directive, it's unclear if the NN has
> recognized it and is actively trying to cache it, or if it's hung for some
> other reason. It'd be nice to expose a "pending caching/uncaching" count the
> same way we expose pending replication work.
> It'd also be nice to display the aggregate cache capacity and usage in
> dfsadmin -report, since we already have have it as a metric and expose it
> per-DN in report output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5301) adding block pool % for each namespace on federated namenode webUI
[
https://issues.apache.org/jira/browse/HDFS-5301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524801#comment-14524801
]
Hadoop QA commented on HDFS-5301:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12606846/HDFS-5301-v1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10611/console |
This message was automatically generated.
> adding block pool % for each namespace on federated namenode webUI
> --
>
> Key: HDFS-5301
> URL: https://issues.apache.org/jira/browse/HDFS-5301
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.5-alpha
>Reporter: Siqi Li
>Assignee: Siqi Li
>Priority: Minor
> Attachments: HDFS-5301-v1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5319) Links resolving either from active/standby should be same (example clicking on datanodes from Standby)
[
https://issues.apache.org/jira/browse/HDFS-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524809#comment-14524809
]
Hadoop QA commented on HDFS-5319:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12607249/HDFS-5319-v1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10614/console |
This message was automatically generated.
> Links resolving either from active/standby should be same (example clicking
> on datanodes from Standby)
> --
>
> Key: HDFS-5319
> URL: https://issues.apache.org/jira/browse/HDFS-5319
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Siqi Li
>Assignee: Siqi Li
>Priority: Minor
> Attachments: HDFS-5319-v1.patch
>
>
> click live nodes from standby namenode will throw exception "Operation
> category READ is not supported in state standby"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6092) DistributedFileSystem#getCanonicalServiceName() and DistributedFileSystem#getUri() may return inconsistent results w.r.t. port
[
https://issues.apache.org/jira/browse/HDFS-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524799#comment-14524799
]
Hadoop QA commented on HDFS-6092:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12634494/HDFS-6092-v4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10610/console |
This message was automatically generated.
> DistributedFileSystem#getCanonicalServiceName() and
> DistributedFileSystem#getUri() may return inconsistent results w.r.t. port
> --
>
> Key: HDFS-6092
> URL: https://issues.apache.org/jira/browse/HDFS-6092
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Ted Yu
> Attachments: HDFS-6092-v4.patch, haosdent-HDFS-6092-v2.patch,
> haosdent-HDFS-6092.patch, hdfs-6092-v1.txt, hdfs-6092-v2.txt, hdfs-6092-v3.txt
>
>
> I discovered this when working on HBASE-10717
> Here is sample code to reproduce the problem:
> {code}
> Path desPath = new Path("hdfs://127.0.0.1/");
> FileSystem desFs = desPath.getFileSystem(conf);
>
> String s = desFs.getCanonicalServiceName();
> URI uri = desFs.getUri();
> {code}
> Canonical name string contains the default port - 8020
> But uri doesn't contain port.
> This would result in the following exception:
> {code}
> testIsSameHdfs(org.apache.hadoop.hbase.util.TestFSHDFSUtils) Time elapsed:
> 0.001 sec <<< ERROR!
> java.lang.IllegalArgumentException: port out of range:-1
> at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
> at java.net.InetSocketAddress.(InetSocketAddress.java:224)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.getNNAddresses(FSHDFSUtils.java:88)
> {code}
> Thanks to Brando Li who helped debug this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs dfs -rmdir" commands can't remove empty directory
[
https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524804#comment-14524804
]
Hadoop QA commented on HDFS-6165:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12642655/HDFS-6165.006.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10612/console |
This message was automatically generated.
> "hdfs dfs -rm -r" and "hdfs dfs -rmdir" commands can't remove empty directory
> --
>
> Key: HDFS-6165
> URL: https://issues.apache.org/jira/browse/HDFS-6165
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Minor
> Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch,
> HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch,
> HDFS-6165.005.patch, HDFS-6165.006.patch, HDFS-6165.006.patch
>
>
> Given a directory owned by user A with WRITE permission containing an empty
> directory owned by user B, it is not possible to delete user B's empty
> directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the
> current implementation requires FULL permission of the empty directory, and
> throws exception.
> On the other hand, on linux, "rm -r" and "rmdir" command can remove empty
> directory as long as the parent directory has WRITE permission (and prefix
> component of the path have EXECUTE permission), For the tested OSes, some
> prompt user asking for confirmation, some don't.
> Here's a reproduction:
> {code}
> [root@vm01 ~]# hdfs dfs -ls /user/
> Found 4 items
> drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc
> drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs
> drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history
> drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive
> [root@vm01 ~]# hdfs dfs -ls /user/userabc
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo
> rm: Permission denied: user=userabc, access=ALL,
> inode="/user/userabc/foo":hdfs:users:drwxr-xr-x
> {code}
> The super user can delete the directory.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo
> Deleted /user/userabc/foo
> {code}
> The same is not true for files, however. They have the correct behavior.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11
> /user/userabc/foo-file
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file
> Deleted /user/userabc/foo-file
> {code}
> Using "hdfs dfs -rmdir" command:
> {code}
> bash-4.1$ hadoop fs -lsr /
> lsr: DEPRECATED: Please use 'ls -R' instead.
> drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user
> drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs
> drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc
> drwxr-xr-x - abcabc 0
