[jira] [Updated] (HDFS-7899) Improve EOF error message
[
https://issues.apache.org/jira/browse/HDFS-7899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated HDFS-7899:
---
Attachment: HDFS-7899-02.patch
Thanks [~qwertymaniac] , Updated the message please review
> Improve EOF error message
> -
>
> Key: HDFS-7899
> URL: https://issues.apache.org/jira/browse/HDFS-7899
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Attachments: HDFS-7899-00.patch, HDFS-7899-01.patch,
> HDFS-7899-02.patch
>
>
> Currently, a DN disconnection for reasons other than connection timeout or
> refused messages, such as an EOF message as a result of rejection or other
> network fault, reports in this manner:
> {code}
> WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /x.x.x.x: for
> block, add to deadNodes and continue. java.io.EOFException: Premature EOF: no
> length prefix available
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:171)
>
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:392)
>
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:137)
>
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1103)
>
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:538)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:750)
>
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:794)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:602)
> {code}
> This is not very clear to a user (warn's at the hdfs-client). It could likely
> be improved with a more diagnosable message, or at least the direct reason
> than an EOF.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9129) Move the safemode block count into BlockManager
[
https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9129:
Status: Patch Available (was: Open)
> Move the safemode block count into BlockManager
> ---
>
> Key: HDFS-9129
> URL: https://issues.apache.org/jira/browse/HDFS-9129
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-9129.000.patch
>
>
> The {{SafeMode}} needs to track whether there are enough blocks so that the
> NN can get out of the safemode. These fields can moved to the
> {{BlockManager}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9129) Move the safemode block count into BlockManager
[
https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9129:
Attachment: HDFS-9129.000.patch
> Move the safemode block count into BlockManager
> ---
>
> Key: HDFS-9129
> URL: https://issues.apache.org/jira/browse/HDFS-9129
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-9129.000.patch
>
>
> The {{SafeMode}} needs to track whether there are enough blocks so that the
> NN can get out of the safemode. These fields can moved to the
> {{BlockManager}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9197) Snapshot FileDiff added to last snapshot when INodeFile accessTime field is updated
Alex Ivanov created HDFS-9197:
-
Summary: Snapshot FileDiff added to last snapshot when INodeFile
accessTime field is updated
Key: HDFS-9197
URL: https://issues.apache.org/jira/browse/HDFS-9197
Project: Hadoop HDFS
Issue Type: Bug
Components: snapshots
Affects Versions: 2.4.0, 2.3.0
Reporter: Alex Ivanov
Summary
When a file in HDFS is read, its corresponding inode's accessTime field is
updated. If the file is present in the last snapshot, the accessTime change
causes a FileDiff to be added to the SnapshotDiff of the last snapshot.
This behavior has the following problems:
- Since FileDiff's reside in memory on the namenodes, snapshots become
progressively more memory-heavy with increasing volume of data in hdfs. On a
system with frequent updates, e.g. hourly, this becomes a big problem since
for, say 2000 snapshots, one can have 2000 FileDiff's per file pointing to the
same inode.
- FSImage grows in size tremendously, and upload operation from standby to
active namenode takes much longer.
- The generated FileDiff does not contain any useful information that I can
see. Since all FileDiff's for that file are pointing to the same inode, the
accessTime they see is the same.
Configuration:
CDH 5.0.5 (Hadoop 2.3 / 2.4)
We are NOT overwriting the default parameter:
DFS_NAMENODE_ACCESSTIME_PRECISION_DEFAULT = 360;
Note that it determines the allowed frequency of accessTime field updates -
every hour by default.
How to reproduce:
{code}
[root@node1076]# hdfs dfs -ls /data/tenants/testenv.testtenant/wddata
Found 3 items
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:52
/data/tenants/testenv.testtenant/wddata/folder1
-rw-r--r-- 3 hdfs hadoop 38 2015-10-05 03:13
/data/tenants/testenv.testtenant/wddata/testfile1
-rw-r--r-- 3 hdfs hadoop 21 2015-10-04 10:45
/data/tenants/testenv.testtenant/wddata/testfile2
[root@node1076]# hdfs dfs -ls /data/tenants/testenv.testtenant/wddata/.snapshot
Found 8 items
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:47
/data/tenants/testenv.testtenant/wddata/.snapshot/sn1
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:47
/data/tenants/testenv.testtenant/wddata/.snapshot/sn2
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:52
/data/tenants/testenv.testtenant/wddata/.snapshot/sn3
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:53
/data/tenants/testenv.testtenant/wddata/.snapshot/sn4
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:57
/data/tenants/testenv.testtenant/wddata/.snapshot/sn5
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:58
/data/tenants/testenv.testtenant/wddata/.snapshot/sn6
drwxr-xr-x - hdfs hadoop 0 2015-10-05 03:13
/data/tenants/testenv.testtenant/wddata/.snapshot/sn7
drwxr-xr-x - hdfs hadoop 0 2015-10-05 04:20
/data/tenants/testenv.testtenant/wddata/.snapshot/sn8
[root@node1076]# hdfs dfs -createSnapshot
/data/tenants/testenv.testtenant/wddata sn9
Created snapshot /data/tenants/testenv.testtenant/wddata/.snapshot/sn9
[root@node1076]# hdfs snapshotDiff /data/tenants/testenv.testtenant/wddata sn8
sn9
Difference between snapshot sn8 and snapshot sn9 under directory
/data/tenants/testenv.testtenant/wddata:
## IMPORTANT: testfile1 was put into HDFS more than 1 hour ago, which triggers
the accessTime update.
[root@node1076]# hdfs dfs -cat /data/tenants/testenv.testtenant/wddata/testfile1
This is test file 1, but now it's 11.
[root@node1076]# hdfs dfs -createSnapshot
/data/tenants/testenv.testtenant/wddata sn10
Created snapshot /data/tenants/testenv.testtenant/wddata/.snapshot/sn10
[root@node1076]# hdfs snapshotDiff /data/tenants/testenv.testtenant/wddata sn9
sn10
Difference between snapshot sn9 and snapshot sn10 under directory
/data/tenants/testenv.testtenant/wddata:
M ./testfile1
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HDFS-9182) Cleanup the findbugs and other issues after HDFS EC merged to trunk.
[ https://issues.apache.org/jira/browse/HDFS-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-9182 started by Uma Maheswara Rao G. - > Cleanup the findbugs and other issues after HDFS EC merged to trunk. > > > Key: HDFS-9182 > URL: https://issues.apache.org/jira/browse/HDFS-9182 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Uma Maheswara Rao G >Priority: Critical > > https://builds.apache.org/job/PreCommit-HDFS-Build/12754/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html > https://builds.apache.org/job/PreCommit-HDFS-Build/12754/artifact/patchprocess/patchReleaseAuditProblems.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943023#comment-14943023
]
Vinayakumar B commented on HDFS-9151:
-
+1 LGTM
Will commit soon.
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7899) Improve EOF error message
[
https://issues.apache.org/jira/browse/HDFS-7899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943025#comment-14943025
]
Hadoop QA commented on HDFS-7899:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 52s | Pre-patch trunk has 7 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 14s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 33s | The applied patch generated 1
new checkstyle issues (total was 51, now 51). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 2s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 0m 30s | Tests passed in
hadoop-hdfs-client. |
| | | 45m 42s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764975/HDFS-7899-02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 30e2f83 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12789/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
|
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12789/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12789/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12789/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12789/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12789/console |
This message was automatically generated.
> Improve EOF error message
> -
>
> Key: HDFS-7899
> URL: https://issues.apache.org/jira/browse/HDFS-7899
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Attachments: HDFS-7899-00.patch, HDFS-7899-01.patch,
> HDFS-7899-02.patch
>
>
> Currently, a DN disconnection for reasons other than connection timeout or
> refused messages, such as an EOF message as a result of rejection or other
> network fault, reports in this manner:
> {code}
> WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /x.x.x.x: for
> block, add to deadNodes and continue. java.io.EOFException: Premature EOF: no
> length prefix available
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:171)
>
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:392)
>
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:137)
>
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1103)
>
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:538)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:750)
>
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:794)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:602)
> {code}
> This is not very clear to a user (warn's at the hdfs-client). It could likely
> be improved with a more diagnosable message, or at least the direct reason
> than an EOF.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9151:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2.
Thanks [~surendrasingh] for contribution.
Thanks [~templedf] for reviews.
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943032#comment-14943032
]
Hudson commented on HDFS-9151:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8566 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8566/])
HDFS-9151. Mover should print the exit status/reason on console like
(vinayakumarb: rev 0faa4efa3dd74de9cc39584bf6e88cfbf3e9a045)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9114) NameNode and DataNode metric log file name should follow the other log file name format.
[
https://issues.apache.org/jira/browse/HDFS-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943038#comment-14943038
]
Vinayakumar B commented on HDFS-9114:
-
1. I feel, you can also add one more env variable to define the metrics logger
, similar to HDFS_AUDIT_LOGGER
2. Also these env variables needs to be updated (commented) in hadoop-env.sh
and hadoop-env.cmd
3. You might need to update hdfs.cmd as well to support this in windows
4. You can set default logger to NullAppender for tests as well as these
metrics outputs are not verified from the file in tests. There will be one more
appender added in tests. Currently there is no NullAppender in test's log4j,
you can add the following entry.
{code}
# Null Appender
log4j.appender.NullAppender=org.apache.log4j.varia.NullAppender
{code}
> NameNode and DataNode metric log file name should follow the other log file
> name format.
>
>
> Key: HDFS-9114
> URL: https://issues.apache.org/jira/browse/HDFS-9114
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-9114-branch-2.01.patch,
> HDFS-9114-branch-2.02.patch, HDFS-9114-trunk.01.patch,
> HDFS-9114-trunk.02.patch
>
>
> Currently datanode and namenode metric log file name is
> {{datanode-metrics.log}} and {{namenode-metrics.log}}.
> This file name should be like {{hadoop-hdfs-namenode-metric-host192.log}}
> same as namenode log file {{hadoop-hdfs-namenode-host192.log}}.
> This will help when we will copy log for issue analysis from different node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943052#comment-14943052
]
Hudson commented on HDFS-9151:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1218 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1218/])
HDFS-9151. Mover should print the exit status/reason on console like
(vinayakumarb: rev 0faa4efa3dd74de9cc39584bf6e88cfbf3e9a045)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-9070: -- Status: Open (was: Patch Available) > Allow fsck display pending replica location information for being-written > blocks > > > Key: HDFS-9070 > URL: https://issues.apache.org/jira/browse/HDFS-9070 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: HDFS-9070--HDFS-7285.00.patch, > HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch, > HDFS-9070-HDFS-7285.02.patch > > > When a EC file is being written, it can be helpful to allow fsck display > datanode information of the being-written EC file block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-9070: -- Attachment: HDFS-9070-trunk.03.patch [~jingzhao] Thanks for your advise, I have uploaded a new patch. Sorry for the delay, I just come back from NYC :) > Allow fsck display pending replica location information for being-written > blocks > > > Key: HDFS-9070 > URL: https://issues.apache.org/jira/browse/HDFS-9070 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: HDFS-9070--HDFS-7285.00.patch, > HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch, > HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch > > > When a EC file is being written, it can be helpful to allow fsck display > datanode information of the being-written EC file block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-9070: -- Status: Patch Available (was: Open) > Allow fsck display pending replica location information for being-written > blocks > > > Key: HDFS-9070 > URL: https://issues.apache.org/jira/browse/HDFS-9070 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: HDFS-9070--HDFS-7285.00.patch, > HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch, > HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch > > > When a EC file is being written, it can be helpful to allow fsck display > datanode information of the being-written EC file block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943060#comment-14943060
]
Surendra Singh Lilhore commented on HDFS-9151:
--
Thanks [~vinayrpet] for review and commit.
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943090#comment-14943090
]
Hudson commented on HDFS-9151:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #487 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/487/])
HDFS-9151. Mover should print the exit status/reason on console like
(vinayakumarb: rev 0faa4efa3dd74de9cc39584bf6e88cfbf3e9a045)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943091#comment-14943091
]
Hadoop QA commented on HDFS-8578:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 19s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 10s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 16s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 23s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 8s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 203m 5s | Tests failed in hadoop-hdfs. |
| | | 248m 40s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.web.TestWebHDFSOAuth2 |
| | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
| | hadoop.hdfs.web.TestWebHdfsContentLength |
| Timed out tests | org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool |
| |
org.apache.hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
| | org.apache.hadoop.hdfs.TestMiniDFSCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764964/HDFS-8578-10.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 30e2f83 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12788/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12788/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12788/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12788/console |
This message was automatically generated.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-branch-2.6.0.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[
https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943097#comment-14943097
]
Vinayakumar B commented on HDFS-8164:
-
Thanks [~xiaochen], patch looks good overall.
Have few comments
1. {{FSN#getCTime()}} is used only in test added. So it would be better to
annotate as @VisibleForTesting.
2. In Test, {{NameNode.format(conf);}}, I think this call is unnecessary.
3. {code}final long millsOneDay = 1000 * 60 * 60 * 24;
assertTrue(ctime < pre + millsOneDay);{code}
This also I feel unnecessary. Is it really required to verify cTime is within a
day. ?
> cTime is 0 in VERSION file for newly formatted NameNode.
>
>
> Key: HDFS-8164
> URL: https://issues.apache.org/jira/browse/HDFS-8164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.3-alpha
>Reporter: Chris Nauroth
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch,
> HDFS-8164.003.patch, HDFS-8164.004.patch
>
>
> After formatting a NameNode and inspecting its VERSION file, the cTime
> property shows 0. The value does get updated to current time during an
> upgrade, but I believe this is intended to be the creation time of the
> cluster, and therefore the initial value of 0 before an upgrade can cause
> confusion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943103#comment-14943103
]
Hudson commented on HDFS-9151:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2423 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2423/])
HDFS-9151. Mover should print the exit status/reason on console like
(vinayakumarb: rev 0faa4efa3dd74de9cc39584bf6e88cfbf3e9a045)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9137) DeadLock between DataNode#refreshVolumes and BPOfferService#registrationSucceeded
[ https://issues.apache.org/jira/browse/HDFS-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943128#comment-14943128 ] Vinayakumar B commented on HDFS-9137: - I agree with [~hitliuyi], exception could be thrown directly. I think Uma trying not to hide the original exception, in case triggerBlockReport throws any exception. And I see, triggering block reports may not be required in case of unchanged configurations. Currently w/ or w/o patch, blockreports are getting triggered when there is nothing changed. This also could be considered, to reduce the load on NameNode on unnecessary block report processing? > DeadLock between DataNode#refreshVolumes and > BPOfferService#registrationSucceeded > -- > > Key: HDFS-9137 > URL: https://issues.apache.org/jira/browse/HDFS-9137 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.7.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-9137.00.patch > > > I can see this code flows between DataNode#refreshVolumes and > BPOfferService#registrationSucceeded could cause deadLock. > In practice situation may be rare as user calling refreshVolumes at the time > DN registration with NN. But seems like issue can happen. > Reason for deadLock: > 1) refreshVolumes will be called with DN lock and after at the end it will > also trigger Block report. In the Block report call, > BPServiceActor#triggerBlockReport calls toString on bpos. Here it takes > readLock on bpos. > DN lock then boos lock > 2) BPOfferSetrvice#registrationSucceeded call is taking writeLock on bpos and > calling dn.bpRegistrationSucceeded which is again synchronized call on DN. > bpos lock and then DN lock. > So, this can clearly create dead lock. > I think simple fix could be to move triggerBlockReport call outside out DN > lock and I feel that call may not be really needed inside DN lock. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9137) DeadLock between DataNode#refreshVolumes and BPOfferService#registrationSucceeded
[ https://issues.apache.org/jira/browse/HDFS-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943133#comment-14943133 ] Vinayakumar B commented on HDFS-9137: - In an another thought.. How about updating the BPOS#toString() itself, without holding any lock. The return value of tostring() could be some string field which could be updated when registrationSucceeded() call comes. > DeadLock between DataNode#refreshVolumes and > BPOfferService#registrationSucceeded > -- > > Key: HDFS-9137 > URL: https://issues.apache.org/jira/browse/HDFS-9137 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.7.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-9137.00.patch > > > I can see this code flows between DataNode#refreshVolumes and > BPOfferService#registrationSucceeded could cause deadLock. > In practice situation may be rare as user calling refreshVolumes at the time > DN registration with NN. But seems like issue can happen. > Reason for deadLock: > 1) refreshVolumes will be called with DN lock and after at the end it will > also trigger Block report. In the Block report call, > BPServiceActor#triggerBlockReport calls toString on bpos. Here it takes > readLock on bpos. > DN lock then boos lock > 2) BPOfferSetrvice#registrationSucceeded call is taking writeLock on bpos and > calling dn.bpRegistrationSucceeded which is again synchronized call on DN. > bpos lock and then DN lock. > So, this can clearly create dead lock. > I think simple fix could be to move triggerBlockReport call outside out DN > lock and I feel that call may not be really needed inside DN lock. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7899) Improve EOF error message
[
https://issues.apache.org/jira/browse/HDFS-7899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943171#comment-14943171
]
Jagadesh Kiran N commented on HDFS-7899:
The Failures are not related to Project Change, please review
> Improve EOF error message
> -
>
> Key: HDFS-7899
> URL: https://issues.apache.org/jira/browse/HDFS-7899
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Attachments: HDFS-7899-00.patch, HDFS-7899-01.patch,
> HDFS-7899-02.patch
>
>
> Currently, a DN disconnection for reasons other than connection timeout or
> refused messages, such as an EOF message as a result of rejection or other
> network fault, reports in this manner:
> {code}
> WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /x.x.x.x: for
> block, add to deadNodes and continue. java.io.EOFException: Premature EOF: no
> length prefix available
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:171)
>
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:392)
>
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:137)
>
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1103)
>
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:538)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:750)
>
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:794)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:602)
> {code}
> This is not very clear to a user (warn's at the hdfs-client). It could likely
> be improved with a more diagnosable message, or at least the direct reason
> than an EOF.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943185#comment-14943185
]
Hudson commented on HDFS-9151:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #454 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/454/])
HDFS-9151. Mover should print the exit status/reason on console like
(vinayakumarb: rev 0faa4efa3dd74de9cc39584bf6e88cfbf3e9a045)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9129) Move the safemode block count into BlockManager
[
https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943189#comment-14943189
]
Hadoop QA commented on HDFS-9129:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 7m 48s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 15s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 16s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated
24 new checkstyle issues (total was 512, now 481). |
| {color:red}-1{color} | whitespace | 0m 4s | The patch has 6 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 34s | The patch appears to introduce 3
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 20s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 150m 21s | Tests failed in hadoop-hdfs. |
| | | 196m 15s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.web.TestWebHdfsContentLength |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | org.apache.hadoop.hdfs.server.common.TestJspHelper |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764976/HDFS-9129.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 30e2f83 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12790/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12790/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12790/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12790/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12790/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12790/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12790/console |
This message was automatically generated.
> Move the safemode block count into BlockManager
> ---
>
> Key: HDFS-9129
> URL: https://issues.apache.org/jira/browse/HDFS-9129
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-9129.000.patch
>
>
> The {{SafeMode}} needs to track whether there are enough blocks so that the
> NN can get out of the safemode. These fields can moved to the
> {{BlockManager}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943199#comment-14943199
]
Hudson commented on HDFS-9151:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #479 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/479/])
HDFS-9151. Mover should print the exit status/reason on console like
(vinayakumarb: rev 0faa4efa3dd74de9cc39584bf6e88cfbf3e9a045)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9194) AlreadyBeingCreatedException ... because pendingCreates is non-null but no leases found.
[
https://issues.apache.org/jira/browse/HDFS-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943270#comment-14943270
]
Tsz Wo Nicholas Sze commented on HDFS-9194:
---
It seems that this bug is already fixed by HDFS-6651.
> AlreadyBeingCreatedException ... because pendingCreates is non-null but no
> leases found.
>
>
> Key: HDFS-9194
> URL: https://issues.apache.org/jira/browse/HDFS-9194
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>
> There is a possible bug in FSDirectory.addFile causing no leases found for
> under construction files.
> {code}
> //FSDirectory
> INodeFile addFile(String path, PermissionStatus permissions,
> short replication, long preferredBlockSize,
> String clientName, String clientMachine)
> throws FileAlreadyExistsException, QuotaExceededException,
> UnresolvedLinkException, SnapshotAccessControlException, AclException {
> long modTime = now();
> INodeFile newNode = newINodeFile(namesystem.allocateNewInodeId(),
> permissions, modTime, modTime, replication, preferredBlockSize);
> newNode.toUnderConstruction(clientName, clientMachine);
> boolean added = false;
> writeLock();
> try {
> added = addINode(path, newNode);
> } finally {
> writeUnlock();
> }
> ...
> }
> {code}
> - newNode.toUnderConstruction(clientName, clientMachine) adds
> FileUnderConstructionFeature to the INode, i.e. the file becomes an under
> construction file. At this moment, there is no lease for this file yet. The
> lease will be added later in FSNamesystem.startFileInternal(..).
> - It is possible that addINode(path, newNode) adds the inode to the namespace
> tree but throws QuotaExceededException later on when calling
> updateModificationTime. (i.e. addINode -> addLastINode -> addChild ->
> parent.addChild -> updateModificationTime throws QuotaExceededException)
> Then, the newly added uc file is left in namespace but the corresponding
> lease won't be added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9194) AlreadyBeingCreatedException ... because pendingCreates is non-null but no leases found.
[
https://issues.apache.org/jira/browse/HDFS-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze resolved HDFS-9194.
---
Resolution: Duplicate
> AlreadyBeingCreatedException ... because pendingCreates is non-null but no
> leases found.
>
>
> Key: HDFS-9194
> URL: https://issues.apache.org/jira/browse/HDFS-9194
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>
> There is a possible bug in FSDirectory.addFile causing no leases found for
> under construction files.
> {code}
> //FSDirectory
> INodeFile addFile(String path, PermissionStatus permissions,
> short replication, long preferredBlockSize,
> String clientName, String clientMachine)
> throws FileAlreadyExistsException, QuotaExceededException,
> UnresolvedLinkException, SnapshotAccessControlException, AclException {
> long modTime = now();
> INodeFile newNode = newINodeFile(namesystem.allocateNewInodeId(),
> permissions, modTime, modTime, replication, preferredBlockSize);
> newNode.toUnderConstruction(clientName, clientMachine);
> boolean added = false;
> writeLock();
> try {
> added = addINode(path, newNode);
> } finally {
> writeUnlock();
> }
> ...
> }
> {code}
> - newNode.toUnderConstruction(clientName, clientMachine) adds
> FileUnderConstructionFeature to the INode, i.e. the file becomes an under
> construction file. At this moment, there is no lease for this file yet. The
> lease will be added later in FSNamesystem.startFileInternal(..).
> - It is possible that addINode(path, newNode) adds the inode to the namespace
> tree but throws QuotaExceededException later on when calling
> updateModificationTime. (i.e. addINode -> addLastINode -> addChild ->
> parent.addChild -> updateModificationTime throws QuotaExceededException)
> Then, the newly added uc file is left in namespace but the corresponding
> lease won't be added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9151) Mover should print the exit status/reason on console like balancer tool.
[
https://issues.apache.org/jira/browse/HDFS-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943308#comment-14943308
]
Hudson commented on HDFS-9151:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2394 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2394/])
HDFS-9151. Mover should print the exit status/reason on console like
(vinayakumarb: rev 0faa4efa3dd74de9cc39584bf6e88cfbf3e9a045)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Mover should print the exit status/reason on console like balancer tool.
>
>
> Key: HDFS-9151
> URL: https://issues.apache.org/jira/browse/HDFS-9151
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9151.01.patch, HDFS-9151.02.patch
>
>
> Mover should print exit reason on console --
> In cases where there is No blocks to move or unavailable Storages or any
> other, Mover tool gives No information on exit reason on the console--
> {code}
> # ./hdfs mover
> ...
> Sep 28, 2015 12:31:25 PM Mover took 10sec
> # echo $?
> 0
> # ./hdfs mover
> ...
> Sep 28, 2015 12:33:10 PM Mover took 1sec
> # echo $?
> 254
> {code}
> Unlike Balancer prints exit reason
> example--
> #./hdfs balancer
> ...
> {color:red}The cluster is balanced. Exiting...{color}
> Sep 28, 2015 12:18:02 PM Balancing took 1.744 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[
https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943307#comment-14943307
]
Hadoop QA commented on HDFS-9070:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 21s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 8s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 17s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 27s | The applied patch generated 3
new checkstyle issues (total was 118, now 120). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 199m 13s | Tests failed in hadoop-hdfs. |
| | | 245m 28s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.web.TestWebHdfsContentLength |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764979/HDFS-9070-trunk.03.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0faa4ef |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12791/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12791/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12791/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12791/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12791/console |
This message was automatically generated.
> Allow fsck display pending replica location information for being-written
> blocks
>
>
> Key: HDFS-9070
> URL: https://issues.apache.org/jira/browse/HDFS-9070
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: GAO Rui
>Assignee: GAO Rui
> Attachments: HDFS-9070--HDFS-7285.00.patch,
> HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch,
> HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch
>
>
> When a EC file is being written, it can be helpful to allow fsck display
> datanode information of the being-written EC file block group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943316#comment-14943316
]
Vinayakumar B commented on HDFS-9158:
-
+1 for latest patch.
Checkstyle comment is related to indentation, but thats intentional to keep
help message look better.
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9158:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2.
Thanks [~nijel] for contribution.
Thanks [~templedf] for reviews.
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943335#comment-14943335
]
Hudson commented on HDFS-9158:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8567 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8567/])
HDFS-9158. [OEV-Doc] : Document does not mention about '-f' and '-r'
(vinayakumarb: rev b925cf100662e155c9efeff06523b09303817394)
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsEditsViewer.md
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943341#comment-14943341
]
Hudson commented on HDFS-9158:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8568 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8568/])
HDFS-9158. [OEV-Doc] : Document does not mention about '-f' and '-r'
(vinayakumarb: rev b925cf100662e155c9efeff06523b09303817394)
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsEditsViewer.md
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943380#comment-14943380
]
Hudson commented on HDFS-9158:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #488 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/488/])
HDFS-9158. [OEV-Doc] : Document does not mention about '-f' and '-r'
(vinayakumarb: rev b925cf100662e155c9efeff06523b09303817394)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsEditsViewer.md
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9193) Fix incorrect references the usages of the DN in dfshealth.js
[ https://issues.apache.org/jira/browse/HDFS-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HDFS-9193: - Affects Version/s: 2.8.0 Fix Version/s: 2.8.0 > Fix incorrect references the usages of the DN in dfshealth.js > - > > Key: HDFS-9193 > URL: https://issues.apache.org/jira/browse/HDFS-9193 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Chang Li >Assignee: Chang Li >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9193.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943429#comment-14943429
]
Hudson commented on HDFS-9158:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #1219 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1219/])
HDFS-9158. [OEV-Doc] : Document does not mention about '-f' and '-r'
(vinayakumarb: rev b925cf100662e155c9efeff06523b09303817394)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsEditsViewer.md
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9193) Fix incorrect references the usages of the DN in dfshealth.js
[ https://issues.apache.org/jira/browse/HDFS-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943441#comment-14943441 ] Rushabh S Shah commented on HDFS-9193: -- [~wheat9]: do we need to commit to 2.7.2 also ? > Fix incorrect references the usages of the DN in dfshealth.js > - > > Key: HDFS-9193 > URL: https://issues.apache.org/jira/browse/HDFS-9193 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Chang Li >Assignee: Chang Li >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9193.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9193) Fix incorrect references the usages of the DN in dfshealth.js
[ https://issues.apache.org/jira/browse/HDFS-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943473#comment-14943473 ] Rushabh S Shah commented on HDFS-9193: -- bq. Haohui Mai: do we need to commit to 2.7.2 also ? My bad. I didn't see that the namenode UI change went only to branch-2 not branch-2.7 Sorry for the noise. > Fix incorrect references the usages of the DN in dfshealth.js > - > > Key: HDFS-9193 > URL: https://issues.apache.org/jira/browse/HDFS-9193 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Chang Li >Assignee: Chang Li >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9193.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943502#comment-14943502
]
Hudson commented on HDFS-9158:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #480 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/480/])
HDFS-9158. [OEV-Doc] : Document does not mention about '-f' and '-r'
(vinayakumarb: rev b925cf100662e155c9efeff06523b09303817394)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsEditsViewer.md
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9142) Namenode Http address is not configured correctly for federated cluster in MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943562#comment-14943562
]
Ming Ma commented on HDFS-9142:
---
Thanks [~l201514] for identifying the issue. Having each namenode use its own
Configuration should work.
* Why is the removal of {{if (!federation && lastDefaultFileSystem != null)}}?
That is still needed for non federated case? It might explain the test failure.
* Some code can be shared in the two steps a) set up the nameservice related
configs for each newly created NN. b) copy all NNs's nameservice related
configs to the all NNs. In both case, it needs something like {{private static
void copyKeys(Configuration srcConf, Configuration destConf, String
nameserviceId, String nnId)}}, where it copies RPC/HTTP/HTTPS addresses for the
specific nameservice/nnid from some conf to another conf.
> Namenode Http address is not configured correctly for federated cluster in
> MiniDFSCluster
> -
>
> Key: HDFS-9142
> URL: https://issues.apache.org/jira/browse/HDFS-9142
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Siqi Li
>Assignee: Siqi Li
> Attachments: HDFS-9142.v1.patch, HDFS-9142.v2.patch,
> HDFS-9142.v3.patch, HDFS-9142.v4.patch
>
>
> When setting up simpleHAFederatedTopology in MiniDFSCluster, each Namenode
> should have its own configuration object, and the configuration should have
> "dfs.namenode.http-address--" set up correctly for
> all pair
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9006) Provide BlockPlacementPolicy that supports upgrade domain
[
https://issues.apache.org/jira/browse/HDFS-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9006:
--
Attachment: HDFS-9006.patch
Here is the draft patch that implements the upgrade domain policy.
> Provide BlockPlacementPolicy that supports upgrade domain
> -
>
> Key: HDFS-9006
> URL: https://issues.apache.org/jira/browse/HDFS-9006
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
> Attachments: HDFS-9006.patch
>
>
> As part of the upgrade domain feature, we need to provide the actual upgrade
> domain block placement.
> Namenode provides a mechanism to specify custom block placement policy. We
> can use that to implement BlockPlacementPolicy with upgrade domain support.
> {noformat}
>
> dfs.block.replicator.classname
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithUpgradeDomain
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9006) Provide BlockPlacementPolicy that supports upgrade domain
[
https://issues.apache.org/jira/browse/HDFS-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9006:
--
Assignee: Ming Ma
Status: Patch Available (was: Open)
> Provide BlockPlacementPolicy that supports upgrade domain
> -
>
> Key: HDFS-9006
> URL: https://issues.apache.org/jira/browse/HDFS-9006
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9006.patch
>
>
> As part of the upgrade domain feature, we need to provide the actual upgrade
> domain block placement.
> Namenode provides a mechanism to specify custom block placement policy. We
> can use that to implement BlockPlacementPolicy with upgrade domain support.
> {noformat}
>
> dfs.block.replicator.classname
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithUpgradeDomain
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943597#comment-14943597
]
Hudson commented on HDFS-9158:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2424 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2424/])
HDFS-9158. [OEV-Doc] : Document does not mention about '-f' and '-r'
(vinayakumarb: rev b925cf100662e155c9efeff06523b09303817394)
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsEditsViewer.md
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9196) TestWebHdfsContentLength fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HDFS-9196:
---
Attachment: HDFS-9196.001.patch
The cause is the change of WebHdfsFileSystem by HDFS-8979. Attached patch
revert the 1 line.
{noformat}
@@ -207,7 +212,7 @@ public synchronized void initialize(URI uri, Configuration
conf
failoverSleepMaxMillis);
}
-this.workingDir = makeQualified(new Path(getHomeDirectoryString(ugi)));
+this.workingDir = makeQualified(getHomeDirectory());
this.canRefreshDelegationToken = UserGroupInformation.isSecurityEnabled();
this.disallowFallbackToInsecureCluster = !conf.getBoolean(
CommonConfigurationKeys.IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY,
{noformat}
QA build seems not able to detect this bug because fixing WebHdfsFileSystem in
hadoop-hdfs-client does not result in checking TestWebHdfsContentLength is in
hadoop-hdfs.
> TestWebHdfsContentLength fails on trunk
> ---
>
> Key: HDFS-9196
> URL: https://issues.apache.org/jira/browse/HDFS-9196
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>Assignee: Masatake Iwasaki
> Attachments: HDFS-9196.001.patch
>
>
> {quote}
> Tests run: 7, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 181.278 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.web.TestWebHdfsContentLength
> testPutOp(org.apache.hadoop.hdfs.web.TestWebHdfsContentLength) Time elapsed:
> 60.05 sec <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.web.TestWebHdfsContentLength.testPutOp(TestWebHdfsContentLength.java:116)
> testPutOpWithRedirect(org.apache.hadoop.hdfs.web.TestWebHdfsContentLength)
> Time elapsed: 0.01 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[chunked]> but was:<[0]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.web.TestWebHdfsContentLength.testPutOpWithRedirect(TestWebHdfsContentLength.java:130)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8164: Status: Open (was: Patch Available) > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8164: Attachment: HDFS-8164.005.patch > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[
https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943640#comment-14943640
]
Xiao Chen commented on HDFS-8164:
-
Thank you very much for the review [~vinayrpet]! I attached patch 005
addressing your comments.
{quote}
1. FSN#getCTime() is used only in test added. So it would be better to annotate
as @VisibleForTesting.
{quote}
Fixed. I used this in another local branch, but it's definitely clearer to set
@VisibleForTesting here, and remove it when it's used in another JIRA.
{quote}
2. In Test, NameNode.format(conf);, I think this call is unnecessary.
{quote}
Fixed. The builder by default formats the namenode when {{build}}.
{quote}
3.
final long millsOneDay = 1000 * 60 * 60 * 24;
assertTrue(ctime < pre + millsOneDay);
This also I feel unnecessary. Is it really required to verify cTime is within a
day. ?
{quote}
Updated. My intention is to test both the upper bound and the lower bound of
the cTime. I updated the test to get {{now}} dynamically after the minicluster
startup. Does this make sense?
> cTime is 0 in VERSION file for newly formatted NameNode.
>
>
> Key: HDFS-8164
> URL: https://issues.apache.org/jira/browse/HDFS-8164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.3-alpha
>Reporter: Chris Nauroth
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch,
> HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch
>
>
> After formatting a NameNode and inspecting its VERSION file, the cTime
> property shows 0. The value does get updated to current time during an
> upgrade, but I believe this is intended to be the creation time of the
> cluster, and therefore the initial value of 0 before an upgrade can cause
> confusion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9195) TestDelegationTokenForProxyUser.testWebHdfsDoAs fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943644#comment-14943644
]
Masatake Iwasaki commented on HDFS-9195:
The cause seems to be same with HDFS-9196.
> TestDelegationTokenForProxyUser.testWebHdfsDoAs fails on trunk
> --
>
> Key: HDFS-9195
> URL: https://issues.apache.org/jira/browse/HDFS-9195
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>
> {quote}
> testWebHdfsDoAs(org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser)
> Time elapsed: 1.299 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<...ocalhost:44528/user/[Proxy]User>
> but was:<...ocalhost:44528/user/[Real]User>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser.testWebHdfsDoAs(TestDelegationTokenForProxyUser.java:163)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8164: Attachment: (was: HDFS-8164.005.patch) > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8164: Status: Patch Available (was: Open) > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8164: Attachment: HDFS-8164.005.patch > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9181) Better handling of exceptions thrown during upgrade shutdown
[ https://issues.apache.org/jira/browse/HDFS-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943666#comment-14943666 ] Daniel Templeton commented on HDFS-9181: I think you can get what you want by just changing the catch from Throwable to Exception. An uncaught Error will generally bring the whole thing down anyway, so not catching Errors won't impact the hanging behavior. (And if it does, we have bigger issues.) > Better handling of exceptions thrown during upgrade shutdown > > > Key: HDFS-9181 > URL: https://issues.apache.org/jira/browse/HDFS-9181 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > > Previously in HDFS-7533, a bug was fixed by suppressing exceptions during > upgrade shutdown. It may be appropriate as a temporary fix, but it would be > better if the exception is handled in some other ways. > One way to handle it is by emitting a warning message. There could exist > other ways to handle it. This lira is created to discuss how to handle this > case better. > Thanks to [~templedf] for bringing this up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9198) Coalesce IBR processing in the NN
Daryn Sharp created HDFS-9198: - Summary: Coalesce IBR processing in the NN Key: HDFS-9198 URL: https://issues.apache.org/jira/browse/HDFS-9198 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.0.0-alpha Reporter: Daryn Sharp Assignee: Daryn Sharp IBRs from thousands of DNs under load will degrade NN performance due to excessive write-lock contention from multiple IPC handler threads. The IBR processing is quick, so the lock contention may be reduced by coalescing multiple IBRs into a single write-lock transaction. The handlers will also be freed up faster for other operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9176) TestDirectoryScanner#testThrottling often fails.
[ https://issues.apache.org/jira/browse/HDFS-9176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943696#comment-14943696 ] Daniel Templeton commented on HDFS-9176: The failure was not what I was expecting. There is a test to validate that things shut down promptly when requested, and that's what was failing. And it was failing big time. The test allows for a couple of seconds. The failured shutdowns were taking hours. But, it only failed twice. Since those two failures, the test has passed in the last 5 Jenkins runs. Since it appears to be working now, I'm going to close this issue. If it reappears later, we can reopen it. > TestDirectoryScanner#testThrottling often fails. > > > Key: HDFS-9176 > URL: https://issues.apache.org/jira/browse/HDFS-9176 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.8.0 >Reporter: Yi Liu >Assignee: Daniel Templeton >Priority: Minor > > https://builds.apache.org/job/PreCommit-HDFS-Build/12736/testReport/ > https://builds.apache.org/job/PreCommit-HADOOP-Build/7732/testReport/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9114) NameNode and DataNode metric log file name should follow the other log file name format.
[
https://issues.apache.org/jira/browse/HDFS-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943706#comment-14943706
]
Allen Wittenauer commented on HDFS-9114:
The more and more I look at this, the more and more I'm inclined to revert
HDFS-8880 . A property of hadoop.metrics.log.file *greatly* implies that we're
configuring the metrics2 file sink and not some HDFS-specific hook. This is
going to be *extremely* confusing for the end user.
The fact that that patch wasn't committed with documentation just makes it that
much worse.
> NameNode and DataNode metric log file name should follow the other log file
> name format.
>
>
> Key: HDFS-9114
> URL: https://issues.apache.org/jira/browse/HDFS-9114
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-9114-branch-2.01.patch,
> HDFS-9114-branch-2.02.patch, HDFS-9114-trunk.01.patch,
> HDFS-9114-trunk.02.patch
>
>
> Currently datanode and namenode metric log file name is
> {{datanode-metrics.log}} and {{namenode-metrics.log}}.
> This file name should be like {{hadoop-hdfs-namenode-metric-host192.log}}
> same as namenode log file {{hadoop-hdfs-namenode-host192.log}}.
> This will help when we will copy log for issue analysis from different node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9176) TestDirectoryScanner#testThrottling often fails.
[ https://issues.apache.org/jira/browse/HDFS-9176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton resolved HDFS-9176. Resolution: Not A Problem The test failed twice is an odd way and hasn't failed since in the last 5 runs. I'm going to blame environment or other changes. > TestDirectoryScanner#testThrottling often fails. > > > Key: HDFS-9176 > URL: https://issues.apache.org/jira/browse/HDFS-9176 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.8.0 >Reporter: Yi Liu >Assignee: Daniel Templeton >Priority: Minor > > https://builds.apache.org/job/PreCommit-HDFS-Build/12736/testReport/ > https://builds.apache.org/job/PreCommit-HADOOP-Build/7732/testReport/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9196) TestWebHdfsContentLength fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943720#comment-14943720
]
Mingliang Liu commented on HDFS-9196:
-
Thank you for this patch. I checked it out and also found the same reason.
The revert patch looks good to me.
> TestWebHdfsContentLength fails on trunk
> ---
>
> Key: HDFS-9196
> URL: https://issues.apache.org/jira/browse/HDFS-9196
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>Assignee: Masatake Iwasaki
> Attachments: HDFS-9196.001.patch
>
>
> {quote}
> Tests run: 7, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 181.278 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.web.TestWebHdfsContentLength
> testPutOp(org.apache.hadoop.hdfs.web.TestWebHdfsContentLength) Time elapsed:
> 60.05 sec <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.web.TestWebHdfsContentLength.testPutOp(TestWebHdfsContentLength.java:116)
> testPutOpWithRedirect(org.apache.hadoop.hdfs.web.TestWebHdfsContentLength)
> Time elapsed: 0.01 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[chunked]> but was:<[0]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.web.TestWebHdfsContentLength.testPutOpWithRedirect(TestWebHdfsContentLength.java:130)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9198) Coalesce IBR processing in the NN
[ https://issues.apache.org/jira/browse/HDFS-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-9198: -- Status: Patch Available (was: Open) > Coalesce IBR processing in the NN > - > > Key: HDFS-9198 > URL: https://issues.apache.org/jira/browse/HDFS-9198 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-9198-branch2.patch, HDFS-9198-trunk.patch > > > IBRs from thousands of DNs under load will degrade NN performance due to > excessive write-lock contention from multiple IPC handler threads. The IBR > processing is quick, so the lock contention may be reduced by coalescing > multiple IBRs into a single write-lock transaction. The handlers will also > be freed up faster for other operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9198) Coalesce IBR processing in the NN
[ https://issues.apache.org/jira/browse/HDFS-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-9198: -- Attachment: HDFS-9198-trunk.patch HDFS-9198-branch2.patch Incremental block reports are dumped into a queue for asynchronous processing by a background thread. This thread acquires the write lock and processes IBRs until the queue drains or a max lock hold is met. The max hold is 4ms which may seem high, but if the NN is backlogged that much, it's better to take the hit to catch up to avoid client issues. Full BR processing also uses the queuing in a synchronous manner. This helps preserve the ordering between the IBRs and full BRs from a node. Another reason for synchronous full BR processing is it may issue a finalize command. IBRs do not send commands so they can be async. However, in the unlikely event that an IBR fails, the DN currently re-queues the IBR, but now the DN always sees success. In practice an IBR fails if the DN is dead or unregistered. On the off-chance that an IBR fails for another reason, I added minimal support to force the DN to re-register which elicits a full BR for re-syncing. (The patches are trivially minor line number conflicts and a @VisibleForTesting) > Coalesce IBR processing in the NN > - > > Key: HDFS-9198 > URL: https://issues.apache.org/jira/browse/HDFS-9198 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-9198-branch2.patch, HDFS-9198-trunk.patch > > > IBRs from thousands of DNs under load will degrade NN performance due to > excessive write-lock contention from multiple IPC handler threads. The IBR > processing is quick, so the lock contention may be reduced by coalescing > multiple IBRs into a single write-lock transaction. The handlers will also > be freed up faster for other operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943727#comment-14943727
]
Hudson commented on HDFS-9158:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #455 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/455/])
HDFS-9158. [OEV-Doc] : Document does not mention about '-f' and '-r'
(vinayakumarb: rev b925cf100662e155c9efeff06523b09303817394)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsEditsViewer.md
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9184) Logging HDFS operation's caller context into audit logs
[
https://issues.apache.org/jira/browse/HDFS-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943730#comment-14943730
]
Daryn Sharp commented on HDFS-9184:
---
bq. This is possibly a dumb question, but couldn't clientId be used for this
purpose?
[~cmccabe], if you are thinking the dfs clientid like I did, it's only passed
in file writing operations. It won't help in storms of opens, listStatus,
getFileInfo, etc. If you are referring to the IPC client id then I'm not sure
how we provide any traceability?
> Logging HDFS operation's caller context into audit logs
> ---
>
> Key: HDFS-9184
> URL: https://issues.apache.org/jira/browse/HDFS-9184
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9184.000.patch
>
>
> For a given HDFS operation (e.g. delete file), it's very helpful to track
> which upper level job issues it. The upper level callers may be specific
> Oozie tasks, MR jobs, and hive queries. One scenario is that the namenode
> (NN) is abused/spammed, the operator may want to know immediately which MR
> job should be blamed so that she can kill it. To this end, the caller context
> contains at least the application-dependent "tracking id".
> There are several existing techniques that may be related to this problem.
> 1. Currently the HDFS audit log tracks the users of the the operation which
> is obviously not enough. It's common that the same user issues multiple jobs
> at the same time. Even for a single top level task, tracking back to a
> specific caller in a chain of operations of the whole workflow (e.g.Oozie ->
> Hive -> Yarn) is hard, if not impossible.
> 2. HDFS integrated {{htrace}} support for providing tracing information
> across multiple layers. The span is created in many places interconnected
> like a tree structure which relies on offline analysis across RPC boundary.
> For this use case, {{htrace}} has to be enabled at 100% sampling rate which
> introduces significant overhead. Moreover, passing additional information
> (via annotations) other than span id from root of the tree to leaf is a
> significant additional work.
> 3. In [HDFS-4680 | https://issues.apache.org/jira/browse/HDFS-4680], there
> are some related discussion on this topic. The final patch implemented the
> tracking id as a part of delegation token. This protects the tracking
> information from being changed or impersonated. However, kerberos
> authenticated connections or insecure connections don't have tokens.
> [HADOOP-8779] proposes to use tokens in all the scenarios, but that might
> mean changes to several upstream projects and is a major change in their
> security implementation.
> We propose another approach to address this problem. We also treat HDFS audit
> log as a good place for after-the-fact root cause analysis. We propose to put
> the caller id (e.g. Hive query id) in threadlocals. Specially, on client side
> the threadlocal object is passed to NN as a part of RPC header (optional),
> while on sever side NN retrieves it from header and put it to {{Handler}}'s
> threadlocals. Finally in {{FSNamesystem}}, HDFS audit logger will record the
> caller context for each operation. In this way, the existing code is not
> affected.
> It is still challenging to keep "lying" client from abusing the caller
> context. Our proposal is to add a {{signature}} field to the caller context.
> The client choose to provide its signature along with the caller id. The
> operator may need to validate the signature at the time of offline analysis.
> The NN is not responsible for validating the signature online.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8855) Webhdfs client leaks active NameNode connections
[ https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943744#comment-14943744 ] Daryn Sharp commented on HDFS-8855: --- Only glanced at the patch, but did you figure out why HDFS-7597 doesn't address the problem? It has for us and it's trivial compared to this patch. > Webhdfs client leaks active NameNode connections > > > Key: HDFS-8855 > URL: https://issues.apache.org/jira/browse/HDFS-8855 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Bob Hansen >Assignee: Xiaobing Zhou > Attachments: HDFS-8855.005.patch, HDFS-8855.006.patch, > HDFS-8855.007.patch, HDFS-8855.1.patch, HDFS-8855.2.patch, HDFS-8855.3.patch, > HDFS-8855.4.patch, HDFS_8855.prototype.patch > > > The attached script simulates a process opening ~50 files via webhdfs and > performing random reads. Note that there are at most 50 concurrent reads, > and all webhdfs sessions are kept open. Each read is ~64k at a random > position. > The script periodically (once per second) shells into the NameNode and > produces a summary of the socket states. For my test cluster with 5 nodes, > it took ~30 seconds for the NameNode to have ~25000 active connections and > fails. > It appears that each request to the webhdfs client is opening a new > connection to the NameNode and keeping it open after the request is complete. > If the process continues to run, eventually (~30-60 seconds), all of the > open connections are closed and the NameNode recovers. > This smells like SoftReference reaping. Are we using SoftReferences in the > webhdfs client to cache NameNode connections but never re-using them? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9158) [OEV-Doc] : Document does not mention about "-f" and "-r" options
[
https://issues.apache.org/jira/browse/HDFS-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943752#comment-14943752
]
Hudson commented on HDFS-9158:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2395 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2395/])
HDFS-9158. [OEV-Doc] : Document does not mention about '-f' and '-r'
(vinayakumarb: rev b925cf100662e155c9efeff06523b09303817394)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsEditsViewer.md
> [OEV-Doc] : Document does not mention about "-f" and "-r" options
> -
>
> Key: HDFS-9158
> URL: https://issues.apache.org/jira/browse/HDFS-9158
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9158.01.patch, HDFS-9158_02.patch,
> HDFS-9158_03.patch
>
>
> 1. Document does not mention about "-f" and "-r" options
> add these options also in document
> {noformat}
> -f,--fix-txids Renumber the transaction IDs in the input,
>so that there are no gaps or invalid transaction IDs.
> -r,--recover When reading binary edit logs, use recovery
>mode. This will give you the chance to skip
>corrupt parts of the edit log.
> {noformat}
> 2. In help message there is some extra white spaces
> {code}
> "so that there are no gaps or invalidtransaction IDs."
> {code}
> can remove this also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9142) Namenode Http address is not configured correctly for federated cluster in MiniDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-9142: -- Attachment: HDFS-9142.v5.patch > Namenode Http address is not configured correctly for federated cluster in > MiniDFSCluster > - > > Key: HDFS-9142 > URL: https://issues.apache.org/jira/browse/HDFS-9142 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: HDFS-9142.v1.patch, HDFS-9142.v2.patch, > HDFS-9142.v3.patch, HDFS-9142.v4.patch, HDFS-9142.v5.patch > > > When setting up simpleHAFederatedTopology in MiniDFSCluster, each Namenode > should have its own configuration object, and the configuration should have > "dfs.namenode.http-address--" set up correctly for > all pair -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8855) Webhdfs client leaks active NameNode connections
[
https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943802#comment-14943802
]
Xiaobing Zhou commented on HDFS-8855:
-
This issue is from DataNodeUGIProvider#tokenUGI
{code}
Token token = params.delegationToken();
ByteArrayInputStream buf =
new ByteArrayInputStream(token.getIdentifier());
DataInputStream in = new DataInputStream(buf);
DelegationTokenIdentifier id = new DelegationTokenIdentifier();
id.readFields(in);
UserGroupInformation ugi = id.getUser();
ugi.addToken(token);
return ugi;
{code}
for every request, a brand new UGI will be created even with HDFS-7597 patch,
because DelegationTokenIdentifier is initialized per request, although
HDFS-7597 returns the same UGI for the same DelegationTokenIdentifier instance.
> Webhdfs client leaks active NameNode connections
>
>
> Key: HDFS-8855
> URL: https://issues.apache.org/jira/browse/HDFS-8855
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Bob Hansen
>Assignee: Xiaobing Zhou
> Attachments: HDFS-8855.005.patch, HDFS-8855.006.patch,
> HDFS-8855.007.patch, HDFS-8855.1.patch, HDFS-8855.2.patch, HDFS-8855.3.patch,
> HDFS-8855.4.patch, HDFS_8855.prototype.patch
>
>
> The attached script simulates a process opening ~50 files via webhdfs and
> performing random reads. Note that there are at most 50 concurrent reads,
> and all webhdfs sessions are kept open. Each read is ~64k at a random
> position.
> The script periodically (once per second) shells into the NameNode and
> produces a summary of the socket states. For my test cluster with 5 nodes,
> it took ~30 seconds for the NameNode to have ~25000 active connections and
> fails.
> It appears that each request to the webhdfs client is opening a new
> connection to the NameNode and keeping it open after the request is complete.
> If the process continues to run, eventually (~30-60 seconds), all of the
> open connections are closed and the NameNode recovers.
> This smells like SoftReference reaping. Are we using SoftReferences in the
> webhdfs client to cache NameNode connections but never re-using them?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[
https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943884#comment-14943884
]
Yongjun Zhang commented on HDFS-8164:
-
Thanks [~vinayrpet] for the review and [~xiaochen] for the new rev.
It looks good, couple of small things:
1. comment for {{getCTime}}, the return value section is not correct, suggest
to change to something like:
{code}
/**
* Get the creation time of the file system.
* Notice that this time is initialized to NameNode format time, and updated
* to upgrade time during upgrades.
* @return time in milli seconds. See {@link
org.apache.hadoop.util.Time#now()}.
*/
{code}
2. Suggest to change the assertion in the test code to (<= instead of <).
{code}
assertTrue(ctime <= post);
{code}
Thanks.
> cTime is 0 in VERSION file for newly formatted NameNode.
>
>
> Key: HDFS-8164
> URL: https://issues.apache.org/jira/browse/HDFS-8164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.3-alpha
>Reporter: Chris Nauroth
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch,
> HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch
>
>
> After formatting a NameNode and inspecting its VERSION file, the cTime
> property shows 0. The value does get updated to current time during an
> upgrade, but I believe this is intended to be the creation time of the
> cluster, and therefore the initial value of 0 before an upgrade can cause
> confusion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9181) Better handling of exceptions thrown during upgrade shutdown
[
https://issues.apache.org/jira/browse/HDFS-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943905#comment-14943905
]
Eric Payne commented on HDFS-9181:
--
{quote}
I think you can get what you want by just changing the catch from Throwable to
Exception. An uncaught Error will generally bring the whole thing down anyway,
so not catching Errors won't impact the hanging behavior. (And if it does, we
have bigger issues.)
{quote}
Thanks, [~templedf]. I'm fine with catching Exceptions and not Throwables. I
would be interested in what [~kihwal] thinks.
> Better handling of exceptions thrown during upgrade shutdown
>
>
> Key: HDFS-9181
> URL: https://issues.apache.org/jira/browse/HDFS-9181
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
>
> Previously in HDFS-7533, a bug was fixed by suppressing exceptions during
> upgrade shutdown. It may be appropriate as a temporary fix, but it would be
> better if the exception is handled in some other ways.
> One way to handle it is by emitting a warning message. There could exist
> other ways to handle it. This lira is created to discuss how to handle this
> case better.
> Thanks to [~templedf] for bringing this up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9137) DeadLock between DataNode#refreshVolumes and BPOfferService#registrationSucceeded
[ https://issues.apache.org/jira/browse/HDFS-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943908#comment-14943908 ] Colin Patrick McCabe commented on HDFS-9137: bq. In an another thought.. How about updating the BPOS#toString() itself, without holding any lock. The return value of tostring() could be some string field which could be updated when registrationSucceeded() call comes. Good idea. > DeadLock between DataNode#refreshVolumes and > BPOfferService#registrationSucceeded > -- > > Key: HDFS-9137 > URL: https://issues.apache.org/jira/browse/HDFS-9137 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.7.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-9137.00.patch > > > I can see this code flows between DataNode#refreshVolumes and > BPOfferService#registrationSucceeded could cause deadLock. > In practice situation may be rare as user calling refreshVolumes at the time > DN registration with NN. But seems like issue can happen. > Reason for deadLock: > 1) refreshVolumes will be called with DN lock and after at the end it will > also trigger Block report. In the Block report call, > BPServiceActor#triggerBlockReport calls toString on bpos. Here it takes > readLock on bpos. > DN lock then boos lock > 2) BPOfferSetrvice#registrationSucceeded call is taking writeLock on bpos and > calling dn.bpRegistrationSucceeded which is again synchronized call on DN. > bpos lock and then DN lock. > So, this can clearly create dead lock. > I think simple fix could be to move triggerBlockReport call outside out DN > lock and I feel that call may not be really needed inside DN lock. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9138) TestDatanodeStartupFixesLegacyStorageIDs fails on Windows due to failure to unpack old image tarball that contains hard links
[
https://issues.apache.org/jira/browse/HDFS-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943915#comment-14943915
]
Colin Patrick McCabe commented on HDFS-9138:
[~cnauroth], just out of curiosity, what was the resolution on Windows? Do we
create the hardlinks there, or do we swap in the referenced file?
> TestDatanodeStartupFixesLegacyStorageIDs fails on Windows due to failure to
> unpack old image tarball that contains hard links
> -
>
> Key: HDFS-9138
> URL: https://issues.apache.org/jira/browse/HDFS-9138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9138.001.patch
>
>
> {{TestDatanodeStartupFixesLegacyStorageIDs#testUpgradeFrom22via26FixesStorageIDs}}
> uses a checked-in DataNode data directory that contains hard links. The
> hard links cannot be handled correctly by the commons-compress library used
> in the Windows implementation of {{FileUtil#unTar}}. The result is that the
> unpacked block files have 0 length, the block files reported to the NameNode
> are invalid, and therefore the mini-cluster never gets enough good blocks
> reported to leave safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9138) TestDatanodeStartupFixesLegacyStorageIDs fails on Windows due to failure to unpack old image tarball that contains hard links
[
https://issues.apache.org/jira/browse/HDFS-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943925#comment-14943925
]
Chris Nauroth commented on HDFS-9138:
-
Hi [~cmccabe]. We create the hard links on Windows now. In HDFS-8554, I
changed the untar logic in {{FileUtil}} to support hard links on Windows. When
I created this issue, I was running older code, and I had forgotten that I
already fixed root cause in HDFS-8554.
> TestDatanodeStartupFixesLegacyStorageIDs fails on Windows due to failure to
> unpack old image tarball that contains hard links
> -
>
> Key: HDFS-9138
> URL: https://issues.apache.org/jira/browse/HDFS-9138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9138.001.patch
>
>
> {{TestDatanodeStartupFixesLegacyStorageIDs#testUpgradeFrom22via26FixesStorageIDs}}
> uses a checked-in DataNode data directory that contains hard links. The
> hard links cannot be handled correctly by the commons-compress library used
> in the Windows implementation of {{FileUtil#unTar}}. The result is that the
> unpacked block files have 0 length, the block files reported to the NameNode
> are invalid, and therefore the mini-cluster never gets enough good blocks
> reported to leave safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9155) [OEV] : The inputFile does not follow case insensitiveness incase of XML file
[
https://issues.apache.org/jira/browse/HDFS-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943924#comment-14943924
]
Colin Patrick McCabe commented on HDFS-9155:
+1. Thanks, nijel
> [OEV] : The inputFile does not follow case insensitiveness incase of XML file
> -
>
> Key: HDFS-9155
> URL: https://issues.apache.org/jira/browse/HDFS-9155
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Attachments: HDFS-9155_01.patch
>
>
> As in document and help
> {noformat}
> -i,--inputFileedits file to process, xml (*case
>insensitive*) extension means XML format,
> {noformat}
> But if i give the file with "XML" extension it falls back to binary
> processing.
> This issue is due the code
> {code}
> int org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.go()
> .
> boolean xmlInput = inputFileName.endsWith(".xml");
> {code}
> Here need to check the xml after converting the file name to lower case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9155) OEV should treat .XML files as XML even when the extension is uppercase
[
https://issues.apache.org/jira/browse/HDFS-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-9155:
---
Summary: OEV should treat .XML files as XML even when the extension is
uppercase (was: [OEV] : The inputFile does not follow case insensitiveness
incase of XML file)
> OEV should treat .XML files as XML even when the extension is uppercase
> ---
>
> Key: HDFS-9155
> URL: https://issues.apache.org/jira/browse/HDFS-9155
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Attachments: HDFS-9155_01.patch
>
>
> As in document and help
> {noformat}
> -i,--inputFileedits file to process, xml (*case
>insensitive*) extension means XML format,
> {noformat}
> But if i give the file with "XML" extension it falls back to binary
> processing.
> This issue is due the code
> {code}
> int org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.go()
> .
> boolean xmlInput = inputFileName.endsWith(".xml");
> {code}
> Here need to check the xml after converting the file name to lower case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9155) OEV should treat .XML files as XML even when the file name extension is uppercase
[
https://issues.apache.org/jira/browse/HDFS-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-9155:
---
Summary: OEV should treat .XML files as XML even when the file name
extension is uppercase (was: OEV should treat .XML files as XML even when the
extension is uppercase)
> OEV should treat .XML files as XML even when the file name extension is
> uppercase
> -
>
> Key: HDFS-9155
> URL: https://issues.apache.org/jira/browse/HDFS-9155
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Attachments: HDFS-9155_01.patch
>
>
> As in document and help
> {noformat}
> -i,--inputFileedits file to process, xml (*case
>insensitive*) extension means XML format,
> {noformat}
> But if i give the file with "XML" extension it falls back to binary
> processing.
> This issue is due the code
> {code}
> int org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.go()
> .
> boolean xmlInput = inputFileName.endsWith(".xml");
> {code}
> Here need to check the xml after converting the file name to lower case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9155) OEV should treat .XML files as XML even when the file name extension is uppercase
[
https://issues.apache.org/jira/browse/HDFS-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-9155:
---
Resolution: Fixed
Fix Version/s: 2.8.0
Target Version/s: 2.8.0
Status: Resolved (was: Patch Available)
> OEV should treat .XML files as XML even when the file name extension is
> uppercase
> -
>
> Key: HDFS-9155
> URL: https://issues.apache.org/jira/browse/HDFS-9155
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9155_01.patch
>
>
> As in document and help
> {noformat}
> -i,--inputFileedits file to process, xml (*case
>insensitive*) extension means XML format,
> {noformat}
> But if i give the file with "XML" extension it falls back to binary
> processing.
> This issue is due the code
> {code}
> int org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.go()
> .
> boolean xmlInput = inputFileName.endsWith(".xml");
> {code}
> Here need to check the xml after converting the file name to lower case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9006) Provide BlockPlacementPolicy that supports upgrade domain
[
https://issues.apache.org/jira/browse/HDFS-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943937#comment-14943937
]
Hadoop QA commented on HDFS-9006:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 12s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 16s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated 4
new checkstyle issues (total was 400, now 404). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 9s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 199m 54s | Tests failed in hadoop-hdfs. |
| | | 245m 21s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.web.TestWebHDFSOAuth2 |
| | hadoop.hdfs.web.TestWebHdfsContentLength |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765018/HDFS-9006.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b925cf1 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12792/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12792/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12792/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12792/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12792/console |
This message was automatically generated.
> Provide BlockPlacementPolicy that supports upgrade domain
> -
>
> Key: HDFS-9006
> URL: https://issues.apache.org/jira/browse/HDFS-9006
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9006.patch
>
>
> As part of the upgrade domain feature, we need to provide the actual upgrade
> domain block placement.
> Namenode provides a mechanism to specify custom block placement policy. We
> can use that to implement BlockPlacementPolicy with upgrade domain support.
> {noformat}
>
> dfs.block.replicator.classname
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithUpgradeDomain
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7087) Ability to list /.reserved
[
https://issues.apache.org/jira/browse/HDFS-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-7087:
Attachment: HDFS-7087.patch.draft
Thanks very much [~andrew.wang] for the explanation!
I have created a draft patch to support listing /.reserved. Would you please
review and see if I'm on the right path? BTW There is a dependency on HDFS-8164
to get the format time, so if you test this patch the mtime is currently 0.
I'll get HDFS-8164 done asap.
Brief description of the draft:
* Add fake statuses of /.reserved, /.reserved/raw, /.reserved/.inodes. The
first is hard-coded, the latter 2 are lazy-initialized to read cTime, which has
to happen *strong*after*strong* {{FSDirectory}}'s initialization (specifically,
after FSNamesystem loads fsimage.
* {{FSDirStatAndListingOp#getFileInfo}} has to work so that
{{Globber#getFileStatus}} can return non-null status in order for the shell to
proceed to the actual command ({{fs.listStatus}})
* {{FSDirStatAndListingOp#getListing}} returns the statuses.
* {{FSDirectory#resolvePath}} doesn't seem to require any change - no special
resolution needed.
* Guard all FSDir**Ops from /.reserved path.
* Test cases
Any feedback is appreciated!
> Ability to list /.reserved
> --
>
> Key: HDFS-7087
> URL: https://issues.apache.org/jira/browse/HDFS-7087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Andrew Wang
>Assignee: Xiao Chen
> Attachments: HDFS-7087.patch.draft
>
>
> We have two special paths within /.reserved now, /.reserved/.inodes and
> /.reserved/raw. It seems like we should be able to list /.reserved to see
> them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8164: Status: Open (was: Patch Available) > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8164: Attachment: HDFS-8164.006.patch Thanks [~yzhangal] for the new review and comments. Both comments makes sense, and the comment you proposed is a lot clearer. :) Attaching updated patch 006. > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch, > HDFS-8164.006.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8164: Status: Patch Available (was: Open) > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch, > HDFS-8164.006.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8632) Erasure Coding: Add InterfaceAudience annotation to the erasure coding classes
[
https://issues.apache.org/jira/browse/HDFS-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8632:
Target Version/s: 3.0.0 (was: HDFS-7285)
> Erasure Coding: Add InterfaceAudience annotation to the erasure coding classes
> --
>
> Key: HDFS-8632
> URL: https://issues.apache.org/jira/browse/HDFS-8632
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8632-HDFS-7285-00.patch,
> HDFS-8632-HDFS-7285-01.patch, HDFS-8632-HDFS-7285-02.patch,
> HDFS-8632-HDFS-7285-03.patch, HDFS-8632-HDFS-7285-04.patch
>
>
> I've noticed some of the erasure coding classes missing
> {{@InterfaceAudience}} annotation. It would be good to identify the classes
> and add proper annotation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers
[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9079:
Component/s: erasure-coding
> Erasure coding: preallocate multiple generation stamps and serialize updates
> from data streamers
>
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-9079-HDFS-7285.00.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4)
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6)
> Updates block on NN
> {code}
> To simplify the above we can preallocate GS when NN creates a new striped
> block group ({{FSN#createNewBlock}}). For each new striped block group we can
> reserve {{NUM_PARITY_BLOCKS}} GS's. Then steps 1~3 in the above sequence can
> be saved. If more than {{NUM_PARITY_BLOCKS}} errors have happened we
> shouldn't try to further recover anyway.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9155) OEV should treat .XML files as XML even when the file name extension is uppercase
[
https://issues.apache.org/jira/browse/HDFS-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943976#comment-14943976
]
Hudson commented on HDFS-9155:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8571 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8571/])
HDFS-9155. OEV should treat .XML files as XML even when the file name (cmccabe:
rev 2fc2b50534c2052ed3f6f38f30d29d98d99fd10a)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> OEV should treat .XML files as XML even when the file name extension is
> uppercase
> -
>
> Key: HDFS-9155
> URL: https://issues.apache.org/jira/browse/HDFS-9155
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9155_01.patch
>
>
> As in document and help
> {noformat}
> -i,--inputFileedits file to process, xml (*case
>insensitive*) extension means XML format,
> {noformat}
> But if i give the file with "XML" extension it falls back to binary
> processing.
> This issue is due the code
> {code}
> int org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.go()
> .
> boolean xmlInput = inputFileName.endsWith(".xml");
> {code}
> Here need to check the xml after converting the file name to lower case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9155) OEV should treat .XML files as XML even when the file name extension is uppercase
[
https://issues.apache.org/jira/browse/HDFS-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944015#comment-14944015
]
Hudson commented on HDFS-9155:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #490 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/490/])
HDFS-9155. OEV should treat .XML files as XML even when the file name (cmccabe:
rev 2fc2b50534c2052ed3f6f38f30d29d98d99fd10a)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> OEV should treat .XML files as XML even when the file name extension is
> uppercase
> -
>
> Key: HDFS-9155
> URL: https://issues.apache.org/jira/browse/HDFS-9155
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9155_01.patch
>
>
> As in document and help
> {noformat}
> -i,--inputFileedits file to process, xml (*case
>insensitive*) extension means XML format,
> {noformat}
> But if i give the file with "XML" extension it falls back to binary
> processing.
> This issue is due the code
> {code}
> int org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.go()
> .
> boolean xmlInput = inputFileName.endsWith(".xml");
> {code}
> Here need to check the xml after converting the file name to lower case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[
https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944045#comment-14944045
]
Hadoop QA commented on HDFS-8164:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 1s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 27s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 16s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated 1
new checkstyle issues (total was 298, now 298). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 30s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 202m 46s | Tests failed in hadoop-hdfs. |
| | | 248m 47s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.TestGenericRefresh |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.web.TestWebHDFSOAuth2 |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| | hadoop.hdfs.web.TestWebHdfsContentLength |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765022/HDFS-8164.005.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b925cf1 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12793/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12793/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12793/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12793/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12793/console |
This message was automatically generated.
> cTime is 0 in VERSION file for newly formatted NameNode.
>
>
> Key: HDFS-8164
> URL: https://issues.apache.org/jira/browse/HDFS-8164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.3-alpha
>Reporter: Chris Nauroth
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch,
> HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch,
> HDFS-8164.006.patch
>
>
> After formatting a NameNode and inspecting its VERSION file, the cTime
> property shows 0. The value does get updated to current time during an
> upgrade, but I believe this is intended to be the creation time of the
> cluster, and therefore the initial value of 0 before an upgrade can cause
> confusion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9199) rename dfs.namenode.replication.min to dfs.replication.min
Allen Wittenauer created HDFS-9199: -- Summary: rename dfs.namenode.replication.min to dfs.replication.min Key: HDFS-9199 URL: https://issues.apache.org/jira/browse/HDFS-9199 Project: Hadoop HDFS Issue Type: Bug Reporter: Allen Wittenauer dfs.namenode.replication.min should be dfs.replication.min to match the other dfs.replication config knobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8632) Erasure Coding: Add InterfaceAudience annotation to the erasure coding classes
[
https://issues.apache.org/jira/browse/HDFS-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944074#comment-14944074
]
Andrew Wang commented on HDFS-8632:
---
There've been some refactors that moved things to hdfs-client, mind rebasing
the patch?
> Erasure Coding: Add InterfaceAudience annotation to the erasure coding classes
> --
>
> Key: HDFS-8632
> URL: https://issues.apache.org/jira/browse/HDFS-8632
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8632-HDFS-7285-00.patch,
> HDFS-8632-HDFS-7285-01.patch, HDFS-8632-HDFS-7285-02.patch,
> HDFS-8632-HDFS-7285-03.patch, HDFS-8632-HDFS-7285-04.patch
>
>
> I've noticed some of the erasure coding classes missing
> {{@InterfaceAudience}} annotation. It would be good to identify the classes
> and add proper annotation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9142) Namenode Http address is not configured correctly for federated cluster in MiniDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-9142: -- Attachment: HDFS-9142.v6.patch > Namenode Http address is not configured correctly for federated cluster in > MiniDFSCluster > - > > Key: HDFS-9142 > URL: https://issues.apache.org/jira/browse/HDFS-9142 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: HDFS-9142.v1.patch, HDFS-9142.v2.patch, > HDFS-9142.v3.patch, HDFS-9142.v4.patch, HDFS-9142.v5.patch, HDFS-9142.v6.patch > > > When setting up simpleHAFederatedTopology in MiniDFSCluster, each Namenode > should have its own configuration object, and the configuration should have > "dfs.namenode.http-address--" set up correctly for > all pair -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9197) Snapshot FileDiff added to last snapshot when INodeFile accessTime field is updated
[
https://issues.apache.org/jira/browse/HDFS-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Ivanov updated HDFS-9197:
--
Description:
Summary
When a file in HDFS is read, its corresponding inode's accessTime field is
updated. If the file is present in the last snapshot, the accessTime change
causes a FileDiff to be added to the SnapshotDiff of the last snapshot.
This behavior has the following problems:
- Since FileDiff's reside in memory on the namenodes, snapshots become
progressively more memory-heavy with increasing volume of data in hdfs. On a
system with frequent updates, e.g. hourly, this becomes a big problem since
for, say 2000 snapshots, one can have 2000 FileDiff's per file pointing to the
same inode.
- FSImage grows in size tremendously, and upload operation from standby to
active namenode takes much longer.
- The generated FileDiff does not contain any useful information that I can
see. Since all FileDiff's for that file are pointing to the same inode, the
accessTime they see is the same. -
- I was wrong about the last point. Each FileDiff includes a SnapshotCopy
attribute, which contains the updated accessTime. This may be a feature, but
I'd question the value of having it enabled by default.
Configuration:
CDH 5.0.5 (Hadoop 2.3 / 2.4)
We are NOT overwriting the default parameter:
DFS_NAMENODE_ACCESSTIME_PRECISION_DEFAULT = 360;
Note that it determines the allowed frequency of accessTime field updates -
every hour by default.
How to reproduce:
{code}
[root@node1076]# hdfs dfs -ls /data/tenants/testenv.testtenant/wddata
Found 3 items
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:52
/data/tenants/testenv.testtenant/wddata/folder1
-rw-r--r-- 3 hdfs hadoop 38 2015-10-05 03:13
/data/tenants/testenv.testtenant/wddata/testfile1
-rw-r--r-- 3 hdfs hadoop 21 2015-10-04 10:45
/data/tenants/testenv.testtenant/wddata/testfile2
[root@node1076]# hdfs dfs -ls /data/tenants/testenv.testtenant/wddata/.snapshot
Found 8 items
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:47
/data/tenants/testenv.testtenant/wddata/.snapshot/sn1
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:47
/data/tenants/testenv.testtenant/wddata/.snapshot/sn2
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:52
/data/tenants/testenv.testtenant/wddata/.snapshot/sn3
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:53
/data/tenants/testenv.testtenant/wddata/.snapshot/sn4
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:57
/data/tenants/testenv.testtenant/wddata/.snapshot/sn5
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:58
/data/tenants/testenv.testtenant/wddata/.snapshot/sn6
drwxr-xr-x - hdfs hadoop 0 2015-10-05 03:13
/data/tenants/testenv.testtenant/wddata/.snapshot/sn7
drwxr-xr-x - hdfs hadoop 0 2015-10-05 04:20
/data/tenants/testenv.testtenant/wddata/.snapshot/sn8
[root@node1076]# hdfs dfs -createSnapshot
/data/tenants/testenv.testtenant/wddata sn9
Created snapshot /data/tenants/testenv.testtenant/wddata/.snapshot/sn9
[root@node1076]# hdfs snapshotDiff /data/tenants/testenv.testtenant/wddata sn8
sn9
Difference between snapshot sn8 and snapshot sn9 under directory
/data/tenants/testenv.testtenant/wddata:
## IMPORTANT: testfile1 was put into HDFS more than 1 hour ago, which triggers
the accessTime update.
[root@node1076]# hdfs dfs -cat /data/tenants/testenv.testtenant/wddata/testfile1
This is test file 1, but now it's 11.
[root@node1076]# hdfs dfs -createSnapshot
/data/tenants/testenv.testtenant/wddata sn10
Created snapshot /data/tenants/testenv.testtenant/wddata/.snapshot/sn10
[root@node1076]# hdfs snapshotDiff /data/tenants/testenv.testtenant/wddata sn9
sn10
Difference between snapshot sn9 and snapshot sn10 under directory
/data/tenants/testenv.testtenant/wddata:
M ./testfile1
{code}
was:
Summary
When a file in HDFS is read, its corresponding inode's accessTime field is
updated. If the file is present in the last snapshot, the accessTime change
causes a FileDiff to be added to the SnapshotDiff of the last snapshot.
This behavior has the following problems:
- Since FileDiff's reside in memory on the namenodes, snapshots become
progressively more memory-heavy with increasing volume of data in hdfs. On a
system with frequent updates, e.g. hourly, this becomes a big problem since
for, say 2000 snapshots, one can have 2000 FileDiff's per file pointing to the
same inode.
- FSImage grows in size tremendously, and upload operation from standby to
active namenode takes much longer.
- The generated FileDiff does not contain any useful information that I can
see. Since all FileDiff's for that file are pointing to the same inode, the
accessTime they see is the same.
Configuration:
CDH 5.0.5 (Hadoop 2.3 / 2.4)
We are NOT overwriting the default parameter:
DFS_NAMENODE_ACCESSTIME_PRECISION_DEFAULT = 360;
Note that it de
[jira] [Updated] (HDFS-9197) Snapshot FileDiff added to last snapshot when INodeFile accessTime field is updated
[
https://issues.apache.org/jira/browse/HDFS-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Ivanov updated HDFS-9197:
--
Description:
Summary
When a file in HDFS is read, its corresponding inode's accessTime field is
updated. If the file is present in the last snapshot, the accessTime change
causes a FileDiff to be added to the SnapshotDiff of the last snapshot.
This behavior has the following problems:
- Since FileDiff's reside in memory on the namenodes, snapshots become
progressively more memory-heavy with increasing volume of data in hdfs. On a
system with frequent updates, e.g. hourly, this becomes a big problem since
for, say 2000 snapshots, one can have 2000 FileDiff's per file pointing to the
same inode.
- FSImage grows in size tremendously, and upload operation from standby to
active namenode takes much longer.
-The generated FileDiff does not contain any useful information that I can see.
Since all FileDiff's for that file are pointing to the same inode, the
accessTime they see is the same.-
- I was wrong about the last point. Each FileDiff includes a SnapshotCopy
attribute, which contains the updated accessTime. This may be a feature, but
I'd question the value of having it enabled by default.
Configuration:
CDH 5.0.5 (Hadoop 2.3 / 2.4)
We are NOT overwriting the default parameter:
DFS_NAMENODE_ACCESSTIME_PRECISION_DEFAULT = 360;
Note that it determines the allowed frequency of accessTime field updates -
every hour by default.
How to reproduce:
{code}
[root@node1076]# hdfs dfs -ls /data/tenants/testenv.testtenant/wddata
Found 3 items
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:52
/data/tenants/testenv.testtenant/wddata/folder1
-rw-r--r-- 3 hdfs hadoop 38 2015-10-05 03:13
/data/tenants/testenv.testtenant/wddata/testfile1
-rw-r--r-- 3 hdfs hadoop 21 2015-10-04 10:45
/data/tenants/testenv.testtenant/wddata/testfile2
[root@node1076]# hdfs dfs -ls /data/tenants/testenv.testtenant/wddata/.snapshot
Found 8 items
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:47
/data/tenants/testenv.testtenant/wddata/.snapshot/sn1
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:47
/data/tenants/testenv.testtenant/wddata/.snapshot/sn2
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:52
/data/tenants/testenv.testtenant/wddata/.snapshot/sn3
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:53
/data/tenants/testenv.testtenant/wddata/.snapshot/sn4
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:57
/data/tenants/testenv.testtenant/wddata/.snapshot/sn5
drwxr-xr-x - hdfs hadoop 0 2015-10-04 10:58
/data/tenants/testenv.testtenant/wddata/.snapshot/sn6
drwxr-xr-x - hdfs hadoop 0 2015-10-05 03:13
/data/tenants/testenv.testtenant/wddata/.snapshot/sn7
drwxr-xr-x - hdfs hadoop 0 2015-10-05 04:20
/data/tenants/testenv.testtenant/wddata/.snapshot/sn8
[root@node1076]# hdfs dfs -createSnapshot
/data/tenants/testenv.testtenant/wddata sn9
Created snapshot /data/tenants/testenv.testtenant/wddata/.snapshot/sn9
[root@node1076]# hdfs snapshotDiff /data/tenants/testenv.testtenant/wddata sn8
sn9
Difference between snapshot sn8 and snapshot sn9 under directory
/data/tenants/testenv.testtenant/wddata:
## IMPORTANT: testfile1 was put into HDFS more than 1 hour ago, which triggers
the accessTime update.
[root@node1076]# hdfs dfs -cat /data/tenants/testenv.testtenant/wddata/testfile1
This is test file 1, but now it's 11.
[root@node1076]# hdfs dfs -createSnapshot
/data/tenants/testenv.testtenant/wddata sn10
Created snapshot /data/tenants/testenv.testtenant/wddata/.snapshot/sn10
[root@node1076]# hdfs snapshotDiff /data/tenants/testenv.testtenant/wddata sn9
sn10
Difference between snapshot sn9 and snapshot sn10 under directory
/data/tenants/testenv.testtenant/wddata:
M ./testfile1
{code}
was:
Summary
When a file in HDFS is read, its corresponding inode's accessTime field is
updated. If the file is present in the last snapshot, the accessTime change
causes a FileDiff to be added to the SnapshotDiff of the last snapshot.
This behavior has the following problems:
- Since FileDiff's reside in memory on the namenodes, snapshots become
progressively more memory-heavy with increasing volume of data in hdfs. On a
system with frequent updates, e.g. hourly, this becomes a big problem since
for, say 2000 snapshots, one can have 2000 FileDiff's per file pointing to the
same inode.
- FSImage grows in size tremendously, and upload operation from standby to
active namenode takes much longer.
- The generated FileDiff does not contain any useful information that I can
see. Since all FileDiff's for that file are pointing to the same inode, the
accessTime they see is the same. -
- I was wrong about the last point. Each FileDiff includes a SnapshotCopy
attribute, which contains the updated accessTime. This may be a feature, but
I'd qu
[jira] [Assigned] (HDFS-9199) rename dfs.namenode.replication.min to dfs.replication.min
[ https://issues.apache.org/jira/browse/HDFS-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reassigned HDFS-9199: --- Assignee: Mingliang Liu > rename dfs.namenode.replication.min to dfs.replication.min > -- > > Key: HDFS-9199 > URL: https://issues.apache.org/jira/browse/HDFS-9199 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Mingliang Liu > > dfs.namenode.replication.min should be dfs.replication.min to match the other > dfs.replication config knobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9180: Attachment: HDFS-9180.003.patch Thanks for the review, Walter and Yi! Update the patch to address Walter's comments. > Update excluded DataNodes in DFSStripedOutputStream based on failures in data > streamers > --- > > Key: HDFS-9180 > URL: https://issues.apache.org/jira/browse/HDFS-9180 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-9180.000.patch, HDFS-9180.001.patch, > HDFS-9180.002.patch, HDFS-9180.003.patch > > > This is a TODO in HDFS-9040: based on the failures all the striped data > streamers hit, the DFSStripedOutputStream should keep a record of all the > DataNodes that should be excluded. > This jira will also fix several bugs in the DFSStripedOutputStream. Will > provide more details in the comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8164) cTime is 0 in VERSION file for newly formatted NameNode.
[ https://issues.apache.org/jira/browse/HDFS-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944103#comment-14944103 ] Yongjun Zhang commented on HDFS-8164: - Thanks for the new rev [~xiaochen]. Patch rev6 looks good to me. Hi [~vinayrpet], wonder if you have further comments. I will commit tomorrow if not. Thanks. > cTime is 0 in VERSION file for newly formatted NameNode. > > > Key: HDFS-8164 > URL: https://issues.apache.org/jira/browse/HDFS-8164 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.3-alpha >Reporter: Chris Nauroth >Assignee: Xiao Chen >Priority: Minor > Attachments: HDFS-8164.001.patch, HDFS-8164.002.patch, > HDFS-8164.003.patch, HDFS-8164.004.patch, HDFS-8164.005.patch, > HDFS-8164.006.patch > > > After formatting a NameNode and inspecting its VERSION file, the cTime > property shows 0. The value does get updated to current time during an > upgrade, but I believe this is intended to be the creation time of the > cluster, and therefore the initial value of 0 before an upgrade can cause > confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9199) rename dfs.namenode.replication.min to dfs.replication.min
[
https://issues.apache.org/jira/browse/HDFS-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944104#comment-14944104
]
Mingliang Liu commented on HDFS-9199:
-
Thanks for reporting this. I agree that we need to make the replication config
keys' name consistent.
I see
{code}
public static final String DFS_REPLICATION_MAX_KEY = "dfs.replication.max";
public static final int DFS_REPLICATION_MAX_DEFAULT = 512;
{code}
and
{code}
public static final String DFS_NAMENODE_REPLICATION_CONSIDERLOAD_KEY =
"dfs.namenode.replication.considerLoad";
public static final boolean DFS_NAMENODE_REPLICATION_CONSIDERLOAD_DEFAULT =
true;
public static final String DFS_NAMENODE_REPLICATION_INTERVAL_KEY =
"dfs.namenode.replication.interval";
public static final int DFS_NAMENODE_REPLICATION_INTERVAL_DEFAULT = 3;
public static final String DFS_NAMENODE_REPLICATION_MIN_KEY =
"dfs.namenode.replication.min";
public static final int DFS_NAMENODE_REPLICATION_MIN_DEFAULT = 1;
{code}
Do we need to consider all of them as well? Which clause of renaming is
preferred?
I think these keys are all used in namenode.
> rename dfs.namenode.replication.min to dfs.replication.min
> --
>
> Key: HDFS-9199
> URL: https://issues.apache.org/jira/browse/HDFS-9199
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Allen Wittenauer
>Assignee: Mingliang Liu
>
> dfs.namenode.replication.min should be dfs.replication.min to match the other
> dfs.replication config knobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944105#comment-14944105 ] Jing Zhao commented on HDFS-1172: - bq. I'm trying to update the pendingReplications only in the code path of completeFile now Yeah, good idea. We can do it not necessarily in checkReplication, but in file completion stage. > Blocks in newly completed files are considered under-replicated too quickly > --- > > Key: HDFS-1172 > URL: https://issues.apache.org/jira/browse/HDFS-1172 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.21.0 >Reporter: Todd Lipcon >Assignee: Masatake Iwasaki > Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch, > HDFS-1172.009.patch, HDFS-1172.010.patch, HDFS-1172.patch, hdfs-1172.txt, > hdfs-1172.txt, replicateBlocksFUC.patch, replicateBlocksFUC1.patch, > replicateBlocksFUC1.patch > > > I've seen this for a long time, and imagine it's a known issue, but couldn't > find an existing JIRA. It often happens that we see the NN schedule > replication on the last block of files very quickly after they're completed, > before the other DNs in the pipeline have a chance to report the new block. > This results in a lot of extra replication work on the cluster, as we > replicate the block and then end up with multiple excess replicas which are > very quickly deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7087) Ability to list /.reserved
[ https://issues.apache.org/jira/browse/HDFS-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-7087: Attachment: (was: HDFS-7087.patch.draft) > Ability to list /.reserved > -- > > Key: HDFS-7087 > URL: https://issues.apache.org/jira/browse/HDFS-7087 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Andrew Wang >Assignee: Xiao Chen > Attachments: HDFS-7087.draft.patch > > > We have two special paths within /.reserved now, /.reserved/.inodes and > /.reserved/raw. It seems like we should be able to list /.reserved to see > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7087) Ability to list /.reserved
[ https://issues.apache.org/jira/browse/HDFS-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-7087: Attachment: HDFS-7087.draft.patch > Ability to list /.reserved > -- > > Key: HDFS-7087 > URL: https://issues.apache.org/jira/browse/HDFS-7087 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Andrew Wang >Assignee: Xiao Chen > Attachments: HDFS-7087.draft.patch > > > We have two special paths within /.reserved now, /.reserved/.inodes and > /.reserved/raw. It seems like we should be able to list /.reserved to see > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8485) Transparent Encryption Fails to work with Yarn/MapReduce
[ https://issues.apache.org/jira/browse/HDFS-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944107#comment-14944107 ] Prasad Alle commented on HDFS-8485: --- Can you please describe fix in more details. > Transparent Encryption Fails to work with Yarn/MapReduce > > > Key: HDFS-8485 > URL: https://issues.apache.org/jira/browse/HDFS-8485 > Project: Hadoop HDFS > Issue Type: Bug > Environment: RHEL-7, Kerberos 5 >Reporter: Ambud Sharma >Priority: Critical > Attachments: core-site.xml, hdfs-site.xml, kms-site.xml, > mapred-site.xml, yarn-site.xml > > > Running a simple MapReduce job that writes to a path configured as an > encryption zone throws exception > 11:26:26,343 INFO [org.apache.hadoop.mapreduce.Job] (pool-14-thread-1) Task > Id : attempt_1432740034176_0001_m_00_2, Status : FAILED > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:424) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:710) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1358) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1457) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1442) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:400) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > com.s3.ingestion.S3ImportMR$S3ImportMapper.map(S3ImportMR.java:112) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > com.s3.ingestion.S3ImportMR$S3ImportMapper.map(S3ImportMR.java:43) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > java.security.AccessController.doPrivileged(Native Method) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > javax.security.auth.Subject.doAs(Subject.java:422) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) > 11:
[jira] [Commented] (HDFS-8485) Transparent Encryption Fails to work with Yarn/MapReduce
[ https://issues.apache.org/jira/browse/HDFS-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944109#comment-14944109 ] Prasad Alle commented on HDFS-8485: --- I am getting the following exception when I read data from the HDFS Encryption zone either through Java MR or Spark Diagnostics: No KeyProvider is configured, cannot access an encrypted file java.io.IOException: No KeyProvider is configured, cannot access an encrypted file at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1352) > Transparent Encryption Fails to work with Yarn/MapReduce > > > Key: HDFS-8485 > URL: https://issues.apache.org/jira/browse/HDFS-8485 > Project: Hadoop HDFS > Issue Type: Bug > Environment: RHEL-7, Kerberos 5 >Reporter: Ambud Sharma >Priority: Critical > Attachments: core-site.xml, hdfs-site.xml, kms-site.xml, > mapred-site.xml, yarn-site.xml > > > Running a simple MapReduce job that writes to a path configured as an > encryption zone throws exception > 11:26:26,343 INFO [org.apache.hadoop.mapreduce.Job] (pool-14-thread-1) Task > Id : attempt_1432740034176_0001_m_00_2, Status : FAILED > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:424) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:710) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1358) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1457) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1442) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:400) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > com.s3.ingestion.S3ImportMR$S3ImportMapper.map(S3ImportMR.java:112) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > com.s3.ingestion.S3ImportMR$S3ImportMapper.map(S3ImportMR.java:43) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > java.security.AccessController.doPrivileged(Native Method) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > javax.security.auth.Subject.doAs(Subject.java:422) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1)at > org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) Caused by: > org.apache.hadoop.security.auth
[jira] [Commented] (HDFS-9198) Coalesce IBR processing in the NN
[
https://issues.apache.org/jira/browse/HDFS-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944117#comment-14944117
]
Hadoop QA commented on HDFS-9198:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 22s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 25s | The applied patch generated 6
new checkstyle issues (total was 424, now 425). |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 33s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 200m 1s | Tests failed in hadoop-hdfs. |
| | | 246m 16s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestParallelRead |
| | hadoop.hdfs.server.namenode.TestDeadDatanode |
| | hadoop.hdfs.web.TestWebHdfsContentLength |
| | hadoop.hdfs.server.blockmanagement.TestPendingReplication |
| | hadoop.hdfs.server.datanode.TestIncrementalBrVariations |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765032/HDFS-9198-trunk.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b925cf1 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12794/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12794/console |
This message was automatically generated.
> Coalesce IBR processing in the NN
> -
>
> Key: HDFS-9198
> URL: https://issues.apache.org/jira/browse/HDFS-9198
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-9198-branch2.patch, HDFS-9198-trunk.patch
>
>
> IBRs from thousands of DNs under load will degrade NN performance due to
> excessive write-lock contention from multiple IPC handler threads. The IBR
> processing is quick, so the lock contention may be reduced by coalescing
> multiple IBRs into a single write-lock transaction. The handlers will also
> be freed up faster for other operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9155) OEV should treat .XML files as XML even when the file name extension is uppercase
[
https://issues.apache.org/jira/browse/HDFS-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944145#comment-14944145
]
Hudson commented on HDFS-9155:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1221 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1221/])
HDFS-9155. OEV should treat .XML files as XML even when the file name (cmccabe:
rev 2fc2b50534c2052ed3f6f38f30d29d98d99fd10a)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsViewer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> OEV should treat .XML files as XML even when the file name extension is
> uppercase
> -
>
> Key: HDFS-9155
> URL: https://issues.apache.org/jira/browse/HDFS-9155
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: nijel
> Fix For: 2.8.0
>
> Attachments: HDFS-9155_01.patch
>
>
> As in document and help
> {noformat}
> -i,--inputFileedits file to process, xml (*case
>insensitive*) extension means XML format,
> {noformat}
> But if i give the file with "XML" extension it falls back to binary
> processing.
> This issue is due the code
> {code}
> int org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.go()
> .
> boolean xmlInput = inputFileName.endsWith(".xml");
> {code}
> Here need to check the xml after converting the file name to lower case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
