[jira] [Resolved] (HDFS-4925) TestQuorumJournalManager.testPurgeLogs intermittently Fails assertNoThreadsMatching
[
https://issues.apache.org/jira/browse/HDFS-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA resolved HDFS-4925.
-
Resolution: Duplicate
Fix Version/s: (was: 2.6.5)
(was: 2.7.3)
(was: 2.8.0)
> TestQuorumJournalManager.testPurgeLogs intermittently Fails
> assertNoThreadsMatching
> ---
>
> Key: HDFS-4925
> URL: https://issues.apache.org/jira/browse/HDFS-4925
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm, test
>Affects Versions: 3.0.0
>Reporter: Tao Luo
>Priority: Minor
> Attachments: TestQuorumJournalManager.rtf
>
>
> On Jenkins, I saw this assert fail in the @After shutdown() for
> TestQuorumJournalManager.testPurgeLogs():
> {code}// Should not leak clients between tests -- this can cause flaky
> tests.
> // (See HDFS-4643)
> GenericTestUtils.assertNoThreadsMatching(".*IPC Client.*");
> {code}
> https://builds.apache.org/job/PreCommit-HDFS-Build/4546//testReport/
> I could not reproduce locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4925) TestQuorumJournalManager.testPurgeLogs intermittently Fails assertNoThreadsMatching
[
https://issues.apache.org/jira/browse/HDFS-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140461#comment-15140461
]
Akira AJISAKA commented on HDFS-4925:
-
Thanks [~jojochuang] for fixing the test. I'll reopen and (re)close it to
change the resolution from "Fixed" to "Duplicate".
> TestQuorumJournalManager.testPurgeLogs intermittently Fails
> assertNoThreadsMatching
> ---
>
> Key: HDFS-4925
> URL: https://issues.apache.org/jira/browse/HDFS-4925
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm, test
>Affects Versions: 3.0.0
>Reporter: Tao Luo
>Priority: Minor
> Attachments: TestQuorumJournalManager.rtf
>
>
> On Jenkins, I saw this assert fail in the @After shutdown() for
> TestQuorumJournalManager.testPurgeLogs():
> {code}// Should not leak clients between tests -- this can cause flaky
> tests.
> // (See HDFS-4643)
> GenericTestUtils.assertNoThreadsMatching(".*IPC Client.*");
> {code}
> https://builds.apache.org/job/PreCommit-HDFS-Build/4546//testReport/
> I could not reproduce locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HDFS-4925) TestQuorumJournalManager.testPurgeLogs intermittently Fails assertNoThreadsMatching
[
https://issues.apache.org/jira/browse/HDFS-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reopened HDFS-4925:
-
> TestQuorumJournalManager.testPurgeLogs intermittently Fails
> assertNoThreadsMatching
> ---
>
> Key: HDFS-4925
> URL: https://issues.apache.org/jira/browse/HDFS-4925
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm, test
>Affects Versions: 3.0.0
>Reporter: Tao Luo
>Priority: Minor
> Attachments: TestQuorumJournalManager.rtf
>
>
> On Jenkins, I saw this assert fail in the @After shutdown() for
> TestQuorumJournalManager.testPurgeLogs():
> {code}// Should not leak clients between tests -- this can cause flaky
> tests.
> // (See HDFS-4643)
> GenericTestUtils.assertNoThreadsMatching(".*IPC Client.*");
> {code}
> https://builds.apache.org/job/PreCommit-HDFS-Build/4546//testReport/
> I could not reproduce locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-4925) TestQuorumJournalManager.testPurgeLogs intermittently Fails assertNoThreadsMatching
[
https://issues.apache.org/jira/browse/HDFS-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HDFS-4925.
---
Resolution: Fixed
Fix Version/s: 2.6.5
2.7.3
2.8.0
I am going to resolve this as fixed. The assertion failure was fixed via
HDFS-9347.
> TestQuorumJournalManager.testPurgeLogs intermittently Fails
> assertNoThreadsMatching
> ---
>
> Key: HDFS-4925
> URL: https://issues.apache.org/jira/browse/HDFS-4925
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm, test
>Affects Versions: 3.0.0
>Reporter: Tao Luo
>Priority: Minor
> Fix For: 2.8.0, 2.7.3, 2.6.5
>
> Attachments: TestQuorumJournalManager.rtf
>
>
> On Jenkins, I saw this assert fail in the @After shutdown() for
> TestQuorumJournalManager.testPurgeLogs():
> {code}// Should not leak clients between tests -- this can cause flaky
> tests.
> // (See HDFS-4643)
> GenericTestUtils.assertNoThreadsMatching(".*IPC Client.*");
> {code}
> https://builds.apache.org/job/PreCommit-HDFS-Build/4546//testReport/
> I could not reproduce locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140396#comment-15140396 ] Guocui Mi commented on HDFS-9787: - Agree with you on this, should create one new workitem for this. 1> Non-primary SNNs check ANN checkpoint version via RPC; 2> Check if checkpoint period beyond threshold Yes : try to upload checkpoint to ANN. become primary SNN if success. 3> repeat 1> 2>. This patch is just a simple bug fix to make it work as expected first. > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > Attachments: HDFS-9786-v000.patch > > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140384#comment-15140384 ] Jesse Yates commented on HDFS-9787: --- The original solution was an attempting to catch the case where we don't flood the NN with checkpoint requests. Instead, maybe the better solution would be to do a small RPC to see when the latest image was uploaded. If it was uploaded the quietMultiplier beyond the checkpoint period, then we attempt to upload the checkpoint. Its a bit more work, but I think this more clearly lays out the intentions in the code, rather than obtaining the same effect, but without the overhead of actually sending the checkpoint along each time we want to find out if its behind. > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > Attachments: HDFS-9786-v000.patch > > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[
https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140372#comment-15140372
]
Guocui Mi commented on HDFS-9787:
-
lastCheckpointTime is also used to check if it is necessary to do another
checkpoint. we can't mix up last checkpoint time and last upload time in order
to honor dfs.namenode.checkpoint.period
FYI: else if (secsSinceLast >= checkpointConf.getPeriod()) {
> SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to
> false.
> ---
>
> Key: HDFS-9787
> URL: https://issues.apache.org/jira/browse/HDFS-9787
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 3.0.0
>Reporter: Guocui Mi
>Assignee: Guocui Mi
> Attachments: HDFS-9786-v000.patch
>
>
> SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false.
> Here is the logic to check if upload FSImage or not.
> In StandbyCheckpointer.java
> boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >=
> checkpointConf.getQuietPeriod();
> doCheckpoint(sendRequest);
> The sendRequest is always false if isPrimaryCheckPointer is false giving
> secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod()
> (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns
> false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[
https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140367#comment-15140367
]
Guocui Mi commented on HDFS-9787:
-
>>> this would imply that the non-primary SNN never sends a checkpoint after
>>> the first time?
It is true according to my observation.
I am trying to add unittest to cover the scenario. Another two scenarios
triggered in our cluster:
1) PrimaryCheckpoint uploading fsimage failure due to ANN not available
temporarily.
2) Restart all NNs at same time.
I afraid the proposal you shared can't work.
1) set lastCheckpointTime before following code in doCheckpoint(): no
difference between putting after each loop iteration.
2) after following code in doCheckpoint() : Non-primary SNN will do checkpoint
one by one continuously since lastCheckpointTime not get updated.
if(!sendCheckpoint){ return;}
> SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to
> false.
> ---
>
> Key: HDFS-9787
> URL: https://issues.apache.org/jira/browse/HDFS-9787
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 3.0.0
>Reporter: Guocui Mi
>Assignee: Guocui Mi
> Attachments: HDFS-9786-v000.patch
>
>
> SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false.
> Here is the logic to check if upload FSImage or not.
> In StandbyCheckpointer.java
> boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >=
> checkpointConf.getQuietPeriod();
> doCheckpoint(sendRequest);
> The sendRequest is always false if isPrimaryCheckPointer is false giving
> secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod()
> (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns
> false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[ https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-9637: --- Attachment: HDFS-9637.006.patch Here's one more little update. I noticed some formatting that could be cleaner, and I moved the flag resets to the setup method. > Add test for HADOOP-12702 and HADOOP-12759 > -- > > Key: HDFS-9637 > URL: https://issues.apache.org/jira/browse/HDFS-9637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9637.001.patch, HDFS-9637.002.patch, > HDFS-9637.003.patch, HDFS-9637.004.patch, HDFS-9637.005.patch, > HDFS-9637.006.patch > > > Per discussion on the dev list, the tests for the new FileSystemSink class > should be added to the HDFS project to avoid creating a dependency for the > common project on the HDFS project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9780) RollingFileSystemSink doesn't work on secure clusters
[
https://issues.apache.org/jira/browse/HDFS-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140277#comment-15140277
]
Karthik Kambatla commented on HDFS-9780:
Quickly skimmed through the patch.
Do we want to checkForProperty even if security is not enabled?
{code}
// Validate config so that we don't get an NPE
checkForProperty(conf, KEYTAB_PROPERTY_KEY);
checkForProperty(conf, USERNAME_PROPERTY_KEY);
{code}
Once the patch is updated based on HDFS-9637, will take a closer look.
> RollingFileSystemSink doesn't work on secure clusters
> -
>
> Key: HDFS-9780
> URL: https://issues.apache.org/jira/browse/HDFS-9780
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: HADOOP-12775.001.patch, HADOOP-12775.002.patch,
> HADOOP-12775.003.patch, HDFS-9780.004.patch
>
>
> If HDFS has kerberos enabled, the sink cannot write its logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9780) RollingFileSystemSink doesn't work on secure clusters
[ https://issues.apache.org/jira/browse/HDFS-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HDFS-9780: --- Status: Open (was: Patch Available) Canceling patch until HDFS-9637 gets committed. > RollingFileSystemSink doesn't work on secure clusters > - > > Key: HDFS-9780 > URL: https://issues.apache.org/jira/browse/HDFS-9780 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: HADOOP-12775.001.patch, HADOOP-12775.002.patch, > HADOOP-12775.003.patch, HDFS-9780.004.patch > > > If HDFS has kerberos enabled, the sink cannot write its logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[
https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140258#comment-15140258
]
Hadoop QA commented on HDFS-9787:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
26 unchanged - 1 fixed = 27 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 54s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 41s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 22s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.fs.TestHdfsNativeCodeLoader |
| JDK v1.7.0_95 Failed junit tests | hadoop.fs.TestHdfsNativeCodeLoader |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787193/HDFS-9786-v000.patch |
| JIRA Issue | HDFS-9787 |
| Optional Tests | asflicense compile javac javad
[jira] [Commented] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[ https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140252#comment-15140252 ] Karthik Kambatla commented on HDFS-9637: Looks good. +1. Will check this in tomorrow morning if I don't hear any objections. > Add test for HADOOP-12702 and HADOOP-12759 > -- > > Key: HDFS-9637 > URL: https://issues.apache.org/jira/browse/HDFS-9637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9637.001.patch, HDFS-9637.002.patch, > HDFS-9637.003.patch, HDFS-9637.004.patch, HDFS-9637.005.patch > > > Per discussion on the dev list, the tests for the new FileSystemSink class > should be added to the HDFS project to avoid creating a dependency for the > common project on the HDFS project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[
https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140247#comment-15140247
]
Hadoop QA commented on HDFS-9637:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 23s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 5s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 21s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.fs.TestHdfsNativeCodeLoader |
| JDK v1.7.0_95 Failed junit tests | hadoop.fs.TestHdfsNativeCodeLoader |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787194/HDFS-9637.005.patch |
| JIRA Issue | HDFS-9637 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 18ad0c04c7cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_6
[jira] [Updated] (HDFS-9703) DiskBalancer : getBandwidth implementation
[ https://issues.apache.org/jira/browse/HDFS-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-9703: --- Attachment: HDFS-9703-HDFS-1312.001.patch > DiskBalancer : getBandwidth implementation > -- > > Key: HDFS-9703 > URL: https://issues.apache.org/jira/browse/HDFS-9703 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-9703-HDFS-1312.001.patch > > > Add getBandwidth call -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9703) DiskBalancer : getBandwidth implementation
[ https://issues.apache.org/jira/browse/HDFS-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-9703: --- Attachment: (was: HDFS-9703-HDFS-1312.001.patch) > DiskBalancer : getBandwidth implementation > -- > > Key: HDFS-9703 > URL: https://issues.apache.org/jira/browse/HDFS-9703 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-9703-HDFS-1312.001.patch > > > Add getBandwidth call -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9779) TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value
[
https://issues.apache.org/jira/browse/HDFS-9779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140221#comment-15140221
]
Kuhu Shukla commented on HDFS-9779:
---
Thank you Uma!
> TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value
> ---
>
> Key: HDFS-9779
> URL: https://issues.apache.org/jira/browse/HDFS-9779
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.2
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9779.001.patch
>
>
> The DatanodeDescriptor object for {{NODE}} calls the wrong constructor on an
> already created DatanodeDescriptor object. This overwrites the
> networkLocation value to "default-rack" instead of expected value of
> "/d2/r4/n7". I didn't find any other such malformed calls to
> {{DFSTestUtil.getDatanodeDescriptor}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[
https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140220#comment-15140220
]
Inigo Goiri commented on HDFS-9787:
---
My bad, this is right.
We will test resetting the {{lastCheckpointTime}} in every loop.
> SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to
> false.
> ---
>
> Key: HDFS-9787
> URL: https://issues.apache.org/jira/browse/HDFS-9787
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 3.0.0
>Reporter: Guocui Mi
>Assignee: Guocui Mi
> Attachments: HDFS-9786-v000.patch
>
>
> SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false.
> Here is the logic to check if upload FSImage or not.
> In StandbyCheckpointer.java
> boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >=
> checkpointConf.getQuietPeriod();
> doCheckpoint(sendRequest);
> The sendRequest is always false if isPrimaryCheckPointer is false giving
> secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod()
> (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns
> false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[
https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140218#comment-15140218
]
Inigo Goiri commented on HDFS-9787:
---
Now that I see this, can it be the problem that we do:
{code}
// Reset checkpoint time so that we don't always checkpoint
// on startup.
lastCheckpointTime = monotonicNow();
{code}
And then:
{code}
lastCheckpointTime = now;
{code}
We'll do the test in our cluster anyway.
> SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to
> false.
> ---
>
> Key: HDFS-9787
> URL: https://issues.apache.org/jira/browse/HDFS-9787
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 3.0.0
>Reporter: Guocui Mi
>Assignee: Guocui Mi
> Attachments: HDFS-9786-v000.patch
>
>
> SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false.
> Here is the logic to check if upload FSImage or not.
> In StandbyCheckpointer.java
> boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >=
> checkpointConf.getQuietPeriod();
> doCheckpoint(sendRequest);
> The sendRequest is always false if isPrimaryCheckPointer is false giving
> secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod()
> (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns
> false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9711) Integrate CSRF prevention filter in WebHDFS.
[ https://issues.apache.org/jira/browse/HDFS-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140195#comment-15140195 ] Larry McCay commented on HDFS-9711: --- Hi [~cnauroth] - Looks great! The effort to add a filter for webhdfs is greater than I anticipated. a couple quick things: * I like the refactoring for an isRequestAllowed method on the filter - I actually meant to go back and do that earlier * I notice that you have to return your own error message in the channelRead0 method of RestCsrfPreventionFilterHandler. Perhaps, we should provide a constant for that in the filter too. As it stands now, the message you return is slightly different and a bit more ambiguous then what is returned by the filter itself (which is why I changed it). * I'd also like to understand why the typical filter processing isn't being used in this code path. Not because I think it should but I'd like to understand the usecase here. > Integrate CSRF prevention filter in WebHDFS. > > > Key: HDFS-9711 > URL: https://issues.apache.org/jira/browse/HDFS-9711 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode, webhdfs >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-9711.001.patch, HDFS-9711.002.patch, > HDFS-9711.003.patch > > > HADOOP-12691 introduced a filter in Hadoop Common to help REST APIs guard > against cross-site request forgery attacks. This issue tracks integration of > that filter in WebHDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140176#comment-15140176 ] Jesse Yates commented on HDFS-9787: --- I don't think you need a new variable - lastCheckpointTime is never used outside of the class and only used for for this check > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > Attachments: HDFS-9786-v000.patch > > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140175#comment-15140175 ] Jesse Yates commented on HDFS-9787: --- Taking a quick look, this would imply that the non-primary SNN never sends a checkpoint after the first time? A good test to ensure that this is the case is to start the NNs, wait until there primary SNN is selected and then remove it from the cluster. Are any more checkpoints sent to the ANN? My inclination is that you are correct, no (unless it takes a long time to build the checkpoint), but I'd like to hear if that's actually the case. I think the fix is to just set lastCheckpointTime in doCheckpoint() rather than after each loop iteration. > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > Attachments: HDFS-9786-v000.patch > > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-9787: -- Status: Patch Available (was: Open) > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > Attachments: HDFS-9786-v000.patch > > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140162#comment-15140162 ] Inigo Goiri commented on HDFS-9787: --- Thanks! > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > Attachments: HDFS-9786-v000.patch > > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[
https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated HDFS-9637:
---
Attachment: HDFS-9637.005.patch
Huh. Good point, [~kasha]. At one time there was a reason for not using
{{\@Before}} and {{\@After}}, but clearly that's no longer the case... This
new patch removes the try-finally blocks.
> Add test for HADOOP-12702 and HADOOP-12759
> --
>
> Key: HDFS-9637
> URL: https://issues.apache.org/jira/browse/HDFS-9637
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: HDFS-9637.001.patch, HDFS-9637.002.patch,
> HDFS-9637.003.patch, HDFS-9637.004.patch, HDFS-9637.005.patch
>
>
> Per discussion on the dev list, the tests for the new FileSystemSink class
> should be added to the HDFS project to avoid creating a dependency for the
> common project on the HDFS project.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guocui Mi updated HDFS-9787: Attachment: HDFS-9786-v000.patch check last upload time instead of last check point time. > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > Attachments: HDFS-9786-v000.patch > > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLastUpload >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guocui Mi updated HDFS-9787: Description: SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. Here is the logic to check if upload FSImage or not. In StandbyCheckpointer.java boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >= checkpointConf.getQuietPeriod(); doCheckpoint(sendRequest); The sendRequest is always false if isPrimaryCheckPointer is false giving secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns false. was: SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. Here is the logic to check if upload FSImage or not. In StandbyCheckpointer.java boolean sendRequest = isPrimaryCheckPointer || secsSinceLastUpload >= checkpointConf.getQuietPeriod(); doCheckpoint(sendRequest); The sendRequest is always false if isPrimaryCheckPointer is false giving secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns false. > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > Attachments: HDFS-9786-v000.patch > > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLast >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140148#comment-15140148 ] Jesse Yates commented on HDFS-9787: --- I think I'll have some time tomorrow - adding it to my list > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLastUpload >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140146#comment-15140146 ] Inigo Goiri commented on HDFS-9787: --- [~jesse_yates], do you mind taking a look at this? > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLastUpload >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guocui Mi updated HDFS-9787: Status: Open (was: Patch Available) > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLastUpload >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guocui Mi updated HDFS-9787: Assignee: Guocui Mi Status: Patch Available (was: Open) check time since last upload instead of time since last checkpoint. > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi >Assignee: Guocui Mi > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLastUpload >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
[ https://issues.apache.org/jira/browse/HDFS-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-9787: -- Affects Version/s: (was: 2.6.2) 3.0.0 > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to > false. > --- > > Key: HDFS-9787 > URL: https://issues.apache.org/jira/browse/HDFS-9787 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0 >Reporter: Guocui Mi > > SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. > Here is the logic to check if upload FSImage or not. > In StandbyCheckpointer.java > boolean sendRequest = isPrimaryCheckPointer || secsSinceLastUpload >= > checkpointConf.getQuietPeriod(); > doCheckpoint(sendRequest); > The sendRequest is always false if isPrimaryCheckPointer is false giving > secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() > (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns > false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9711) Integrate CSRF prevention filter in WebHDFS.
[ https://issues.apache.org/jira/browse/HDFS-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140135#comment-15140135 ] Chris Nauroth commented on HDFS-9711: - The last round of JDK 8 test failures are unrelated. I cannot repro. > Integrate CSRF prevention filter in WebHDFS. > > > Key: HDFS-9711 > URL: https://issues.apache.org/jira/browse/HDFS-9711 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode, webhdfs >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-9711.001.patch, HDFS-9711.002.patch, > HDFS-9711.003.patch > > > HADOOP-12691 introduced a filter in Hadoop Common to help REST APIs guard > against cross-site request forgery attacks. This issue tracks integration of > that filter in WebHDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[ https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140132#comment-15140132 ] Karthik Kambatla commented on HDFS-9637: Reviewing the tests here, following up from the feature added in Common. The patch looks pretty good. Nice tests and very happy to see the detailed documentation. My only comment would be: use @Before and @After methods to do the setup and cleanup, that way we don't have to do try-finally blocks. > Add test for HADOOP-12702 and HADOOP-12759 > -- > > Key: HDFS-9637 > URL: https://issues.apache.org/jira/browse/HDFS-9637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9637.001.patch, HDFS-9637.002.patch, > HDFS-9637.003.patch, HDFS-9637.004.patch > > > Per discussion on the dev list, the tests for the new FileSystemSink class > should be added to the HDFS project to avoid creating a dependency for the > common project on the HDFS project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9787) SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false.
Guocui Mi created HDFS-9787: --- Summary: SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false. Key: HDFS-9787 URL: https://issues.apache.org/jira/browse/HDFS-9787 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: 2.6.2 Reporter: Guocui Mi SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer become false. Here is the logic to check if upload FSImage or not. In StandbyCheckpointer.java boolean sendRequest = isPrimaryCheckPointer || secsSinceLastUpload >= checkpointConf.getQuietPeriod(); doCheckpoint(sendRequest); The sendRequest is always false if isPrimaryCheckPointer is false giving secsSinceLast (~checkpointPeriod) >= checkpointConf.getQuietPeriod() (checkpointPeriod * this.quietMultiplier(default value 1.5)) always returns false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9711) Integrate CSRF prevention filter in WebHDFS.
[
https://issues.apache.org/jira/browse/HDFS-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140056#comment-15140056
]
Hadoop QA commented on HDFS-9711:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 11s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} root: patch generated 0 new + 107 unchanged - 5
fixed = 107 total (was 112) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 0s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 26s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 21s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s
{color} | {color:green} hadoop-hdfs-clie
[jira] [Commented] (HDFS-9779) TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value
[
https://issues.apache.org/jira/browse/HDFS-9779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140030#comment-15140030
]
Hudson commented on HDFS-9779:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #9271 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9271/])
HDFS-9779 . TestReplicationPolicyWithNodeGroup NODE variable picks wrong
(uma.gangumalla: rev a7fce9ab415fcc834ca1d91f913157ffd103fa5f)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value
> ---
>
> Key: HDFS-9779
> URL: https://issues.apache.org/jira/browse/HDFS-9779
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.2
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9779.001.patch
>
>
> The DatanodeDescriptor object for {{NODE}} calls the wrong constructor on an
> already created DatanodeDescriptor object. This overwrites the
> networkLocation value to "default-rack" instead of expected value of
> "/d2/r4/n7". I didn't find any other such malformed calls to
> {{DFSTestUtil.getDatanodeDescriptor}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7955) Improve naming of classes, methods, and variables related to block replication and recovery
[
https://issues.apache.org/jira/browse/HDFS-7955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139998#comment-15139998
]
Uma Maheswara Rao G commented on HDFS-7955:
---
Thanks [~rakeshr] for the work. I think we can close this now. Do you have any
other sub tasks planned?
> Improve naming of classes, methods, and variables related to block
> replication and recovery
> ---
>
> Key: HDFS-7955
> URL: https://issues.apache.org/jira/browse/HDFS-7955
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Reporter: Zhe Zhang
>Assignee: Rakesh R
> Attachments: HDFS-7955-001.patch, HDFS-7955-002.patch,
> HDFS-7955-003.patch, HDFS-7955-004.patch, HDFS-7955-5.patch
>
>
> Many existing names should be revised to avoid confusion when blocks can be
> both replicated and erasure coded. This JIRA aims to solicit opinions on
> making those names more consistent and intuitive.
> # In current HDFS _block recovery_ refers to the process of finalizing the
> last block of a file, triggered by _lease recovery_. It is different from the
> intuitive meaning of _recovering a lost block_. To avoid confusion, I can
> think of 2 options:
> #* Rename this process as _block finalization_ or _block completion_. I
> prefer this option because this is literally not a recovery.
> #* If we want to keep existing terms unchanged we can name all EC recovery
> and re-replication logics as _reconstruction_.
> # As Kai [suggested |
> https://issues.apache.org/jira/browse/HDFS-7369?focusedCommentId=14361131&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14361131]
> under HDFS-7369, several replication-based names should be made more generic:
> #* {{UnderReplicatedBlocks}} and {{neededReplications}}. E.g. we can use
> {{LowRedundancyBlocks}}/{{AtRiskBlocks}}, and
> {{neededRecovery}}/{{neededReconstruction}}.
> #* {{PendingReplicationBlocks}}
> #* {{ReplicationMonitor}}
> I'm sure the above list is incomplete; discussions and comments are very
> welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9779) TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value
[
https://issues.apache.org/jira/browse/HDFS-9779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-9779:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Thanks for the contribution [~kshukla]. Thanks for the reviews [~shahrs87]
+1, I have just pushed it to trunk, branch-2 and 2.8
> TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value
> ---
>
> Key: HDFS-9779
> URL: https://issues.apache.org/jira/browse/HDFS-9779
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.2
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9779.001.patch
>
>
> The DatanodeDescriptor object for {{NODE}} calls the wrong constructor on an
> already created DatanodeDescriptor object. This overwrites the
> networkLocation value to "default-rack" instead of expected value of
> "/d2/r4/n7". I didn't find any other such malformed calls to
> {{DFSTestUtil.getDatanodeDescriptor}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9775) Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork
[ https://issues.apache.org/jira/browse/HDFS-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139971#comment-15139971 ] Hudson commented on HDFS-9775: -- FAILURE: Integrated in Hadoop-trunk-Commit #9270 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9270/]) HDFS-9775. Erasure Coding : Rename BlockRecoveryWork to (zhz: rev a0fb2eff9b71e2e2c0e53262773b34bed82585d4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicationWork.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockReconstructionWork.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ErasureCodingWork.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockRecoveryWork.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork > > > Key: HDFS-9775 > URL: https://issues.apache.org/jira/browse/HDFS-9775 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Rakesh R >Assignee: Rakesh R > Fix For: 3.0.0 > > Attachments: HDFS-9775-01.patch > > > This sub-task is to visit the block recovery work and make the logic as > reconstruction. ie, rename "recovery" to "reconstruction" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9702) DiskBalancer : getVolumeMap implementation
[ https://issues.apache.org/jira/browse/HDFS-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-9702: --- Attachment: HDFS-9702-HDFS-1312.001.patch > DiskBalancer : getVolumeMap implementation > -- > > Key: HDFS-9702 > URL: https://issues.apache.org/jira/browse/HDFS-9702 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-9702-HDFS-1312.001.patch > > > Add get volume map -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9786) HttpFS doesn't write the proxyuser information in logfile
HeeSoo Kim created HDFS-9786:
Summary: HttpFS doesn't write the proxyuser information in logfile
Key: HDFS-9786
URL: https://issues.apache.org/jira/browse/HDFS-9786
Project: Hadoop HDFS
Issue Type: Bug
Reporter: HeeSoo Kim
Assignee: HeeSoo Kim
According to the httpfs-log4j.properties, the log pattern indicates that
{code}
log4j.appender.httpfsaudit.layout.ConversionPattern=%d{ISO8601} %5p
[%X{hostname}][%X{user}:%X{doAs}] %X{op} %m%n
{code}
However, the httpfsaudit doesn't write right information for user and proxyuser
information. It is better to write ugi on audit log in HttpFS GW.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9702) DiskBalancer : getVolumeMap implementation
[ https://issues.apache.org/jira/browse/HDFS-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-9702: --- Attachment: (was: HDFS-9702-HDFS-1312.001.patch) > DiskBalancer : getVolumeMap implementation > -- > > Key: HDFS-9702 > URL: https://issues.apache.org/jira/browse/HDFS-9702 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-9702-HDFS-1312.001.patch > > > Add get volume map -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9775) Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork
[ https://issues.apache.org/jira/browse/HDFS-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-9775: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork > > > Key: HDFS-9775 > URL: https://issues.apache.org/jira/browse/HDFS-9775 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Rakesh R >Assignee: Rakesh R > Fix For: 3.0.0 > > Attachments: HDFS-9775-01.patch > > > This sub-task is to visit the block recovery work and make the logic as > reconstruction. ie, rename "recovery" to "reconstruction" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9775) Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork
[ https://issues.apache.org/jira/browse/HDFS-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139925#comment-15139925 ] Zhe Zhang commented on HDFS-9775: - +1 on the patch. I just committed it to trunk. Thanks Rakesh for the work! > Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork > > > Key: HDFS-9775 > URL: https://issues.apache.org/jira/browse/HDFS-9775 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Rakesh R >Assignee: Rakesh R > Fix For: 3.0.0 > > Attachments: HDFS-9775-01.patch > > > This sub-task is to visit the block recovery work and make the logic as > reconstruction. ie, rename "recovery" to "reconstruction" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9760) WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler
[ https://issues.apache.org/jira/browse/HDFS-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139896#comment-15139896 ] Hudson commented on HDFS-9760: -- FAILURE: Integrated in Hadoop-trunk-Commit #9269 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9269/]) HDFS-9760. WebHDFS AuthFilter cannot be configured with custom (aw: rev 401ae4ecdb64e1ae2730976f96f7949831305c07) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestAuthFilter.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler > > > Key: HDFS-9760 > URL: https://issues.apache.org/jira/browse/HDFS-9760 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Ryan Sasson >Assignee: Ryan Sasson > Fix For: 2.8.0 > > Attachments: HDFS-9760.patch > > > Currently the WebHDFS AuthFilter selects its authentication type based on a > call to UserGroupInformation.isSecurityEnabled() with only two choices, > KerberosAuthentication or PsuedoAuthentication. Thus there is no condition > where the WebHDFS server can be configured with a custom AltKerberos > authentication handler. > Additionally, at the time the WebHDFS AuthFilter is initialized the method > getAuthFilterParams(conf) is called in NameNodeHttpServer which picks and > chooses a certain few configurations with the prefix > 'dfs.web.authentication'. The issue is this method strips away the > configuration that could set the authentication type AND additional > configurations that are specific to the custom auth handler (using the prefix > 'dfs.web.authentication.alt-kerberos'). > The consequence of this lack of configurability is that a user that makes > authenticated access to the namenode web UI (through a custom authentication > handler) will not be able to access the namenode file browser (because it is > making ajax calls to WebHDFS that has a different authentication type). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9760) WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler
[ https://issues.apache.org/jira/browse/HDFS-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-9760: --- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) +1 committed to 2.8 thanks! > WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler > > > Key: HDFS-9760 > URL: https://issues.apache.org/jira/browse/HDFS-9760 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Ryan Sasson >Assignee: Ryan Sasson > Fix For: 2.8.0 > > Attachments: HDFS-9760.patch > > > Currently the WebHDFS AuthFilter selects its authentication type based on a > call to UserGroupInformation.isSecurityEnabled() with only two choices, > KerberosAuthentication or PsuedoAuthentication. Thus there is no condition > where the WebHDFS server can be configured with a custom AltKerberos > authentication handler. > Additionally, at the time the WebHDFS AuthFilter is initialized the method > getAuthFilterParams(conf) is called in NameNodeHttpServer which picks and > chooses a certain few configurations with the prefix > 'dfs.web.authentication'. The issue is this method strips away the > configuration that could set the authentication type AND additional > configurations that are specific to the custom auth handler (using the prefix > 'dfs.web.authentication.alt-kerberos'). > The consequence of this lack of configurability is that a user that makes > authenticated access to the namenode web UI (through a custom authentication > handler) will not be able to access the namenode file browser (because it is > making ajax calls to WebHDFS that has a different authentication type). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.
[ https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-9311: Release Note: There is now support for offloading HA health check RPC activity to a separate RPC server endpoint running within the NameNode process. This may improve reliability of HA health checks and prevent spurious failovers in highly overloaded conditions. For more details, please refer to the hdfs-default.xml documentation for properties dfs.namenode.lifeline.rpc-address, dfs.namenode.lifeline.rpc-bind-host and dfs.namenode.lifeline.handler.count. > Support optional offload of NameNode HA service health checks to a separate > RPC server. > --- > > Key: HDFS-9311 > URL: https://issues.apache.org/jira/browse/HDFS-9311 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, > HDFS-9311.003.patch > > > When a NameNode is overwhelmed with load, it can lead to resource exhaustion > of the RPC handler pools (both client-facing and service-facing). > Eventually, this blocks the health check RPC issued from ZKFC, which triggers > a failover. Depending on fencing configuration, the former active NameNode > may be killed. In an overloaded situation, the new active NameNode is likely > to suffer the same fate, because client load patterns don't change after the > failover. This can degenerate into flapping between the 2 NameNodes without > real recovery. If a NameNode had been killed by fencing, then it would have > to transition through safe mode, further delaying time to recovery. > This issue proposes a separate, optional RPC server at the NameNode for > isolating the HA health checks. These health checks are lightweight > operations that do not suffer from contention issues on the namesystem lock > or other shared resources. Isolating the RPC handlers is sufficient to avoid > this situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9683) DiskBalancer : Add cancelPlan implementation
[ https://issues.apache.org/jira/browse/HDFS-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-9683: --- Attachment: HDFS-9683-HDFS-1312.001.patch > DiskBalancer : Add cancelPlan implementation > > > Key: HDFS-9683 > URL: https://issues.apache.org/jira/browse/HDFS-9683 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-9683-HDFS-1312.001.patch > > > Add datanode side code for Cancel Plan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9683) DiskBalancer : Add cancelPlan implementation
[ https://issues.apache.org/jira/browse/HDFS-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-9683: --- Attachment: (was: HDFS-9683-HDFS-1312.001.patch) > DiskBalancer : Add cancelPlan implementation > > > Key: HDFS-9683 > URL: https://issues.apache.org/jira/browse/HDFS-9683 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > > Add datanode side code for Cancel Plan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9711) Integrate CSRF prevention filter in WebHDFS.
[ https://issues.apache.org/jira/browse/HDFS-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-9711: Attachment: HDFS-9711.003.patch Here is patch v003. I updated this in response HADOOP-12758, where [~lmccay] added support to the filter for checking the User-Agent. With consideration of Larry's recent comments here, I have kept the configurability of the header name. > Integrate CSRF prevention filter in WebHDFS. > > > Key: HDFS-9711 > URL: https://issues.apache.org/jira/browse/HDFS-9711 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode, webhdfs >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-9711.001.patch, HDFS-9711.002.patch, > HDFS-9711.003.patch > > > HADOOP-12691 introduced a filter in Hadoop Common to help REST APIs guard > against cross-site request forgery attacks. This issue tracks integration of > that filter in WebHDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9785) Remove unused TrashPolicy#getInstance and initiate code
[
https://issues.apache.org/jira/browse/HDFS-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139667#comment-15139667
]
Zhe Zhang commented on HDFS-9785:
-
Actually {{TrashPolicy}} could be directly used by applications. Marking as
incompatible and we should consider it for 3.0.
> Remove unused TrashPolicy#getInstance and initiate code
> ---
>
> Key: HDFS-9785
> URL: https://issues.apache.org/jira/browse/HDFS-9785
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Zhe Zhang
>Priority: Minor
> Labels: incompatibleChange
>
> A follow-on from HDFS-8831: now the {{getInstance}} and {{initiate}} APIs
> with Path is not used anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9785) Remove unused TrashPolicy#getInstance and initiate code
[
https://issues.apache.org/jira/browse/HDFS-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9785:
Labels: incompatibleChange (was: )
> Remove unused TrashPolicy#getInstance and initiate code
> ---
>
> Key: HDFS-9785
> URL: https://issues.apache.org/jira/browse/HDFS-9785
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Zhe Zhang
>Priority: Minor
> Labels: incompatibleChange
>
> A follow-on from HDFS-8831: now the {{getInstance}} and {{initiate}} APIs
> with Path is not used anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9785) Remove unused TrashPolicy#getInstance and initiate code
[
https://issues.apache.org/jira/browse/HDFS-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9785:
Target Version/s: 3.0.0 (was: 2.8.0)
> Remove unused TrashPolicy#getInstance and initiate code
> ---
>
> Key: HDFS-9785
> URL: https://issues.apache.org/jira/browse/HDFS-9785
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Zhe Zhang
>Priority: Minor
> Labels: incompatibleChange
>
> A follow-on from HDFS-8831: now the {{getInstance}} and {{initiate}} APIs
> with Path is not used anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9785) Remove unused TrashPolicy#getInstance and initiate code
[
https://issues.apache.org/jira/browse/HDFS-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9785:
Description: A follow-on from HDFS-8831: now the {{getInstance}} and
{{initiate}} APIs with Path is not used anymore. (was: A follow-on from
HDFS-8831: now the {{getInstance}} API with Path is not used anymore.)
> Remove unused TrashPolicy#getInstance and initiate code
> ---
>
> Key: HDFS-9785
> URL: https://issues.apache.org/jira/browse/HDFS-9785
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Zhe Zhang
>Priority: Minor
>
> A follow-on from HDFS-8831: now the {{getInstance}} and {{initiate}} APIs
> with Path is not used anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9785) Remove unused TrashPolicy#getInstance and initiate code
[
https://issues.apache.org/jira/browse/HDFS-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9785:
Summary: Remove unused TrashPolicy#getInstance and initiate code (was:
Remove unused TrashPolicy#getInstance code)
> Remove unused TrashPolicy#getInstance and initiate code
> ---
>
> Key: HDFS-9785
> URL: https://issues.apache.org/jira/browse/HDFS-9785
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Zhe Zhang
>Priority: Minor
>
> A follow-on from HDFS-8831: now the {{getInstance}} API with Path is not used
> anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9785) Remove unused TrashPolicy#getInstance code
Zhe Zhang created HDFS-9785:
---
Summary: Remove unused TrashPolicy#getInstance code
Key: HDFS-9785
URL: https://issues.apache.org/jira/browse/HDFS-9785
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Zhe Zhang
Priority: Minor
A follow-on from HDFS-8831: now the {{getInstance}} API with Path is not used
anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9784) Example usage is not correct in Transparent Encryption document
[
https://issues.apache.org/jira/browse/HDFS-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139402#comment-15139402
]
Hudson commented on HDFS-9784:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9268 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9268/])
HDFS-9784. Example usage is not correct in Transparent Encryption (aajisaka:
rev 60d2011b7c0fe55b8bc44a141660d3e2df37a68d)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md
> Example usage is not correct in Transparent Encryption document
> ---
>
> Key: HDFS-9784
> URL: https://issues.apache.org/jira/browse/HDFS-9784
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0, 2.7.2
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Fix For: 2.8.0, 2.7.3
>
> Attachments: HADOOP-12779.1.patch
>
>
> It says
> {code}
> # As the normal user, create a new encryption key
> hadoop key create myKey
> {code}
> But, this actually fails with the below error.
> {code}
> $ hadoop key create myKey
> java.lang.IllegalArgumentException: Uppercase key names are unsupported: myKey
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:677)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:685)
> at
> org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
> at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
> {code}
> Though I'm not sure why it is so, I think the document should be fixed to use
> only lowercase in key names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9752) Permanent write failures may happen to slow writers during datanode rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-9752: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.6.5 2.7.3 2.8.0 Status: Resolved (was: Patch Available) Committed the branch-2.7 patch. Thanks [~walter.k.su] for the contribution. > Permanent write failures may happen to slow writers during datanode rolling > upgrades > > > Key: HDFS-9752 > URL: https://issues.apache.org/jira/browse/HDFS-9752 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Walter Su >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.6.5 > > Attachments: HDFS-9752-branch-2.6.03.patch, > HDFS-9752-branch-2.7.03.patch, HDFS-9752.01.patch, HDFS-9752.02.patch, > HDFS-9752.03.patch, HdfsWriter.java > > > When datanodes are being upgraded, an out-of-band ack is sent upstream and > the client does a pipeline recovery. The client may hit this multiple times > as more nodes get upgraded. This normally does not cause any issue, but if > the client is holding the stream open without writing any data during this > time, a permanent write failure can occur. > This is because there is a limit of 5 recovery trials for the same packet, > which is tracked by "last acked sequence number". Since the empty heartbeat > packets for an idle output stream does not increment the sequence number, the > write will fail after it seeing 5 pipeline breakages by datanode upgrades. > This check/limit was added to avoid spinning until running out of nodes in > the cluster due to a corruption or any other irrecoverable conditions. The > datanode upgrade-restart should be excluded from the count. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9752) Permanent write failures may happen to slow writers during datanode rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139393#comment-15139393 ] Akira AJISAKA commented on HDFS-9752: - +1 for the branch-2.7 patch. I'll commit it shortly. > Permanent write failures may happen to slow writers during datanode rolling > upgrades > > > Key: HDFS-9752 > URL: https://issues.apache.org/jira/browse/HDFS-9752 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Walter Su >Priority: Critical > Attachments: HDFS-9752-branch-2.6.03.patch, > HDFS-9752-branch-2.7.03.patch, HDFS-9752.01.patch, HDFS-9752.02.patch, > HDFS-9752.03.patch, HdfsWriter.java > > > When datanodes are being upgraded, an out-of-band ack is sent upstream and > the client does a pipeline recovery. The client may hit this multiple times > as more nodes get upgraded. This normally does not cause any issue, but if > the client is holding the stream open without writing any data during this > time, a permanent write failure can occur. > This is because there is a limit of 5 recovery trials for the same packet, > which is tracked by "last acked sequence number". Since the empty heartbeat > packets for an idle output stream does not increment the sequence number, the > write will fail after it seeing 5 pipeline breakages by datanode upgrades. > This check/limit was added to avoid spinning until running out of nodes in > the cluster due to a corruption or any other irrecoverable conditions. The > datanode upgrade-restart should be excluded from the count. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-9784) Example usage is not correct in Transparent Encryption document
[
https://issues.apache.org/jira/browse/HDFS-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139378#comment-15139378
]
Akira AJISAKA edited comment on HDFS-9784 at 2/9/16 6:43 PM:
-
Filed HADOOP-12786 for documenting "hadoop key" command usage.
was (Author: ajisakaa):
Filed HADOOP-12876 for documenting "hadoop key" command usage.
> Example usage is not correct in Transparent Encryption document
> ---
>
> Key: HDFS-9784
> URL: https://issues.apache.org/jira/browse/HDFS-9784
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0, 2.7.2
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Fix For: 2.8.0, 2.7.3
>
> Attachments: HADOOP-12779.1.patch
>
>
> It says
> {code}
> # As the normal user, create a new encryption key
> hadoop key create myKey
> {code}
> But, this actually fails with the below error.
> {code}
> $ hadoop key create myKey
> java.lang.IllegalArgumentException: Uppercase key names are unsupported: myKey
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:677)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:685)
> at
> org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
> at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
> {code}
> Though I'm not sure why it is so, I think the document should be fixed to use
> only lowercase in key names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9784) Example usage is not correct in Transparent Encryption document
[
https://issues.apache.org/jira/browse/HDFS-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139378#comment-15139378
]
Akira AJISAKA commented on HDFS-9784:
-
Filed HADOOP-12876 for documenting "hadoop key" command usage.
> Example usage is not correct in Transparent Encryption document
> ---
>
> Key: HDFS-9784
> URL: https://issues.apache.org/jira/browse/HDFS-9784
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0, 2.7.2
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Fix For: 2.8.0, 2.7.3
>
> Attachments: HADOOP-12779.1.patch
>
>
> It says
> {code}
> # As the normal user, create a new encryption key
> hadoop key create myKey
> {code}
> But, this actually fails with the below error.
> {code}
> $ hadoop key create myKey
> java.lang.IllegalArgumentException: Uppercase key names are unsupported: myKey
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:677)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:685)
> at
> org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
> at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
> {code}
> Though I'm not sure why it is so, I think the document should be fixed to use
> only lowercase in key names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9784) Example usage is not correct in Transparent Encryption document
[
https://issues.apache.org/jira/browse/HDFS-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-9784:
Assignee: Takashi Ohnishi
> Example usage is not correct in Transparent Encryption document
> ---
>
> Key: HDFS-9784
> URL: https://issues.apache.org/jira/browse/HDFS-9784
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0, 2.7.2
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Fix For: 2.8.0, 2.7.3
>
> Attachments: HADOOP-12779.1.patch
>
>
> It says
> {code}
> # As the normal user, create a new encryption key
> hadoop key create myKey
> {code}
> But, this actually fails with the below error.
> {code}
> $ hadoop key create myKey
> java.lang.IllegalArgumentException: Uppercase key names are unsupported: myKey
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:677)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:685)
> at
> org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
> at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
> {code}
> Though I'm not sure why it is so, I think the document should be fixed to use
> only lowercase in key names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9784) Example usage is not correct in Transparent Encryption document
[
https://issues.apache.org/jira/browse/HDFS-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-9784:
Resolution: Fixed
Fix Version/s: 2.7.3
2.8.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-2, branch-2.8, and branch-2.7. Thanks
[~bwtakacy] for the contribution.
> Example usage is not correct in Transparent Encryption document
> ---
>
> Key: HDFS-9784
> URL: https://issues.apache.org/jira/browse/HDFS-9784
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0, 2.7.2
>Reporter: Takashi Ohnishi
> Fix For: 2.8.0, 2.7.3
>
> Attachments: HADOOP-12779.1.patch
>
>
> It says
> {code}
> # As the normal user, create a new encryption key
> hadoop key create myKey
> {code}
> But, this actually fails with the below error.
> {code}
> $ hadoop key create myKey
> java.lang.IllegalArgumentException: Uppercase key names are unsupported: myKey
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:677)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:685)
> at
> org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
> at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
> {code}
> Though I'm not sure why it is so, I think the document should be fixed to use
> only lowercase in key names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9784) Example usage is not correct in Transparent Encryption document
[
https://issues.apache.org/jira/browse/HDFS-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-9784:
Hadoop Flags: Reviewed
Summary: Example usage is not correct in Transparent Encryption
document (was: Example usage is not correct in "Transparent Encryption in
HDFS" document.)
> Example usage is not correct in Transparent Encryption document
> ---
>
> Key: HDFS-9784
> URL: https://issues.apache.org/jira/browse/HDFS-9784
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0, 2.7.2
>Reporter: Takashi Ohnishi
> Attachments: HADOOP-12779.1.patch
>
>
> It says
> {code}
> # As the normal user, create a new encryption key
> hadoop key create myKey
> {code}
> But, this actually fails with the below error.
> {code}
> $ hadoop key create myKey
> java.lang.IllegalArgumentException: Uppercase key names are unsupported: myKey
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:677)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:685)
> at
> org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
> at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
> {code}
> Though I'm not sure why it is so, I think the document should be fixed to use
> only lowercase in key names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (HDFS-9784) Example usage is not correct in "Transparent Encryption in HDFS" document.
[
https://issues.apache.org/jira/browse/HDFS-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA moved HADOOP-12779 to HDFS-9784:
--
Affects Version/s: (was: 2.7.2)
(was: 3.0.0)
3.0.0
2.7.2
Component/s: (was: documentation)
documentation
Key: HDFS-9784 (was: HADOOP-12779)
Project: Hadoop HDFS (was: Hadoop Common)
> Example usage is not correct in "Transparent Encryption in HDFS" document.
> --
>
> Key: HDFS-9784
> URL: https://issues.apache.org/jira/browse/HDFS-9784
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.7.2, 3.0.0
>Reporter: Takashi Ohnishi
> Attachments: HADOOP-12779.1.patch
>
>
> It says
> {code}
> # As the normal user, create a new encryption key
> hadoop key create myKey
> {code}
> But, this actually fails with the below error.
> {code}
> $ hadoop key create myKey
> java.lang.IllegalArgumentException: Uppercase key names are unsupported: myKey
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:677)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:685)
> at
> org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
> at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
> {code}
> Though I'm not sure why it is so, I think the document should be fixed to use
> only lowercase in key names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8959) Provide an iterator-based API for listing all the snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-8959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139311#comment-15139311
]
Rakesh R commented on HDFS-8959:
I've rebased the patch in the latest trunk code. I'm planning to use this API
in the {{lsSnapshottableDir}} command. Reference:
[comment|https://issues.apache.org/jira/browse/HDFS-8643?focusedCommentId=14658800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14658800].
[~jingzhao], [~umamaheswararao] could you please check the patch when you get
a chance. Thanks!
> Provide an iterator-based API for listing all the snapshottable directories
> ---
>
> Key: HDFS-8959
> URL: https://issues.apache.org/jira/browse/HDFS-8959
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8959-00.patch, HDFS-8959-01.patch,
> HDFS-8959-02.patch
>
>
> Presently {{DistributedFileSystem#getSnapshottableDirListing()}} is sending
> all the {{SnapshottableDirectoryStatus[]}} array to the clients. Now the
> client should have enough space to hold it in memory. There could be chance
> that the client JVMs running out of memory because of this. Also, some time
> back there was a
> [comment|https://issues.apache.org/jira/browse/HDFS-8643?focusedCommentId=14658800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14658800]
> about RPC packet limitation and a large number of snapshot list can again
> cause issues.
> I believe iterator based {{DistributedFileSystem#listSnapshottableDirs()}}
> API would be a good addition!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9752) Permanent write failures may happen to slow writers during datanode rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139280#comment-15139280 ] Arpit Agarwal commented on HDFS-9752: - +1 for the branch-2.6 patch. I committed it to branch-2.6. Thanks for taking care of this backport [~walter.k.su]. > Permanent write failures may happen to slow writers during datanode rolling > upgrades > > > Key: HDFS-9752 > URL: https://issues.apache.org/jira/browse/HDFS-9752 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Walter Su >Priority: Critical > Attachments: HDFS-9752-branch-2.6.03.patch, > HDFS-9752-branch-2.7.03.patch, HDFS-9752.01.patch, HDFS-9752.02.patch, > HDFS-9752.03.patch, HdfsWriter.java > > > When datanodes are being upgraded, an out-of-band ack is sent upstream and > the client does a pipeline recovery. The client may hit this multiple times > as more nodes get upgraded. This normally does not cause any issue, but if > the client is holding the stream open without writing any data during this > time, a permanent write failure can occur. > This is because there is a limit of 5 recovery trials for the same packet, > which is tracked by "last acked sequence number". Since the empty heartbeat > packets for an idle output stream does not increment the sequence number, the > write will fail after it seeing 5 pipeline breakages by datanode upgrades. > This check/limit was added to avoid spinning until running out of nodes in > the cluster due to a corruption or any other irrecoverable conditions. The > datanode upgrade-restart should be excluded from the count. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8701) WebHDFS write commands fail when running multiple DataNodes on one machine
[
https://issues.apache.org/jira/browse/HDFS-8701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu resolved HDFS-8701.
---
Resolution: Cannot Reproduce
> WebHDFS write commands fail when running multiple DataNodes on one machine
> --
>
> Key: HDFS-8701
> URL: https://issues.apache.org/jira/browse/HDFS-8701
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Anthony Hsu
>
> For testing purposes, we are running multiple DataNodes per machine. {{hadoop
> fs}} commands work fine when using the {{hdfs://}} protocol, but when using
> {{webhdfs://}}, any command that writes to HDFS (e.g.: {{-put}} or
> {{-touchz}}) fails:
> {code}
> $ hadoop fs -put test.txt webhdfs://:/user/foo
> put: -dn-4
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9781) FsDatasetImpl#getBlockReports can occasionally throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-9781:
--
Description:
FsDatasetImpl#getBlockReports occasionally throws NPE. The NPE caused
TestFsDatasetImpl#testRemoveVolumeBeingWritten to time out, because the test
waits for the call to FsDatasetImpl#getBlockReports to complete without
exceptions.
Additionally, the test should be updated to identify an expected exception,
using {{GenericTestUtils.assertExceptionContains()}}
{noformat}
Exception in thread "Thread-20" java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1709)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1BlockReportThread.run(TestFsDatasetImpl.java:587)
2016-02-08 15:47:30,379 [Thread-21] WARN impl.TestFsDatasetImpl
(TestFsDatasetImpl.java:run(606)) - Exception caught. This should not affect
the test
java.io.IOException: Failed to move meta file for ReplicaBeingWritten, blk_0_0,
RBW
getNumBytes() = 0
getBytesOnDisk() = 0
getVisibleLength()= 0
getVolume() =
/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current
getBlockFile()=
/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0
bytesAcked=0
bytesOnDisk=0 from
/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta
to
/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:857)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addFinalizedBlock(BlockPoolSlice.java:295)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addFinalizedBlock(FsVolumeImpl.java:819)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeReplica(FsDatasetImpl.java:1620)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:1601)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1ResponderThread.run(TestFsDatasetImpl.java:603)
Caused by: java.io.IOException:
renameTo(src=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta,
dst=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta)
failed.
at org.apache.hadoop.io.nativeio.NativeIO.renameTo(NativeIO.java:873)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:855)
... 5 more
2016-02-08 15:47:34,381 [Thread-19] INFO impl.FsDatasetImpl
(FsVolumeList.java:waitVolumeRemoved(287)) - Volume reference is released.
2016-02-08 15:47:34,384 [Thread-19] INFO impl.TestFsDatasetImpl
(TestFsDatasetImpl.java:testRemoveVolumeBeingWritten(622)) - Volumes removed
{noformat}
was:
FsDatasetImpl#getBlockReports occasionally throws NPE. The NPE caused
TestFsDatasetImpl#testRemoveVolumeBeingWritten to fail, because the test waits
for the call to FsDatasetImpl#getBlockReports to complete without exceptions.
Additionally, the test should be updated to identify an expected exception,
using {{GenericTestUtils.assertExceptionContains()}}
{noformat}
Exception in thread "Thread-20" java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1709)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1BlockReportThread.run(TestFsDatasetImpl.java:587)
2016-02-08 15:47:30,379 [Thread-21] WARN impl.TestFsDatasetImpl
(TestFsDatasetImpl.java:run(606)) - Exception caught. This should not affect
the test
java.io.IOException: Failed to move meta file for ReplicaBeingWritten, blk_0_0,
RBW
getNumBytes() = 0
getBytesOnDisk() = 0
getVisibleLength()= 0
getVolume() =
/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current
getBlockFile()=
/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0
bytesAcked=0
bytesOnDisk=0 from
/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta
to
/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta
at
org.apache.hadoop
[jira] [Commented] (HDFS-9781) FsDatasetImpl#getBlockReports can occasionally throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139140#comment-15139140
]
Wei-Chiu Chuang commented on HDFS-9781:
---
This is related to HDFS-9701, which created
{TestFsDatasetImpl#testRemoveVolumeBeingWritten}.
The NPE is caused by accessing a volume after it is removed in the test. So the
test case should be improved, and FsDatasetImpl should also be improved to
handle a potential NPE.
> FsDatasetImpl#getBlockReports can occasionally throw NullPointerException
> -
>
> Key: HDFS-9781
> URL: https://issues.apache.org/jira/browse/HDFS-9781
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>
> FsDatasetImpl#getBlockReports occasionally throws NPE. The NPE caused
> TestFsDatasetImpl#testRemoveVolumeBeingWritten to fail, because the test
> waits for the call to FsDatasetImpl#getBlockReports to complete without
> exceptions.
> Additionally, the test should be updated to identify an expected exception,
> using {{GenericTestUtils.assertExceptionContains()}}
> {noformat}
> Exception in thread "Thread-20" java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1709)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1BlockReportThread.run(TestFsDatasetImpl.java:587)
> 2016-02-08 15:47:30,379 [Thread-21] WARN impl.TestFsDatasetImpl
> (TestFsDatasetImpl.java:run(606)) - Exception caught. This should not affect
> the test
> java.io.IOException: Failed to move meta file for ReplicaBeingWritten,
> blk_0_0, RBW
> getNumBytes() = 0
> getBytesOnDisk() = 0
> getVisibleLength()= 0
> getVolume() =
> /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current
> getBlockFile()=
> /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0
> bytesAcked=0
> bytesOnDisk=0 from
> /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta
> to
> /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:857)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addFinalizedBlock(BlockPoolSlice.java:295)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addFinalizedBlock(FsVolumeImpl.java:819)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeReplica(FsDatasetImpl.java:1620)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:1601)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1ResponderThread.run(TestFsDatasetImpl.java:603)
> Caused by: java.io.IOException:
> renameTo(src=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta,
>
> dst=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta)
> failed.
> at org.apache.hadoop.io.nativeio.NativeIO.renameTo(NativeIO.java:873)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:855)
> ... 5 more
> 2016-02-08 15:47:34,381 [Thread-19] INFO impl.FsDatasetImpl
> (FsVolumeList.java:waitVolumeRemoved(287)) - Volume reference is released.
> 2016-02-08 15:47:34,384 [Thread-19] INFO impl.TestFsDatasetImpl
> (TestFsDatasetImpl.java:testRemoveVolumeBeingWritten(622)) - Volumes removed
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9783) DataNode deadlocks after running hdfs mover
[ https://issues.apache.org/jira/browse/HDFS-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B resolved HDFS-9783. - Resolution: Duplicate Resolving as duplicate. Feel free to re-open if fix in HDFS-9661 doesnt work for you. > DataNode deadlocks after running hdfs mover > --- > > Key: HDFS-9783 > URL: https://issues.apache.org/jira/browse/HDFS-9783 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: David Watzke > Attachments: datanode.stack.txt > > > We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of > ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran > the hdfs mover to enforce the storage policy. > After a few minutes, the mover hangs because one or more datanodes hang as > well. Please check out the deadlock revealed by jstack. Also, here's what > appeared in DN log: > 2016-02-09 15:58:15,676 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Not able to copy block 1157230686 to /5.45.60.142:40144 because threads quota > is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9783) DataNode deadlocks after running hdfs mover
[
https://issues.apache.org/jira/browse/HDFS-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139135#comment-15139135
]
Vinayakumar B commented on HDFS-9783:
-
This issue has been already fixed in HDFS-9661 which is available in branch-2.7
( 2.7.3)
And the stacktrace shown is not part of hadoop-2.6 code.
{{FsDataSetImpl#moveBlockAcrossStorage(..)}} not present in 2.6 code.
So IMO this would be a duplicate.
> DataNode deadlocks after running hdfs mover
> ---
>
> Key: HDFS-9783
> URL: https://issues.apache.org/jira/browse/HDFS-9783
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.6.0
>Reporter: David Watzke
> Attachments: datanode.stack.txt
>
>
> We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of
> ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran
> the hdfs mover to enforce the storage policy.
> After a few minutes, the mover hangs because one or more datanodes hang as
> well. Please check out the deadlock revealed by jstack. Also, here's what
> appeared in DN log:
> 2016-02-09 15:58:15,676 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Not able to copy block 1157230686 to /5.45.60.142:40144 because threads quota
> is exceeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[ https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139120#comment-15139120 ] Daniel Templeton commented on HDFS-9637: Test failures are unrelated. > Add test for HADOOP-12702 and HADOOP-12759 > -- > > Key: HDFS-9637 > URL: https://issues.apache.org/jira/browse/HDFS-9637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9637.001.patch, HDFS-9637.002.patch, > HDFS-9637.003.patch, HDFS-9637.004.patch > > > Per discussion on the dev list, the tests for the new FileSystemSink class > should be added to the HDFS project to avoid creating a dependency for the > common project on the HDFS project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9783) DataNode deadlocks after running hdfs mover
[ https://issues.apache.org/jira/browse/HDFS-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Watzke updated HDFS-9783: --- Description: We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran the hdfs mover to enforce the storage policy. After a few minutes, the mover hangs because one or more datanodes hang as well. Please check out the deadlock revealed by jstack. Also, here's what appeared in DN log: 2016-02-09 15:58:15,676 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Not able to copy block 1157230686 to /5.45.60.142:40144 because threads quota is exceeded. was: We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran the hdfs mover to enforce the storage policy. After a few minutes, the mover hangs because one or more datanodes hang as well. Please check out the deadlock revealed by jstack . > DataNode deadlocks after running hdfs mover > --- > > Key: HDFS-9783 > URL: https://issues.apache.org/jira/browse/HDFS-9783 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: David Watzke > Attachments: datanode.stack.txt > > > We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of > ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran > the hdfs mover to enforce the storage policy. > After a few minutes, the mover hangs because one or more datanodes hang as > well. Please check out the deadlock revealed by jstack. Also, here's what > appeared in DN log: > 2016-02-09 15:58:15,676 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Not able to copy block 1157230686 to /5.45.60.142:40144 because threads quota > is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9783) DataNode deadlocks after running hdfs mover
[ https://issues.apache.org/jira/browse/HDFS-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Watzke updated HDFS-9783: --- Description: We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran the hdfs mover to enforce the storage policy. After a few minutes, the mover hangs because one or more datanodes hang as well. Please check out the deadlock revealed by jstack . was: We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran the hdfs mover to enforce the storage policy. After a few minutes, the mover hangs and a few datanodes hang as well. > DataNode deadlocks after running hdfs mover > --- > > Key: HDFS-9783 > URL: https://issues.apache.org/jira/browse/HDFS-9783 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: David Watzke > Attachments: datanode.stack.txt > > > We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of > ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran > the hdfs mover to enforce the storage policy. > After a few minutes, the mover hangs because one or more datanodes hang as > well. Please check out the deadlock revealed by jstack . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9783) DataNode deadlocks after running hdfs mover
[ https://issues.apache.org/jira/browse/HDFS-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Watzke updated HDFS-9783: --- Attachment: datanode.stack.txt deadlock revealed by jstack > DataNode deadlocks after running hdfs mover > --- > > Key: HDFS-9783 > URL: https://issues.apache.org/jira/browse/HDFS-9783 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: David Watzke > Attachments: datanode.stack.txt > > > We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of > ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran > the hdfs mover to enforce the storage policy. > After a few minutes, the mover hangs and a few datanodes hang as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9783) DataNode deadlocks after running hdfs mover
David Watzke created HDFS-9783: -- Summary: DataNode deadlocks after running hdfs mover Key: HDFS-9783 URL: https://issues.apache.org/jira/browse/HDFS-9783 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.6.0 Reporter: David Watzke We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran the hdfs mover to enforce the storage policy. After a few minutes, the mover hangs and a few datanodes hang as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9395) HDFS operations vary widely in which failures they put in the audit log and which they leave out
[
https://issues.apache.org/jira/browse/HDFS-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139027#comment-15139027
]
Kuhu Shukla commented on HDFS-9395:
---
Thank you Vinayakumar. While making this change I also noticed the
{{checkAccess}} method which logs only when there is an ACE and not for
successful attempts. My concern is from an older JIRA, HDFS-6570.
{quote}
getAclStatus is probably the simplest method to look at for an example that
does everything. Do you think we need to write to the audit log for this
method? I'm thinking that we shouldn't, because the purpose of this method is
to query whether or not the user has access.
{quote}
[~cnauroth], Request for some comments on the methods that dont log successful
attempts like checkAccess and getAclStatus. Thanks a lot!
> HDFS operations vary widely in which failures they put in the audit log and
> which they leave out
>
>
> Key: HDFS-9395
> URL: https://issues.apache.org/jira/browse/HDFS-9395
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kuhu Shukla
> Attachments: HDFS-9395.001.patch, HDFS-9395.002.patch,
> HDFS-9395.003.patch, HDFS-9395.004.patch, HDFS-9395.005.patch
>
>
> So, the big question here is what should go in the audit log? All failures,
> or just "permission denied" failures? Or, to put it a different way, if
> someone attempts to do something and it fails because a file doesn't exist,
> is that worth an audit log entry?
> We are currently inconsistent on this point. For example, concat,
> getContentSummary, addCacheDirective, and setErasureEncodingPolicy create an
> audit log entry for all failures, but setOwner, delete, and setAclEntries
> attempt to only create an entry for AccessControlException-based failures.
> There are a few operations, like allowSnapshot, disallowSnapshot, and
> startRollingUpgrade that never create audit log failure entries at all. They
> simply log nothing for any failure, and log success for a successful
> operation.
> So to summarize, different HDFS operations currently fall into 3 categories:
> 1. audit-log all failures
> 2. audit-log only AccessControlException failures
> 3. never audit-log failures
> Which category is right? And how can we fix the inconsistency
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[
https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138924#comment-15138924
]
Hadoop QA commented on HDFS-9637:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 52s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 53s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 199m 3s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.fs.TestHdfsNativeCodeLoader |
| | hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| JDK v1.7.0_95 Failed junit tests | hadoop.fs.TestHdfsNativeCodeLoader |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
\\
\\
|| Subsystem || Report/Notes ||
| Do
[jira] [Commented] (HDFS-8959) Provide an iterator-based API for listing all the snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-8959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138869#comment-15138869
]
Hadoop QA commented on HDFS-8959:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 8s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-hdfs-project: patch generated 1 new + 797
unchanged - 0 fixed = 798 total (was 797) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 5s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 34s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 11s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 18s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:gr
[jira] [Commented] (HDFS-9579) Provide bytes-read-by-network-distance metrics at FileSystem.Statistics level
[
https://issues.apache.org/jira/browse/HDFS-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138779#comment-15138779
]
Hadoop QA commented on HDFS-9579:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 0s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 11s
{color} | {color:red} root: patch generated 32 new + 402 unchanged - 0 fixed =
434 total (was 402) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 47s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 53s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 27s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 14s
{color} | {color:red} hadoo
[jira] [Updated] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[ https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-9637: --- Attachment: HDFS-9637.004.patch Reposting patch 4 as it wasn't the current code. > Add test for HADOOP-12702 and HADOOP-12759 > -- > > Key: HDFS-9637 > URL: https://issues.apache.org/jira/browse/HDFS-9637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9637.001.patch, HDFS-9637.002.patch, > HDFS-9637.003.patch, HDFS-9637.004.patch > > > Per discussion on the dev list, the tests for the new FileSystemSink class > should be added to the HDFS project to avoid creating a dependency for the > common project on the HDFS project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[ https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-9637: --- Attachment: (was: HDFS-9637.004.patch) > Add test for HADOOP-12702 and HADOOP-12759 > -- > > Key: HDFS-9637 > URL: https://issues.apache.org/jira/browse/HDFS-9637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9637.001.patch, HDFS-9637.002.patch, > HDFS-9637.003.patch > > > Per discussion on the dev list, the tests for the new FileSystemSink class > should be added to the HDFS project to avoid creating a dependency for the > common project on the HDFS project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9780) RollingFileSystemSink doesn't work on secure clusters
[ https://issues.apache.org/jira/browse/HDFS-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138646#comment-15138646 ] Daniel Templeton commented on HDFS-9780: HDFS-9782 has been created to make the interval configurable. > RollingFileSystemSink doesn't work on secure clusters > - > > Key: HDFS-9780 > URL: https://issues.apache.org/jira/browse/HDFS-9780 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: HADOOP-12775.001.patch, HADOOP-12775.002.patch, > HADOOP-12775.003.patch, HDFS-9780.004.patch > > > If HDFS has kerberos enabled, the sink cannot write its logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9782) RollingFileSystemSink should have configurable roll interval
[ https://issues.apache.org/jira/browse/HDFS-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-9782: --- Attachment: HDFS-9782.001.patch Here's an initial sketch of the plan. > RollingFileSystemSink should have configurable roll interval > > > Key: HDFS-9782 > URL: https://issues.apache.org/jira/browse/HDFS-9782 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9782.001.patch > > > Right now it defaults to rolling at the top of every hour. Instead that > interval should be configurable. The interval should also allow for some > play so that all hosts don't try to flush their files simultaneously. > I'm filing this in HDFS because I suspect it will involve touching the HDFS > tests. If it turns out not to, I'll move it into common instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8959) Provide an iterator-based API for listing all the snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-8959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8959:
---
Attachment: HDFS-8959-02.patch
> Provide an iterator-based API for listing all the snapshottable directories
> ---
>
> Key: HDFS-8959
> URL: https://issues.apache.org/jira/browse/HDFS-8959
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8959-00.patch, HDFS-8959-01.patch,
> HDFS-8959-02.patch
>
>
> Presently {{DistributedFileSystem#getSnapshottableDirListing()}} is sending
> all the {{SnapshottableDirectoryStatus[]}} array to the clients. Now the
> client should have enough space to hold it in memory. There could be chance
> that the client JVMs running out of memory because of this. Also, some time
> back there was a
> [comment|https://issues.apache.org/jira/browse/HDFS-8643?focusedCommentId=14658800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14658800]
> about RPC packet limitation and a large number of snapshot list can again
> cause issues.
> I believe iterator based {{DistributedFileSystem#listSnapshottableDirs()}}
> API would be a good addition!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9637) Add test for HADOOP-12702 and HADOOP-12759
[
https://issues.apache.org/jira/browse/HDFS-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138523#comment-15138523
]
Hadoop QA commented on HDFS-9637:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 44s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 11s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 26s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs |
| | hadoop.fs.TestHdfsNativeCodeLoader |
| JDK v1.7.0_95 Failed junit tests |
hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs |
| | hadoop.fs.TestHdfsNativeCodeLoader |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786998/HDFS-9637.004.patch |
| JIRA Issue | HDFS-9637 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyl
