[jira] [Commented] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335471#comment-16335471 ] hu xiaodong commented on HDFS-13046: I submit a sample patch. welcome to discuss. thank you. > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong updated HDFS-13046: --- Attachment: HDFS-13046-sample.patch > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335460#comment-16335460
]
genericqa commented on HDFS-13025:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} root in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} HDFS-10285 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
54s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 19 new + 0 unchanged - 0 fixed = 19 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
9s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907235/HDFS-13025-HDFS-10285-3.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 63512b31ec36 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / 0c41c08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22768/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22768/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/22768/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
[jira] [Created] (HDFS-13049) RBF: Inconsistent Router OPTS config in branch-2 and branch-3
Wei Yan created HDFS-13049: -- Summary: RBF: Inconsistent Router OPTS config in branch-2 and branch-3 Key: HDFS-13049 URL: https://issues.apache.org/jira/browse/HDFS-13049 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan For router's OPS config: * In trunk/branch-3, the cmd looks for (command)_(subcommad)_OPTS --> HDFS_DFSROUTER_OPTS / HADOOP_DFSROUTER_OPTS. * In branch-2, the cmd looks for HADOOP_ROUTER_OPTS. Also, it would be better to add commented corresponding config in hadoop-env.sh, for better user visibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335438#comment-16335438
]
genericqa commented on HDFS-13025:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
11s{color} | {color:red} root in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} HDFS-10285 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
54s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 19 new + 0 unchanged - 0 fixed = 19 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
9s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907235/HDFS-13025-HDFS-10285-3.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0c2ed25d3aaa 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / 0c41c08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22767/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22767/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/22767/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
[jira] [Updated] (HDFS-13048) LowRedundancyReplicatedBlocks metric can be negative
[
https://issues.apache.org/jira/browse/HDFS-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-13048:
-
Attachment: HDFS-13048-sample.patch
> LowRedundancyReplicatedBlocks metric can be negative
>
>
> Key: HDFS-13048
> URL: https://issues.apache.org/jira/browse/HDFS-13048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: metrics
>Affects Versions: 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13048-sample.patch
>
>
> I'm seeing {{LowRedundancyReplicatedBlocks}} become negative. This should be
> 0 or positive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13048) LowRedundancyReplicatedBlocks metric can be negative
[
https://issues.apache.org/jira/browse/HDFS-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335432#comment-16335432
]
Akira Ajisaka commented on HDFS-13048:
--
Attached a sample patch. I'll add test cases and fix some exiting regression
tests.
> LowRedundancyReplicatedBlocks metric can be negative
>
>
> Key: HDFS-13048
> URL: https://issues.apache.org/jira/browse/HDFS-13048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: metrics
>Affects Versions: 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13048-sample.patch
>
>
> I'm seeing {{LowRedundancyReplicatedBlocks}} become negative. This should be
> 0 or positive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12986) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-0f7169d-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HDFS-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-12986: --- Description: This jira will update ozone to latest snapshot release-0.1.1-alpha-0f7169d-SNAPSHOT (was: This jira will update ozone to latest snapshot release-0.1.1-alpha-00f80b4-SNAPSHOT) > Ozone: Update ozone to latest ratis snapshot build > (0.1.1-alpha-0f7169d-SNAPSHOT) > - > > Key: HDFS-12986 > URL: https://issues.apache.org/jira/browse/HDFS-12986 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-12986-HDFS-7240.001.patch, > HDFS-12986-HDFS-7240.002.patch, HDFS-12986-HDFS-7240.003.patch, > HDFS-12986-HDFS-7240.004.patch, HDFS-12986-HDFS-7240.005.patch > > > This jira will update ozone to latest snapshot > release-0.1.1-alpha-0f7169d-SNAPSHOT -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12986) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-0f7169d-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HDFS-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-12986: --- Attachment: HDFS-12986-HDFS-7240.005.patch > Ozone: Update ozone to latest ratis snapshot build > (0.1.1-alpha-0f7169d-SNAPSHOT) > - > > Key: HDFS-12986 > URL: https://issues.apache.org/jira/browse/HDFS-12986 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-12986-HDFS-7240.001.patch, > HDFS-12986-HDFS-7240.002.patch, HDFS-12986-HDFS-7240.003.patch, > HDFS-12986-HDFS-7240.004.patch, HDFS-12986-HDFS-7240.005.patch > > > This jira will update ozone to latest snapshot > release-0.1.1-alpha-0f7169d-SNAPSHOT -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13048) LowRedundancyReplicatedBlocks metric can be negative
[
https://issues.apache.org/jira/browse/HDFS-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335421#comment-16335421
]
Akira Ajisaka commented on HDFS-13048:
--
LowRedundancyReplicatedBlocks is now calculated by (lowRedundancyBlocks -
MissingReplicatedBlocks). Therefore if there are some MissingReplicatedBlocks,
the value can be negative.
> LowRedundancyReplicatedBlocks metric can be negative
>
>
> Key: HDFS-13048
> URL: https://issues.apache.org/jira/browse/HDFS-13048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: metrics
>Affects Versions: 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>
> I'm seeing {{LowRedundancyReplicatedBlocks}} become negative. This should be
> 0 or positive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13047) Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read timeout
[
https://issues.apache.org/jira/browse/HDFS-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13047:
-
Attachment: HDFS-13047-HDFS-7240.001.patch
> Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read
> timeout
> -
>
> Key: HDFS-13047
> URL: https://issues.apache.org/jira/browse/HDFS-13047
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-13047-HDFS-7240.001.patch
>
>
> The tests are failing because of the following error.
> {code}
> org.apache.hadoop.ozone.web.client.OzoneRestClientException: Failed to
> putKey: keyName=01e8b923-5876-4d5e-8adc-4214caf33f64,
> file=/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/2/TestKeys/f8f75b8d-f15b-482f-afa0-babe1d6c4bf6
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:253)
> at
> org.apache.hadoop.ozone.web.client.TestKeys$PutHelper.putKey(TestKeys.java:218)
> at
> org.apache.hadoop.ozone.web.client.TestKeys$PutHelper.access$100(TestKeys.java:168)
> at
> org.apache.hadoop.ozone.web.client.TestKeys.runTestPutAndGetKeyWithDnRestart(TestKeys.java:297)
> at
> org.apache.hadoop.ozone.web.client.TestKeys.testPutAndGetKeyWithDnRestart(TestKeys.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
> at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
> at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.executePutKey(OzoneBucket.java:276)
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:250)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Updated] (HDFS-13047) Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read timeout
[
https://issues.apache.org/jira/browse/HDFS-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13047:
-
Status: Patch Available (was: Open)
> Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read
> timeout
> -
>
> Key: HDFS-13047
> URL: https://issues.apache.org/jira/browse/HDFS-13047
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-13047-HDFS-7240.001.patch
>
>
> The tests are failing because of the following error.
> {code}
> org.apache.hadoop.ozone.web.client.OzoneRestClientException: Failed to
> putKey: keyName=01e8b923-5876-4d5e-8adc-4214caf33f64,
> file=/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/2/TestKeys/f8f75b8d-f15b-482f-afa0-babe1d6c4bf6
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:253)
> at
> org.apache.hadoop.ozone.web.client.TestKeys$PutHelper.putKey(TestKeys.java:218)
> at
> org.apache.hadoop.ozone.web.client.TestKeys$PutHelper.access$100(TestKeys.java:168)
> at
> org.apache.hadoop.ozone.web.client.TestKeys.runTestPutAndGetKeyWithDnRestart(TestKeys.java:297)
> at
> org.apache.hadoop.ozone.web.client.TestKeys.testPutAndGetKeyWithDnRestart(TestKeys.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
> at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
> at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.executePutKey(OzoneBucket.java:276)
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:250)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13048) LowRedundancyReplicatedBlocks metric can be negative
Akira Ajisaka created HDFS-13048:
Summary: LowRedundancyReplicatedBlocks metric can be negative
Key: HDFS-13048
URL: https://issues.apache.org/jira/browse/HDFS-13048
Project: Hadoop HDFS
Issue Type: Bug
Components: metrics
Affects Versions: 3.0.0
Reporter: Akira Ajisaka
Assignee: Akira Ajisaka
I'm seeing {{LowRedundancyReplicatedBlocks}} become negative. This should be 0
or positive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12838) Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation requests
[
https://issues.apache.org/jira/browse/HDFS-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335411#comment-16335411
]
genericqa commented on HDFS-12838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 1
unchanged - 1 fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}137m 56s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}214m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-12838 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907225/HDFS-12838-HDFS-7240.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs
[jira] [Commented] (HDFS-12990) Change default NameNode RPC port back to 8020
[
https://issues.apache.org/jira/browse/HDFS-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335408#comment-16335408
]
Xiao Chen commented on HDFS-12990:
--
Thanks all for the comments. I believe the [discussion
thread|https://lists.apache.org/thread.html/00d12c451e2870b35408b1c738467f82710ccdbfaded7ddfcc62d6f7@%3Ccommon-dev.hadoop.apache.org%3E]
on common-dev is getting more attention, commenting here to keep things at 1
place.
While Sanjay's suggestion definitely helps making this change (and potentially
later ones before 'stable') comply with compatibility guidelines, I'm more
inclined to agree with [~chris.douglas] that for this one specifically, it's
fixing a 3.0.0 compat bug that will help almost everyone; and in general, this
kind of Jira would best be discussed case-by-base than making a rule, because
every of them differs by nature and hard to generalize.
That said, I think the 'stable' versioning proposal may help future issues
potentially, but perhaps we can discuss that concurrently on another mailing
list thread.
{quote}If the downstream projects have hard coded 8020, is it the case that
they will not work if user has set the port to a different number in the conf?
{quote}
True, but that will not work even simply on 2.x without upgrade. The specific
cases that motivates this appears to be thoroughly discussed on the mailing
list, let's please continue the discussion there if you're okay.
> Change default NameNode RPC port back to 8020
> -
>
> Key: HDFS-12990
> URL: https://issues.apache.org/jira/browse/HDFS-12990
> Project: Hadoop HDFS
> Issue Type: Task
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-12990.01.patch
>
>
> In HDFS-9427 (HDFS should not default to ephemeral ports), we changed all
> default ports to ephemeral ports, which is very appreciated by admin. As part
> of that change, we also modified the NN RPC port from the famous 8020 to
> 9820, to be closer to other ports changed there.
> With more integration going on, it appears that all the other ephemeral port
> changes are fine, but the NN RPC port change is painful for downstream on
> migrating to Hadoop 3. Some examples include:
> # Hive table locations pointing to hdfs://nn:port/dir
> # Downstream minicluster unit tests that assumed 8020
> # Oozie workflows / downstream scripts that used 8020
> This isn't a problem for HA URLs, since that does not include the port
> number. But considering the downstream impact, instead of requiring all of
> them change their stuff, it would be a way better experience to leave the NN
> port unchanged. This will benefit Hadoop 3 adoption and ease unnecessary
> upgrade burdens.
> It is of course incompatible, but giving 3.0.0 is just out, IMO it worths to
> switch the port back.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13024:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I have committed this to the feature branch.
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer and writeChunk
> -
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch, HDFS-13024-HDFS-7240.003.patch,
> HDFS-13024-HDFS-7240.004.patch, HDFS-13024-HDFS-7240.005.patch,
> HDFS-13024-HDFS-7240.006.patch, HDFS-13024-HDFS-7240.007.patch,
> HDFS-13024-HDFS-7240.008.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13024:
-
Summary: Ozone: ContainerStateMachine should synchronize operations between
createContainer and writeChunk (was: Ozone: ContainerStateMachine should
synchronize operations between createContainer op and writeChunk)
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer and writeChunk
> -
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch, HDFS-13024-HDFS-7240.003.patch,
> HDFS-13024-HDFS-7240.004.patch, HDFS-13024-HDFS-7240.005.patch,
> HDFS-13024-HDFS-7240.006.patch, HDFS-13024-HDFS-7240.007.patch,
> HDFS-13024-HDFS-7240.008.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335398#comment-16335398
]
Mukul Kumar Singh commented on HDFS-13024:
--
Thanks for the review [~szetszwo] and [~ljain]. I will commit this shortly.
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch, HDFS-13024-HDFS-7240.003.patch,
> HDFS-13024-HDFS-7240.004.patch, HDFS-13024-HDFS-7240.005.patch,
> HDFS-13024-HDFS-7240.006.patch, HDFS-13024-HDFS-7240.007.patch,
> HDFS-13024-HDFS-7240.008.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13047) Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read timeout
[
https://issues.apache.org/jira/browse/HDFS-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13047:
-
Description:
The tests are failing because of the following error.
{code}
org.apache.hadoop.ozone.web.client.OzoneRestClientException: Failed to putKey:
keyName=01e8b923-5876-4d5e-8adc-4214caf33f64,
file=/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/2/TestKeys/f8f75b8d-f15b-482f-afa0-babe1d6c4bf6
at
org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:253)
at
org.apache.hadoop.ozone.web.client.TestKeys$PutHelper.putKey(TestKeys.java:218)
at
org.apache.hadoop.ozone.web.client.TestKeys$PutHelper.access$100(TestKeys.java:168)
at
org.apache.hadoop.ozone.web.client.TestKeys.runTestPutAndGetKeyWithDnRestart(TestKeys.java:297)
at
org.apache.hadoop.ozone.web.client.TestKeys.testPutAndGetKeyWithDnRestart(TestKeys.java:287)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.hadoop.ozone.web.client.OzoneBucket.executePutKey(OzoneBucket.java:276)
at
org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:250)
... 13 more
{code}
> Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read
> timeout
> -
>
> Key: HDFS-13047
> URL: https://issues.apache.org/jira/browse/HDFS-13047
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
>
> The tests are failing because of the following error.
> {code}
> org.apache.hadoop.ozone.web.client.OzoneRestClientException: Failed to
> putKey: keyName=01e8b923-5876-4d5e-8adc-4214caf33f64,
> file=/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/2/TestKeys/f8f75b8d-f15b-482f-afa0-babe1d6c4bf6
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:253)
> at
>
[jira] [Updated] (HDFS-13047) Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read timeout
[
https://issues.apache.org/jira/browse/HDFS-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13047:
-
Priority: Minor (was: Major)
> Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read
> timeout
> -
>
> Key: HDFS-13047
> URL: https://issues.apache.org/jira/browse/HDFS-13047
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Priority: Minor
> Fix For: HDFS-7240
>
>
> The tests are failing because of the following error.
> {code}
> org.apache.hadoop.ozone.web.client.OzoneRestClientException: Failed to
> putKey: keyName=01e8b923-5876-4d5e-8adc-4214caf33f64,
> file=/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/2/TestKeys/f8f75b8d-f15b-482f-afa0-babe1d6c4bf6
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:253)
> at
> org.apache.hadoop.ozone.web.client.TestKeys$PutHelper.putKey(TestKeys.java:218)
> at
> org.apache.hadoop.ozone.web.client.TestKeys$PutHelper.access$100(TestKeys.java:168)
> at
> org.apache.hadoop.ozone.web.client.TestKeys.runTestPutAndGetKeyWithDnRestart(TestKeys.java:297)
> at
> org.apache.hadoop.ozone.web.client.TestKeys.testPutAndGetKeyWithDnRestart(TestKeys.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
> at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
> at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.executePutKey(OzoneBucket.java:276)
> at
> org.apache.hadoop.ozone.web.client.OzoneBucket.putKey(OzoneBucket.java:250)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[
https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335397#comment-16335397
]
genericqa commented on HDFS-12462:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12462 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907232/HDFS-12462.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d06a6ff1 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6347b22 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22761/testReport/ |
| Max. process+thread count | 1348 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22761/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure coding policy extra options should
[jira] [Created] (HDFS-13047) Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read timeout
Mukul Kumar Singh created HDFS-13047: Summary: Ozone: TestKeys, TestKeysRatis and TestOzoneShell are failing because of read timeout Key: HDFS-13047 URL: https://issues.apache.org/jira/browse/HDFS-13047 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Mukul Kumar Singh Fix For: HDFS-7240 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13046) consider load of datanodes when read blocks of file
hu xiaodong created HDFS-13046: -- Summary: consider load of datanodes when read blocks of file Key: HDFS-13046 URL: https://issues.apache.org/jira/browse/HDFS-13046 Project: Hadoop HDFS Issue Type: New Feature Reporter: hu xiaodong Assignee: hu xiaodong When sorting block locations, we just consider the distance of datanodes. can we consider the load of datanodes? We can add a configuration such as 'dfs.namenode.reading.considerLoad', if set to true, then sort the blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335380#comment-16335380
]
Uma Maheswara Rao G edited comment on HDFS-13033 at 1/23/18 4:51 AM:
-
#
{quote}This class handles the external SPS block movements. This will assign
blockmovement tasks to target datanode descriptors.{quote}Please correct this
comments.
# moveBlock logic seems to be duplicated. Can we refactor to keep common?
# please cleanup unused imports
# BlocksMovementsStatusHandler: class docs seems representing internal spa
# you may want to update @InterfaceAudience level for new classes created
# BlockMovementAttemptFinished: please add javadocs for public methods
# probably we should add todo about need to maintain scheduled counts per
datanode.
was (Author: umamaheswararao):
# . \{code}
* This class handles the external SPS block movements. This will assign block
* movement tasks to target datanode descriptors.
*/
{code}
Please correct this comments.
# moveBlock logic seems to be duplicated. Can we refactor to keep common?
# please cleanup unused imports
# BlocksMovementsStatusHandler: class docs seems representing internal spa
# you may want to update interfaceAudience level for new classes created
# BlockMovementAttemptFinished: please add javadocs for public methods
# probably we should add todo about need to maintain scheduled counts per
datanode.
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335380#comment-16335380
]
Uma Maheswara Rao G commented on HDFS-13033:
# . \{code}
* This class handles the external SPS block movements. This will assign block
* movement tasks to target datanode descriptors.
*/
{code}
Please correct this comments.
# moveBlock logic seems to be duplicated. Can we refactor to keep common?
# please cleanup unused imports
# BlocksMovementsStatusHandler: class docs seems representing internal spa
# you may want to update interfaceAudience level for new classes created
# BlockMovementAttemptFinished: please add javadocs for public methods
# probably we should add todo about need to maintain scheduled counts per
datanode.
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335370#comment-16335370
]
genericqa commented on HDFS-13025:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} root in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} HDFS-10285 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
17s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 19 new + 0 unchanged - 0 fixed = 19 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
9s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 6m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907235/HDFS-13025-HDFS-10285-3.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 907b09b3c560 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / 0c41c08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22763/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22763/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/22763/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
[jira] [Commented] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[ https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335367#comment-16335367 ] Uma Maheswara Rao G commented on HDFS-13025: Minor changes to address check style warnings, all of them from test code. > [SPS]: Implement a mechanism to scan the files for external SPS > --- > > Key: HDFS-13025 > URL: https://issues.apache.org/jira/browse/HDFS-13025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-13025-HDFS-10285-0.patch, > HDFS-13025-HDFS-10285-1.patch, HDFS-13025-HDFS-10285-2.patch, > HDFS-13025-HDFS-10285-3.patch > > > HDFS-12911 modularization is introducing FileIDCollector interface for > scanning the files. That will help us to plugin different ways of scanning > mechanisms if needed. > For Internal SPS, we have INode based scanning. For external SPS, we should > go via client API scanning as we can not access NN internal structures. > This is the task to implement the scanning plugin for external SPS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[ https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-13025: --- Attachment: HDFS-13025-HDFS-10285-3.patch > [SPS]: Implement a mechanism to scan the files for external SPS > --- > > Key: HDFS-13025 > URL: https://issues.apache.org/jira/browse/HDFS-13025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-13025-HDFS-10285-0.patch, > HDFS-13025-HDFS-10285-1.patch, HDFS-13025-HDFS-10285-2.patch, > HDFS-13025-HDFS-10285-3.patch > > > HDFS-12911 modularization is introducing FileIDCollector interface for > scanning the files. That will help us to plugin different ways of scanning > mechanisms if needed. > For Internal SPS, we have INode based scanning. For external SPS, we should > go via client API scanning as we can not access NN internal structures. > This is the task to implement the scanning plugin for external SPS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[
https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335364#comment-16335364
]
genericqa commented on HDFS-12772:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 57s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 19 new + 375 unchanged
- 19 fixed = 394 total (was 394) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestParallelUnixDomainRead |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
| | hadoop.hdfs.TestBlockStoragePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12772 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907217/HDFS-12772.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 359cba0d3cab 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6347b22 |
| maven |
[jira] [Updated] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[ https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-13025: --- Attachment: (was: HDFS-13025-HDFS-10285-3.patch) > [SPS]: Implement a mechanism to scan the files for external SPS > --- > > Key: HDFS-13025 > URL: https://issues.apache.org/jira/browse/HDFS-13025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-13025-HDFS-10285-0.patch, > HDFS-13025-HDFS-10285-1.patch, HDFS-13025-HDFS-10285-2.patch > > > HDFS-12911 modularization is introducing FileIDCollector interface for > scanning the files. That will help us to plugin different ways of scanning > mechanisms if needed. > For Internal SPS, we have INode based scanning. For external SPS, we should > go via client API scanning as we can not access NN internal structures. > This is the task to implement the scanning plugin for external SPS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[ https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-13025: --- Attachment: HDFS-13025-HDFS-10285-3.patch > [SPS]: Implement a mechanism to scan the files for external SPS > --- > > Key: HDFS-13025 > URL: https://issues.apache.org/jira/browse/HDFS-13025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-13025-HDFS-10285-0.patch, > HDFS-13025-HDFS-10285-1.patch, HDFS-13025-HDFS-10285-2.patch, > HDFS-13025-HDFS-10285-3.patch > > > HDFS-12911 modularization is introducing FileIDCollector interface for > scanning the files. That will help us to plugin different ways of scanning > mechanisms if needed. > For Internal SPS, we have INode based scanning. For external SPS, we should > go via client API scanning as we can not access NN internal structures. > This is the task to implement the scanning plugin for external SPS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[
https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Istvan Fajth updated HDFS-13040:
Attachment: TestDFSInotifyEventInputStreamKerberized.java
> Kerberized inotify client fails despite kinit properly
> --
>
> Key: HDFS-13040
> URL: https://issues.apache.org/jira/browse/HDFS-13040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
> Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-13040.001.patch,
> TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java
>
>
> This issue is similar to HDFS-10799.
> HDFS-10799 turned out to be a client side issue where client is responsible
> for renewing kerberos ticket actively.
> However we found in a slightly setup even if client has valid Kerberos
> credentials, inotify still fails.
> Suppose client uses principal h...@example.com,
> namenode 1 uses server principal hdfs/nn1.example@example.com
> namenode 2 uses server principal hdfs/nn2.example@example.com
> *After Namenodes starts for longer than kerberos ticket lifetime*, the client
> fails with the following error:
> {noformat}
> 18/01/19 11:23:02 WARN security.UserGroupInformation:
> PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS)
> cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We
> encountered an error reading
> https://nn2.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3,
>
> https://nn1.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3.
> During automatic edit log failover, we noticed that all of the remaining
> edit log streams are shorter than the current one! The best remaining edit
> log ends at transaction 8683, but we thought we could read up to transaction
> 8684. If you continue, metadata will be lost forever!
> at
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
> {noformat}
> Typically if NameNode has an expired Kerberos ticket, the error handling for
> the typical edit log tailing would let NameNode to relogin with its own
> Kerberos principal. However, when inotify uses the same code path to retrieve
> edits, since the current user is the inotify client's principal, unless
> client uses the same principal as the NameNode, NameNode can't do it on
> behalf of the client.
> Therefore, a more appropriate approach is to use proxy user so that NameNode
> can retrieving edits on behalf of the client.
> I will attach a patch to fix it. This patch has been verified to work for a
> CDH5.10.2 cluster, however it seems impossible to craft a unit test for this
> fix because the way Hadoop UGI handles Kerberos credentials (I can't have a
> single process that logins as two Kerberos principals simultaneously and let
> them establish connection)
> A possible workaround is for the inotify client to use the active NameNode's
> server principal. However, that's not going to work when there's a namenode
> failover, because then the client's principal will not be consistent with the
> active NN's one, and then fails to authenticate.
> Credit: this
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[
https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335359#comment-16335359
]
Istvan Fajth commented on HDFS-13040:
-
Hi [~jojochuang],
let me add as well a quick comment on the patch, it seems that you have added
the proxyUser usage to the getCurrentEditLogTxid() method.
In the DFSInotifyEventInputStream there are two RPC calls, one is to the
getCurrentEditLogTxid(), and an other to getEditsFromTxid(), I believe the
problem can arise with any one of those, however the current call sequence is
implying that if the getCurrentEditLogTxid succeeds, then getEditsFromTxid can
succeed unless the ticket experies exactly between the two calls.
[~daryn], sorry for the long comment, but let me add what I know, and think
important at this point, I am happy to share more if there are any questions.
I have tried to set up a test as well with MiniKDC, MiniJournalCluster, and
MiniDFSCluster, I had struggles when I wanted to login with a new keytab, and
call doAs with the new UGI, as the UGI contained as well traces of the
principal that was used by the cluster.
What I see:
When I do a UserGroupInformation.reset() after cluster startup, this code in
the JUnit test:
{code:java}
UserGroupInformation ugi =
UserGroupInformation.loginUserFromKeytabAndReturnUGI("hdfs",
generalHDFSKeytabFile.getAbsolutePath());
LOG.info("global: "+UserGroupInformation.getCurrentUser() + ", login as:"
+ UserGroupInformation.getLoginUser());
LOG.info("ugi: "+ugi.getUserName() + " authenticated via " +
ugi.getAuthenticationMethod());
ugi.doAs(new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
LOG.info("Current user in doAs:
"+UserGroupInformation.getCurrentUser()+" loginuser
is:"+UserGroupInformation.getLoginUser());
fs.mkdirs(new Path("/user/test"));
fs.listFiles(new Path("/"), true);
LOG.info("ListFiles success");
return null;
}
});
{code}
produces the following log:
{code:java}
018-01-23 04:08:45,689 INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:waitActive(2346)) - Cluster is active
2018-01-23 04:08:45,715 INFO hdfs.TestDFSInotifyEventInputStream
(TestDFSInotifyEventInputStream.java:testWithKerberizedCluster(534)) - global:
pifta (auth:KERBEROS), login as:pifta (auth:KERBEROS)
2018-01-23 04:08:45,715 INFO hdfs.TestDFSInotifyEventInputStream
(TestDFSInotifyEventInputStream.java:testWithKerberizedCluster(535)) - ugi:
h...@example.com authenticated via KERBEROS
2018-01-23 04:08:45,715 INFO hdfs.TestDFSInotifyEventInputStream
(TestDFSInotifyEventInputStream.java:run(539)) - Current user in doAs:
h...@example.com (auth:KERBEROS) loginuser is:pifta (auth:KERBEROS)
2018-01-23 04:08:45,722 WARN security.UserGroupInformation
(UserGroupInformation.java:doAs(1923)) - PriviledgedActionException as:pifta
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]
2018-01-23 04:08:45,723 WARN ipc.Client (Client.java:run(713)) - Exception
encountered while connecting to the server : javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
2018-01-23 04:08:45,723 WARN security.UserGroupInformation
(UserGroupInformation.java:doAs(1923)) - PriviledgedActionException as:pifta
(auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
2018-01-23 04:08:45,728 WARN security.UserGroupInformation
(UserGroupInformation.java:doAs(1923)) - PriviledgedActionException
as:h...@example.com (auth:KERBEROS) cause:java.io.IOException: Failed on local
exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]; Host Details : local host is:
"host.local/192.168.0.122"; destination host is: "localhost":58506;
2018-01-23 04:08:45,728 INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:shutdown(1744)) - Shutting down the Mini HDFS Cluster
{code}
So this time I do have a loginUser called pifta that does not have any
authentication or tgt it is the system user I use.
When I don't reset the ugi, the same code provides the following log:
{code:java}
2018-01-23 04:17:49,936 INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:waitActive(2346)) - Cluster is active
2018-01-23 04:17:49,953 INFO hdfs.TestDFSInotifyEventInputStream
(TestDFSInotifyEventInputStream.java:testWithKerberizedCluster(534)) - global:
hdfs/localh...@example.com (auth:KERBEROS), login as:hdfs/localh...@example.com
(a
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Status: Patch Available (was: Open) > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch, HDFS-12462.002.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Attachment: HDFS-12462.002.patch > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch, HDFS-12462.002.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Status: Open (was: Patch Available) > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335343#comment-16335343
]
genericqa commented on HDFS-13033:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
48s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
0s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-10285 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 36 new + 177 unchanged - 0 fixed = 213 total (was 177) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
15s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 1
unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | new
org.apache.hadoop.hdfs.server.sps.ExternalSPSBlockMoveTaskHandler(Configuration,
NameNodeConnector, SPSService) invokes org.apache.hadoop.util.Daemon.start()
At ExternalSPSBlockMoveTaskHandler.java:org.apache.hadoop.util.Daemon.start()
At ExternalSPSBlockMoveTaskHandler.java:[line 116] |
| Failed junit tests | hadoop.hdfs.TestDatanodeDeath |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907128/HDFS-13033-HDFS-10285-01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335341#comment-16335341
]
genericqa commented on HDFS-13033:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} root in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} HDFS-10285 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 1m
14s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 36 new + 177 unchanged - 0 fixed = 213 total (was 177) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
10s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 5m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907128/HDFS-13033-HDFS-10285-01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux cd25ab8f783d 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / 0c41c08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22760/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22760/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
[jira] [Commented] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[
https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335337#comment-16335337
]
genericqa commented on HDFS-12462:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 6 unchanged - 0 fixed = 8 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 53s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12462 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907219/HDFS-12462.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a80e31acba40 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6347b22 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22758/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22758/testReport/ |
| Max. process+thread count | 1447 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console
[jira] [Commented] (HDFS-13023) Journal Sync does not work on a secure cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335330#comment-16335330
]
genericqa commented on HDFS-13023:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 27s{color} | {color:orange} root: The patch generated 1 new + 107 unchanged
- 3 fixed = 108 total (was 110) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
32s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}211m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestErasureCodingMultipleRacks |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| |
[jira] [Updated] (HDFS-12838) Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation requests
[
https://issues.apache.org/jira/browse/HDFS-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12838:
-
Attachment: HDFS-12838-HDFS-7240.006.patch
> Ozone: Optimize number of allocated block rpc by aggregating multiple block
> allocation requests
> ---
>
> Key: HDFS-12838
> URL: https://issues.apache.org/jira/browse/HDFS-12838
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-12838-HDFS-7240.001.patch,
> HDFS-12838-HDFS-7240.002.patch, HDFS-12838-HDFS-7240.003.patch,
> HDFS-12838-HDFS-7240.004.patch, HDFS-12838-HDFS-7240.005.patch,
> HDFS-12838-HDFS-7240.006.patch
>
>
> Currently KeySpaceManager allocates multiple blocks by sending multiple block
> allocation requests over the RPC. This can be optimized to aggregate multiple
> block allocation request over one rpc.
> {code}
> while (requestedSize > 0) {
> long allocateSize = Math.min(scmBlockSize, requestedSize);
> AllocatedBlock allocatedBlock =
> scmBlockClient.allocateBlock(allocateSize, type, factor);
> KsmKeyLocationInfo subKeyInfo = new KsmKeyLocationInfo.Builder()
> .setContainerName(allocatedBlock.getPipeline().getContainerName())
> .setBlockID(allocatedBlock.getKey())
> .setShouldCreateContainer(allocatedBlock.getCreateContainer())
> .setIndex(idx++)
> .setLength(allocateSize)
> .setOffset(0)
> .build();
> locations.add(subKeyInfo);
> requestedSize -= allocateSize;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335282#comment-16335282
]
genericqa commented on HDFS-13025:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
53s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
10s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-10285 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}183m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.TestDisableConnCache |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907193/HDFS-13025-HDFS-10285-2.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9a3574426b8b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git
[jira] [Commented] (HDFS-11225) NameNode crashed because deleteSnapshot held FSNamesystem lock too long
[
https://issues.apache.org/jira/browse/HDFS-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335277#comment-16335277
]
Manoj Govindassamy commented on HDFS-11225:
---
[~shashikant], please feel free to own this bug and followup with your
proposal. I am on to other things and not able to spend time on this. My
apologies.
> NameNode crashed because deleteSnapshot held FSNamesystem lock too long
> ---
>
> Key: HDFS-11225
> URL: https://issues.apache.org/jira/browse/HDFS-11225
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
> Environment: CDH5.8.2, HA
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Critical
> Labels: high-availability
> Attachments: Snaphot_Deletion_Design_Proposal.pdf
>
>
> The deleteSnapshot operation is synchronous. In certain situations this
> operation may hold FSNamesystem lock for too long, bringing almost every
> NameNode operation to a halt.
> We have observed one incidence where it took so long that ZKFC believes the
> NameNode is down. All other IPC threads were waiting to acquire FSNamesystem
> lock. This specific deleteSnapshot took ~70 seconds. ZKFC has connection
> timeout of 45 seconds by default, and if all IPC threads wait for
> FSNamesystem lock and can't accept new incoming connection, ZKFC times out,
> advances epoch and NameNode will therefore lose its active NN role and then
> fail.
> Relevant log:
> {noformat}
> Thread 154 (IPC Server handler 86 on 8020):
> State: RUNNABLE
> Blocked count: 2753455
> Waited count: 89201773
> Stack:
>
> org.apache.hadoop.hdfs.server.namenode.INode$BlocksMapUpdateInfo.addDeleteBlock(INode.java:879)
>
> org.apache.hadoop.hdfs.server.namenode.INodeFile.destroyAndCollectBlocks(INodeFile.java:508)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.destroyAndCollectBlocks(INodeDirectory.java:763)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.destroyAndCollectBlocks(INodeDirectory.java:763)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.destroyAndCollectBlocks(INodeDirectory.java:763)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.destroyAndCollectBlocks(INodeDirectory.java:763)
>
> org.apache.hadoop.hdfs.server.namenode.INodeReference.destroyAndCollectBlocks(INodeReference.java:339)
>
> org.apache.hadoop.hdfs.server.namenode.INodeReference$WithName.destroyAndCollectBlocks(INodeReference.java:606)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature$ChildrenDiff.destroyDeletedList(DirectoryWithSnapshotFeature.java:119)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature$ChildrenDiff.access$400(DirectoryWithSnapshotFeature.java:61)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature$DirectoryDiff.destroyDiffAndCollectBlocks(DirectoryWithSnapshotFeature.java:319)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature$DirectoryDiff.destroyDiffAndCollectBlocks(DirectoryWithSnapshotFeature.java:167)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.AbstractINodeDiffList.deleteSnapshotDiff(AbstractINodeDiffList.java:83)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature.cleanDirectory(DirectoryWithSnapshotFeature.java:745)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtree(INodeDirectory.java:776)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtreeRecursively(INodeDirectory.java:747)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature.cleanDirectory(DirectoryWithSnapshotFeature.java:747)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtree(INodeDirectory.java:776)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtreeRecursively(INodeDirectory.java:747)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtree(INodeDirectory.java:789)
> {noformat}
> After the ZKFC determined NameNode was down and advanced epoch, the NN
> finished deleting snapshot, and sent the edit to journal nodes, but it was
> rejected because epoch was updated. See the following stacktrace:
> {noformat}
> 10.0.16.21:8485: IPC's epoch 17 is less than the last promised epoch 18
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:429)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:457)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:352)
> at
>
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Attachment: HDFS-12462.001.patch > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Status: Patch Available (was: Open) > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12462.001.patch > > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Status: Open (was: Patch Available) > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12462) Erasure coding policy extra options should be sorted by key value
[ https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-12462: -- Attachment: (was: HDFS-12462.001.patch) > Erasure coding policy extra options should be sorted by key value > - > > Key: HDFS-12462 > URL: https://issues.apache.org/jira/browse/HDFS-12462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: liaoyuxiangqin >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > > To make sure the serialized fsimage and editlog binary equal, Erasure coding > policy extra options should be sorted by key value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12772) RBF: Federation Router State State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12772: --- Attachment: HDFS-12772.004.patch > RBF: Federation Router State State Store internal API > - > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch, HDFS-12772.003.patch, HDFS-12772.004.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13023) Journal Sync does not work on a secure cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335221#comment-16335221
]
Hanisha Koneru commented on HDFS-13023:
---
Committed to {{trunk}}. Thanks [~bharatviswa] for working on this.
> Journal Sync does not work on a secure cluster
> --
>
> Key: HDFS-13023
> URL: https://issues.apache.org/jira/browse/HDFS-13023
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13023.00.patch, HDFS-13023.01.patch,
> HDFS-13023.02.patch, HDFS-13023.03.patch
>
>
> Fails with the following exception.
> {code}
> 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal
> /0.0.0.0:8485 with xxx, journal id: mycluster
> 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with
> Journal at xxx/xxx:8485
> com.google.protobuf.ServiceException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/x...@example.com
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/xxx
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13023) Journal Sync does not work on a secure cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335220#comment-16335220
]
Hudson commented on HDFS-13023:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13537 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13537/])
HDFS-13023. Journal Sync does not work on a secure cluster. Contributed
(hanishakoneru: rev 6347b2253d1b912d1d8d89a4b3d470c596f2c596)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HDFSPolicyProvider.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/InterQJournalProtocolServerSideTranslatorPB.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/InterQJournalProtocolTranslatorPB.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/InterQJournalProtocolPB.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeSyncer.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/proto/InterQJournalProtocol.proto
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocol/InterQJournalProtocol.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
* (edit) hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeRpcServer.java
> Journal Sync does not work on a secure cluster
> --
>
> Key: HDFS-13023
> URL: https://issues.apache.org/jira/browse/HDFS-13023
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13023.00.patch, HDFS-13023.01.patch,
> HDFS-13023.02.patch, HDFS-13023.03.patch
>
>
> Fails with the following exception.
> {code}
> 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal
> /0.0.0.0:8485 with xxx, journal id: mycluster
> 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with
> Journal at xxx/xxx:8485
> com.google.protobuf.ServiceException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/x...@example.com
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/xxx
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[
https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335212#comment-16335212
]
genericqa commented on HDFS-12772:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}166m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12772 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907188/HDFS-12772.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 7920938c40bb 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 97fe3cc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22753/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-13009) Creation of Encryption zone should succeed even if directory is not empty.
[ https://issues.apache.org/jira/browse/HDFS-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335195#comment-16335195 ] Xiao Chen commented on HDFS-13009: -- Thanks for the discussions guys. This may come as a side-product of what Andrew asked: it would be good to have a tool to get more details of an EZ (especially when it's not finalized). We have getFileEncryptionInfo API to check per-file, but IMO helpful to know about the status based on entire EZ, and hopefully which files are still unencrypted so people know what's left to 'finalize' the EZ. Previously the assumption is if you have an EZ, everything is certainly encrypted, so motivation for the tool was lesser. :) > Creation of Encryption zone should succeed even if directory is not empty. > -- > > Key: HDFS-13009 > URL: https://issues.apache.org/jira/browse/HDFS-13009 > Project: Hadoop HDFS > Issue Type: Improvement > Components: encryption >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Currently we have a restriction that creation of encryption zone can be done > only on an empty directory. > This jira is to remove that restriction. > Motivation: > New customers who wants to start using Encryption zone can make an existing > directory encrypted. > They will be able to read the old data as it is and will be decrypting the > newly written data. > Internally we have many customers asking for this feature. > Currently they have to ask for more space quota, encrypt the old data. > This will make their life much more easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13023) Journal Sync does not work on a secure cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335186#comment-16335186
]
Hanisha Koneru commented on HDFS-13023:
---
Failed test cases pass locally for me too.
> Journal Sync does not work on a secure cluster
> --
>
> Key: HDFS-13023
> URL: https://issues.apache.org/jira/browse/HDFS-13023
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13023.00.patch, HDFS-13023.01.patch,
> HDFS-13023.02.patch, HDFS-13023.03.patch
>
>
> Fails with the following exception.
> {code}
> 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal
> /0.0.0.0:8485 with xxx, journal id: mycluster
> 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with
> Journal at xxx/xxx:8485
> com.google.protobuf.ServiceException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/x...@example.com
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/xxx
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13023) Journal Sync does not work on a secure cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335182#comment-16335182
]
Bharat Viswanadham commented on HDFS-13023:
---
{quote}1. Regarding checkstyle issue.
{quote}
versionID is used like below to get the version no in RPC.java to get
versionField.
Field versionField = protocol.getField("versionID");
versionField.getLong(protocol);
So, i think we can ignore this checkstyle issue.
{quote}2. Regarding testing
{quote}
The code has been tested on a kerberos cluster, after the fix, JournalNodeSync
is working fine.
Will create a follow up Jira for testing JournalNode test with kerberos.
3. Ran failed test cases locally, they are passing.
> Journal Sync does not work on a secure cluster
> --
>
> Key: HDFS-13023
> URL: https://issues.apache.org/jira/browse/HDFS-13023
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13023.00.patch, HDFS-13023.01.patch,
> HDFS-13023.02.patch, HDFS-13023.03.patch
>
>
> Fails with the following exception.
> {code}
> 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal
> /0.0.0.0:8485 with xxx, journal id: mycluster
> 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with
> Journal at xxx/xxx:8485
> com.google.protobuf.ServiceException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/x...@example.com
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/xxx
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13023) Journal Sync does not work on a secure cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335172#comment-16335172
]
genericqa commented on HDFS-13023:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 17m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 22s{color} | {color:orange} root: The patch generated 1 new + 107 unchanged
- 3 fixed = 108 total (was 110) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
34s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
4s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}124m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}253m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Commented] (HDFS-13035) Owner should be allowed to set xattr if not already set.
[ https://issues.apache.org/jira/browse/HDFS-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335164#comment-16335164 ] Daryn Sharp commented on HDFS-13035: [~andrew.wang], need advice. The core requirement is we need to seamlessly support raw encrypted data transfer by arbitrary users. (Webhdfs support is just bonus points) Distcp already copies via reserved raw paths and then transfers over xattrs including the EZ/FE xattrs. This only works for super users but could be extended to support non-super users. On another Jira we both shared concerns about allowing a user to alter the FE info xattr but that's essentially required. What about relaxing the restrictions for setting the FE info to: a) file must be in an EZ; b) file must not have an existing FE info. > Owner should be allowed to set xattr if not already set. > > > Key: HDFS-13035 > URL: https://issues.apache.org/jira/browse/HDFS-13035 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs, namenode >Reporter: Rushabh S Shah >Priority: Major > > Motivation: This is needed to support encryption zones on WebhdfsFileSystem. > For writing into EZ directory, webhdfs client will encrypt data on client > side and will always write into /.reserved/raw/ directory so that datanode > will not encrypt since its writing to /.reserved/raw directory. > But then we have to somehow set crypto related x-attrs on the file. > Currently only super user is allowed to set x-attrs. > So I am proposing that the file owner(and superuser) should be allowed to set > crypto related x-attrs if they are already not set. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13009) Creation of Encryption zone should succeed even if directory is not empty.
[ https://issues.apache.org/jira/browse/HDFS-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335150#comment-16335150 ] Andrew Wang commented on HDFS-13009: Thanks for the reply Daryn. If this option is only intended for encryption of new data, then this makes sense. The current limitation is only a policy thing, and I think it'd be fine to have a flag that makes EZs behave like storage policies or EC policies. There are potential touchpoints with the rename restrictions and reencrypt EDEK functionality, so let's make sure to cover those in the test suite. I think it'd also be useful to have a way of checking if all the data in an EZ is encrypted and "finalizing" it. This makes it easy for users to understand when 100% of the data is encrypted, which I assume is the end goal even with the retention policy. For example, a flag on the EZ xattr while it's in mixed mode, and after iterating it, removing the flag to indicate it's fully encrypted. > Creation of Encryption zone should succeed even if directory is not empty. > -- > > Key: HDFS-13009 > URL: https://issues.apache.org/jira/browse/HDFS-13009 > Project: Hadoop HDFS > Issue Type: Improvement > Components: encryption >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Currently we have a restriction that creation of encryption zone can be done > only on an empty directory. > This jira is to remove that restriction. > Motivation: > New customers who wants to start using Encryption zone can make an existing > directory encrypted. > They will be able to read the old data as it is and will be decrypting the > newly written data. > Internally we have many customers asking for this feature. > Currently they have to ask for more space quota, encrypt the old data. > This will make their life much more easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13009) Creation of Encryption zone should succeed even if directory is not empty.
[ https://issues.apache.org/jira/browse/HDFS-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335103#comment-16335103 ] Daryn Sharp commented on HDFS-13009: (Rushabh is out on a long vacation) [~andrew.wang], agree, the proposal isn't to change the default semantics. We need to simplify the migration of customers with existing datasets. Those with a hard retention limit, esp. short ones like 30 days, only care about encrypting new data because unencrypted data will be expunged as retention expires. It does not address the pain of converting old data, which is exactly why we are investigating this feature. Is the current limitation only based on policy? We can't find a technical reason why an EZ applied to a pre-existing directory won't allow a "hybrid" directory of old data is unencrypted, new data is encrypted. > Creation of Encryption zone should succeed even if directory is not empty. > -- > > Key: HDFS-13009 > URL: https://issues.apache.org/jira/browse/HDFS-13009 > Project: Hadoop HDFS > Issue Type: Improvement > Components: encryption >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Currently we have a restriction that creation of encryption zone can be done > only on an empty directory. > This jira is to remove that restriction. > Motivation: > New customers who wants to start using Encryption zone can make an existing > directory encrypted. > They will be able to read the old data as it is and will be decrypting the > newly written data. > Internally we have many customers asking for this feature. > Currently they have to ask for more space quota, encrypt the old data. > This will make their life much more easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[ https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335087#comment-16335087 ] Uma Maheswara Rao G commented on HDFS-13025: [~rakeshr] Thanks for the helpful reviews. I have accepted the comments and addressed all of them. Thanks. > [SPS]: Implement a mechanism to scan the files for external SPS > --- > > Key: HDFS-13025 > URL: https://issues.apache.org/jira/browse/HDFS-13025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-13025-HDFS-10285-0.patch, > HDFS-13025-HDFS-10285-1.patch, HDFS-13025-HDFS-10285-2.patch > > > HDFS-12911 modularization is introducing FileIDCollector interface for > scanning the files. That will help us to plugin different ways of scanning > mechanisms if needed. > For Internal SPS, we have INode based scanning. For external SPS, we should > go via client API scanning as we can not access NN internal structures. > This is the task to implement the scanning plugin for external SPS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13025) [SPS]: Implement a mechanism to scan the files for external SPS
[ https://issues.apache.org/jira/browse/HDFS-13025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-13025: --- Attachment: HDFS-13025-HDFS-10285-2.patch > [SPS]: Implement a mechanism to scan the files for external SPS > --- > > Key: HDFS-13025 > URL: https://issues.apache.org/jira/browse/HDFS-13025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-13025-HDFS-10285-0.patch, > HDFS-13025-HDFS-10285-1.patch, HDFS-13025-HDFS-10285-2.patch > > > HDFS-12911 modularization is introducing FileIDCollector interface for > scanning the files. That will help us to plugin different ways of scanning > mechanisms if needed. > For Internal SPS, we have INode based scanning. For external SPS, we should > go via client API scanning as we can not access NN internal structures. > This is the task to implement the scanning plugin for external SPS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13045) RBF: Improve error message returned from subcsluter
Wei Yan created HDFS-13045: -- Summary: RBF: Improve error message returned from subcsluter Key: HDFS-13045 URL: https://issues.apache.org/jira/browse/HDFS-13045 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Currently, Router directly returns exception response from subcluster to client, which may not have the correct error message, especially when the error message containing a path. One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If user1 does a chown operation on "/a/b", and he doesn't have corresponding privilege, currently the error msg looks like "Permission denied. user=user1 is not the owner of inode=/c/d", which may confuse user. Would be better to reverse the path back to original mount path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12838) Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation requests
[
https://issues.apache.org/jira/browse/HDFS-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335005#comment-16335005
]
genericqa commented on HDFS-12838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
1 unchanged - 1 fixed = 2 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
49s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}139m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}212m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.ozone.container.replication.TestContainerReplicationManager |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-12838 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907162/HDFS-12838-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
[jira] [Commented] (HDFS-13023) Journal Sync does not work on a secure cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334989#comment-16334989
]
genericqa commented on HDFS-13023:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
53s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 16m
32s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 14m 26s{color} |
{color:red} root generated 5 new + 2 unchanged - 0 fixed = 7 total (was 2)
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 26s{color}
| {color:red} root generated 183 new + 1058 unchanged - 0 fixed = 1241 total
(was 1058) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 54s{color} | {color:orange} root: The patch generated 2 new + 107 unchanged
- 3 fixed = 109 total (was 110) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 3s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
14s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
28s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}237m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| |
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[
https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334979#comment-16334979
]
Íñigo Goiri commented on HDFS-12772:
[~hanishakoneru], [~linyiqun], at the end I went ahead and split this in 4:
* HDFS-12772 for storing the Router state in the State Store
* HDFS-13042 for the Router to heartbeat its state to the State Store
* HDFS-13043 to expose this in the federation UI
* HDFS-13044 to add a new safe mode
[~linyiqun] thanks for the review, I fixed most in [^HDFS-12772.003.patch],
leaving a couple open points:
* {{getRouterRegistration()}} is to get the registration for a particular
Router while {{getRouterRegistrations()}} gets all of them. As we use the
Request/Reponse interface this is not as clear but I think it should be clear
enough with: {{RouterState getRouterRegistration(String routerId)}} and
{{List getRouterRegistrations()}}
* For the timestamp, the single one doesn't have the timestamp of the query as
it's a particular entity that we query. I could remove the timestamp from the
{{getRouterRegistrations()}}.
> RBF: Federation Router State State Store internal API
> -
>
> Key: HDFS-12772
> URL: https://issues.apache.org/jira/browse/HDFS-12772
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch,
> HDFS-12772.002.patch, HDFS-12772.003.patch
>
>
> To monitor the state of the cluster, we should track the state of the
> routers. This should be exposed in the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13009) Creation of Encryption zone should succeed even if directory is not empty.
[ https://issues.apache.org/jira/browse/HDFS-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334973#comment-16334973 ] Andrew Wang commented on HDFS-13009: Hi Rushabh, The original design intent of the zone was to make the security properties easier to reason about, since the entire directory is encrypted, and with the same encryption key. Many of our security-conscious users want everything in HDFS encrypted, and the presence of any unencrypted data would be a compliance issue. So, I don't think we can change the default semantics of the zone, though possibly we could add a flag or new concept to support the usecase you describe. IIUC, the motivation is to make the initial encryption process easier, with the goal of encrypting everything within the directory? In any case, the encryption of existing data still happens via copies which might blow quotas. I think this change helps with encrypting the newly written data, but not that much with the quota problem when converting existing data. > Creation of Encryption zone should succeed even if directory is not empty. > -- > > Key: HDFS-13009 > URL: https://issues.apache.org/jira/browse/HDFS-13009 > Project: Hadoop HDFS > Issue Type: Improvement > Components: encryption >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Currently we have a restriction that creation of encryption zone can be done > only on an empty directory. > This jira is to remove that restriction. > Motivation: > New customers who wants to start using Encryption zone can make an existing > directory encrypted. > They will be able to read the old data as it is and will be decrypting the > newly written data. > Internally we have many customers asking for this feature. > Currently they have to ask for more space quota, encrypt the old data. > This will make their life much more easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12772) RBF: Federation Router State State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12772: --- Attachment: HDFS-12772.003.patch > RBF: Federation Router State State Store internal API > - > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch, HDFS-12772.003.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[
https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334903#comment-16334903
]
genericqa commented on HDFS-12772:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 16 new + 0 unchanged - 0 fixed = 16 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}152m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}199m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocksWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits |
| | hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| |
[jira] [Comment Edited] (HDFS-7240) Object store in HDFS
[
https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334748#comment-16334748
]
Anu Engineer edited comment on HDFS-7240 at 1/22/18 7:03 PM:
-
{quote}I think we should at least have a design doc for security before merging
in order to avoid API changes.
{quote}
[~shv] I have attached a security design document,
[^HadoopStorageLayerSecurity.pdf] for the block layer and how it can be used
with HDFS and Ozone. Please take a look when you get a chance. Thanks.
was (Author: anu):
{quote}I think we should at least have a design doc for security before merging
in order to avoid API changes.
{quote}
[~shv] I have attached a [^HadoopStorageLayerSecurity.pdf] for the block layer
and how it can be used with HDFS and Ozone. Please take a look when you get a
chance. Thanks.
> Object store in HDFS
>
>
> Key: HDFS-7240
> URL: https://issues.apache.org/jira/browse/HDFS-7240
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
>Priority: Major
> Attachments: HDFS Scalability and Ozone.pdf, HDFS-7240.001.patch,
> HDFS-7240.002.patch, HDFS-7240.003.patch, HDFS-7240.003.patch,
> HDFS-7240.004.patch, HDFS-7240.005.patch, HDFS-7240.006.patch,
> HadoopStorageLayerSecurity.pdf, MeetingMinutes.pdf,
> Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, ozone_user_v0.pdf
>
>
> This jira proposes to add object store capabilities into HDFS.
> As part of the federation work (HDFS-1052) we separated block storage as a
> generic storage layer. Using the Block Pool abstraction, new kinds of
> namespaces can be built on top of the storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode
> storage, but independent of namespace metadata.
> I will soon update with a detailed design document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7240) Object store in HDFS
[
https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334748#comment-16334748
]
Anu Engineer commented on HDFS-7240:
{quote}I think we should at least have a design doc for security before merging
in order to avoid API changes.
{quote}
[~shv] I have attached a [^HadoopStorageLayerSecurity.pdf] for the block layer
and how it can be used with HDFS and Ozone. Please take a look when you get a
chance. Thanks.
> Object store in HDFS
>
>
> Key: HDFS-7240
> URL: https://issues.apache.org/jira/browse/HDFS-7240
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
>Priority: Major
> Attachments: HDFS Scalability and Ozone.pdf, HDFS-7240.001.patch,
> HDFS-7240.002.patch, HDFS-7240.003.patch, HDFS-7240.003.patch,
> HDFS-7240.004.patch, HDFS-7240.005.patch, HDFS-7240.006.patch,
> HadoopStorageLayerSecurity.pdf, MeetingMinutes.pdf,
> Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, ozone_user_v0.pdf
>
>
> This jira proposes to add object store capabilities into HDFS.
> As part of the federation work (HDFS-1052) we separated block storage as a
> generic storage layer. Using the Block Pool abstraction, new kinds of
> namespaces can be built on top of the storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode
> storage, but independent of namespace metadata.
> I will soon update with a detailed design document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-7240: --- Attachment: HadoopStorageLayerSecurity.pdf > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey >Priority: Major > Attachments: HDFS Scalability and Ozone.pdf, HDFS-7240.001.patch, > HDFS-7240.002.patch, HDFS-7240.003.patch, HDFS-7240.003.patch, > HDFS-7240.004.patch, HDFS-7240.005.patch, HDFS-7240.006.patch, > HadoopStorageLayerSecurity.pdf, MeetingMinutes.pdf, > Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12990) Change default NameNode RPC port back to 8020
[ https://issues.apache.org/jira/browse/HDFS-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334743#comment-16334743 ] Sanjay Radia commented on HDFS-12990: - For this Jira, I can go either way. In this specific case there is a very simple work around for the 2.x customer if he reads the release notes and adds a config to use his old port number. Anyone going from 2.x to 3.0 should read the release. However I can see that other harder cases may come up over the next couple of months where there isn’t such a convenient workaround. For example we are likely to come up with new issues when we do upgrade testing and there will be similar debates including arguments on the importance of supporting rolling upgrade from 2x to 3.0. To avoid such debates I suggest we amend our guideline to something of the following lines: On each new major release, till a tag such as “stable” is assigned by the PMC, we allow changes that make the software *compatible with the previous major release, even if that change breaks compatibility within the major release.* These changes may be for where we accidentally break compatibility or even when we concisely made a change without fully understanding or appreciating the impact of the incompatible change. The PMC will determine the testing criteria for assigning the stable tag. The above guideline will reduce such debates during a GA of a Major release. Further, users will realize that they need to wait for the stable tag before going production. One might argue that a GA release without a “stable” tag is merely a glorified beta or a GA-candidate. Yes i agree. Perhaps in this case we may have rushed from beta to GA a little too early. I think the above forces the PMC to seriously determine GA testing criteria and validate whether or they have been met. > Change default NameNode RPC port back to 8020 > - > > Key: HDFS-12990 > URL: https://issues.apache.org/jira/browse/HDFS-12990 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Critical > Attachments: HDFS-12990.01.patch > > > In HDFS-9427 (HDFS should not default to ephemeral ports), we changed all > default ports to ephemeral ports, which is very appreciated by admin. As part > of that change, we also modified the NN RPC port from the famous 8020 to > 9820, to be closer to other ports changed there. > With more integration going on, it appears that all the other ephemeral port > changes are fine, but the NN RPC port change is painful for downstream on > migrating to Hadoop 3. Some examples include: > # Hive table locations pointing to hdfs://nn:port/dir > # Downstream minicluster unit tests that assumed 8020 > # Oozie workflows / downstream scripts that used 8020 > This isn't a problem for HA URLs, since that does not include the port > number. But considering the downstream impact, instead of requiring all of > them change their stuff, it would be a way better experience to leave the NN > port unchanged. This will benefit Hadoop 3 adoption and ease unnecessary > upgrade burdens. > It is of course incompatible, but giving 3.0.0 is just out, IMO it worths to > switch the port back. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334721#comment-16334721
]
genericqa commented on HDFS-13024:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
25s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 0
unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
8s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}144m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}218m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| |
[jira] [Updated] (HDFS-12986) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-0f7169d-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HDFS-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12986: - Summary: Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-0f7169d-SNAPSHOT) (was: Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-00f80b4-SNAPSHOT)) > Ozone: Update ozone to latest ratis snapshot build > (0.1.1-alpha-0f7169d-SNAPSHOT) > - > > Key: HDFS-12986 > URL: https://issues.apache.org/jira/browse/HDFS-12986 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-12986-HDFS-7240.001.patch, > HDFS-12986-HDFS-7240.002.patch, HDFS-12986-HDFS-7240.003.patch, > HDFS-12986-HDFS-7240.004.patch > > > This jira will update ozone to latest snapshot > release-0.1.1-alpha-00f80b4-SNAPSHOT -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12838) Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation requests
[
https://issues.apache.org/jira/browse/HDFS-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12838:
-
Attachment: HDFS-12838-HDFS-7240.005.patch
> Ozone: Optimize number of allocated block rpc by aggregating multiple block
> allocation requests
> ---
>
> Key: HDFS-12838
> URL: https://issues.apache.org/jira/browse/HDFS-12838
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-12838-HDFS-7240.001.patch,
> HDFS-12838-HDFS-7240.002.patch, HDFS-12838-HDFS-7240.003.patch,
> HDFS-12838-HDFS-7240.004.patch, HDFS-12838-HDFS-7240.005.patch
>
>
> Currently KeySpaceManager allocates multiple blocks by sending multiple block
> allocation requests over the RPC. This can be optimized to aggregate multiple
> block allocation request over one rpc.
> {code}
> while (requestedSize > 0) {
> long allocateSize = Math.min(scmBlockSize, requestedSize);
> AllocatedBlock allocatedBlock =
> scmBlockClient.allocateBlock(allocateSize, type, factor);
> KsmKeyLocationInfo subKeyInfo = new KsmKeyLocationInfo.Builder()
> .setContainerName(allocatedBlock.getPipeline().getContainerName())
> .setBlockID(allocatedBlock.getKey())
> .setShouldCreateContainer(allocatedBlock.getCreateContainer())
> .setIndex(idx++)
> .setLength(allocateSize)
> .setOffset(0)
> .build();
> locations.add(subKeyInfo);
> requestedSize -= allocateSize;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334683#comment-16334683
]
genericqa commented on HDFS-13024:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 0
unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}213m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| |
[jira] [Updated] (HDFS-13044) RBF: Add a safe mode for the Router
[ https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13044: --- Summary: RBF: Add a safe mode for the Router (was: RBF: Add a new safe mode for the Router) > RBF: Add a safe mode for the Router > --- > > Key: HDFS-13044 > URL: https://issues.apache.org/jira/browse/HDFS-13044 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > > When a Router cannot communicate with the State Store, it should enter into a > safe mode that disallows certain operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13044) RBF: Add a new safe mode for the Router
Íñigo Goiri created HDFS-13044: -- Summary: RBF: Add a new safe mode for the Router Key: HDFS-13044 URL: https://issues.apache.org/jira/browse/HDFS-13044 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri Assignee: Íñigo Goiri When a Router cannot communicate with the State Store, it should enter into a safe mode that disallows certain operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13043) RBF: Expose the state of the Routers in the federation
Íñigo Goiri created HDFS-13043: -- Summary: RBF: Expose the state of the Routers in the federation Key: HDFS-13043 URL: https://issues.apache.org/jira/browse/HDFS-13043 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri Assignee: Íñigo Goiri The Router should expose the state of the other Routers in the federation through a user UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13042) RBF: Heartbeat Router State
Íñigo Goiri created HDFS-13042: -- Summary: RBF: Heartbeat Router State Key: HDFS-13042 URL: https://issues.apache.org/jira/browse/HDFS-13042 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri Assignee: Íñigo Goiri The Router should heartbeat its state to the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12772) RBF: Federation Router State State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12772: --- Summary: RBF: Federation Router State State Store internal API (was: RBF: Track Router states) > RBF: Federation Router State State Store internal API > - > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12772) RBF: Track Router states
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12772: --- Attachment: HDFS-12772.002.patch > RBF: Track Router states > > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334543#comment-16334543 ] Mukul Kumar Singh commented on HDFS-13017: -- Hi [~elek], the v3 patch looks really good to me. Can you please take care of the checkstyle issues?. I am +1 after that. > Block Storage: implement simple iscsi discovery in jscsi server > --- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13017-HDFS-7240.001.patch, > HDFS-13017-HDFS-7240.002.patch, HDFS-13017-HDFS-7240.003.patch > > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334542#comment-16334542
]
genericqa commented on HDFS-13033:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} root in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
7s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} HDFS-10285 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
7s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
8s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
6s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
6s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 36 new + 178 unchanged - 0 fixed = 214 total (was 178) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
7s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
6s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
7s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 5m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907128/HDFS-13033-HDFS-10285-01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d180db8b2c04 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / 0c41c08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22747/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22747/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
[jira] [Commented] (HDFS-13039) StripedBlockReader#createBlockReader leaks socket on IOException
[
https://issues.apache.org/jira/browse/HDFS-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334531#comment-16334531
]
Jason Lowe commented on HDFS-13039:
---
Note that branch-3 should not be a target of commits per the discussion in
common-dev. It was created by mistake and has since been deleted.
> StripedBlockReader#createBlockReader leaks socket on IOException
>
>
> Key: HDFS-13039
> URL: https://issues.apache.org/jira/browse/HDFS-13039
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Critical
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-13039.00.patch, HDFS-13039.01.patch
>
>
> When running EC on one cluster, DataNode has millions of {{CLOSE_WAIT}}
> connections
> {code:java}
> $ grep CLOSE_WAIT lsof.out | wc -l
> 10358700
> // All CLOSW_WAITs belong to the same DataNode process (pid=88527)
> $ grep CLOSE_WAIT lsof.out | awk '{print $2}' | sort | uniq
> 88527
> {code}
> And DN can not open any file / socket, as shown in the log:
> {noformat}
> 2018-01-19 06:47:09,424 WARN io.netty.channel.DefaultChannelPipeline: An
> exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at
> io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:135)
> at
> io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:563)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13038) User with no permission on file is able to run getfacl for that file
[
https://issues.apache.org/jira/browse/HDFS-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334406#comment-16334406
]
Daryn Sharp commented on HDFS-13038:
Don't agree with the patch. There's no formal posix spec for acls. This
patches appears even more restrictive than linux by requiring path read access
instead of possibly parent read access. Linux docs for getfacl state:
{panel:title=PERMISSIONS}
Process with search access to a file (i.e., processes with read access to the
containing directory of a file) are also granted read access to the file's
ACLs. This is analogous to the permissions required for accessing the file mode.
{panel}
This is confusing at best. It implies parent "search access" equals "read
access". "Search access" means execute bit regardless of read bit – which is
what hdfs enforces today. Please test how linux actually works and I'll verify
when time permits.
> User with no permission on file is able to run getfacl for that file
>
>
> Key: HDFS-13038
> URL: https://issues.apache.org/jira/browse/HDFS-13038
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Lokesh Jain
>Priority: Major
> Attachments: HDFS-13038.001.patch
>
>
> Currently any user with EXECUTE permission can run getfacl on a file or
> directory. This Jira adds a check for READ access of user on the inode path.
> {code:java}
> [root@host ~]$ hdfs dfs -copyFromLocal /etc/a.txt /tmp
> [root@host ~]$ hdfs dfs -setfacl -m user:abc:--- /tmp/a.txt
> {code}
> Since user abc does not have read permission on the file 'cat' command throws
> Permission Denied error but getfacl executes normally.
> {code:java}
> [abc@host ~]$ hdfs dfs -cat /tmp/a.txt
> cat: Permission denied: user=abc, access=READ,
> inode="/tmp/a.txt":abc:hdfs:-rw-r--r--
> [abc@host ~]$ hdfs dfs -getfacl /tmp/a.txt
> # file: /tmp/a.txt
> # owner:root
> # group: hdfs
> user::rw-
> user:abc:---
> group::r--
> mask::r--
> other::r--
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[
https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334397#comment-16334397
]
Daryn Sharp commented on HDFS-13040:
I don’t think this looks right. At a minimum, the short user should not be
used or it may cause issues in a multi-realm environment. Using a proxy user
also means you are allowing the client privs to directly access the external
service used by the NN which may not be intended.
The client's credentials are clearly valid or it couldn’t get a RemoteException
from the NN. The client doesn’t transmit credentials, so the server-side doAs
context cannot authenticate to any external service. For it ever to work,
which it shouldn’t, there is a bug causing implicit fallback to the login
user’s credentials. Whether the client principal matches the NN’s principal
should be irrevelant.
The underlying problem needs to be more concisely stated. Let's start with the
exception the NN receives that triggers the long-winded RemoteException to the
client. Presumably the edit log is a REST based? Probably using the buggy
AuthenticatedURL?
BTW, you can actually use minikdc to login 2 distinct principals for a test.
> Kerberized inotify client fails despite kinit properly
> --
>
> Key: HDFS-13040
> URL: https://issues.apache.org/jira/browse/HDFS-13040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
> Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-13040.001.patch, TransactionReader.java
>
>
> This issue is similar to HDFS-10799.
> HDFS-10799 turned out to be a client side issue where client is responsible
> for renewing kerberos ticket actively.
> However we found in a slightly setup even if client has valid Kerberos
> credentials, inotify still fails.
> Suppose client uses principal h...@example.com,
> namenode 1 uses server principal hdfs/nn1.example@example.com
> namenode 2 uses server principal hdfs/nn2.example@example.com
> *After Namenodes starts for longer than kerberos ticket lifetime*, the client
> fails with the following error:
> {noformat}
> 18/01/19 11:23:02 WARN security.UserGroupInformation:
> PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS)
> cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We
> encountered an error reading
> https://nn2.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3,
>
> https://nn1.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3.
> During automatic edit log failover, we noticed that all of the remaining
> edit log streams are shorter than the current one! The best remaining edit
> log ends at transaction 8683, but we thought we could read up to transaction
> 8684. If you continue, metadata will be lost forever!
> at
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
> {noformat}
> Typically if NameNode has an expired Kerberos ticket, the error handling for
> the typical edit log tailing would let NameNode to relogin with its own
> Kerberos principal. However, when inotify uses the same code path to
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334384#comment-16334384
]
genericqa commented on HDFS-13033:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
9s{color} | {color:red} root in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} HDFS-10285 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
7s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
49s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 36 new + 178 unchanged - 0 fixed = 214 total (was 178) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
9s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907128/HDFS-13033-HDFS-10285-01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 25e62fe06f98 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / 0c41c08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22746/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22746/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334379#comment-16334379
]
Rakesh R commented on HDFS-13033:
-
Thanks [~umamaheswararao] for the reviews.
Attached new patch. Here, I've moved commonly used classes for the external and
internal to "org.apache.hadoop.hdfs.server.common.sps" package, does this make
sense. Also, I've created {{TestExternalStoragePolicySatisfier}} test class to
run unit testing. I will rebase my patch once this test class will get
committed via HDFS-13025.
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13033:
Fix Version/s: HDFS-10285
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13033:
Attachment: HDFS-13033-HDFS-10285-01.patch
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13033:
Status: Patch Available (was: Open)
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334364#comment-16334364
]
Lokesh Jain commented on HDFS-13024:
[~msingh] Thanks for updating the patch! +1 on v8 patch.
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch, HDFS-13024-HDFS-7240.003.patch,
> HDFS-13024-HDFS-7240.004.patch, HDFS-13024-HDFS-7240.005.patch,
> HDFS-13024-HDFS-7240.006.patch, HDFS-13024-HDFS-7240.007.patch,
> HDFS-13024-HDFS-7240.008.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12781) After Datanode down, In Namenode UI Datanode tab is throwing warning message.
[
https://issues.apache.org/jira/browse/HDFS-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334359#comment-16334359
]
genericqa commented on HDFS-12781:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12896409/HDFS-12781-001.patch |
| Optional Tests | asflicense shadedclient |
| uname | Linux 51347629bb66 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6e27b20 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 332 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22743/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> After Datanode down, In Namenode UI Datanode tab is throwing warning message.
> -
>
> Key: HDFS-12781
> URL: https://issues.apache.org/jira/browse/HDFS-12781
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HDFS-12781-001.patch
>
>
> Scenario:
> Stop one Datanode
> Refresh or click on the Datanode tab in namenode UI.
> Actual Output:
> ==
> it's throwing the warning message. please find the bellow warning message.
> DataTables warning: table id=table-datanodes - Requested unknown parameter
> '7' for row 2. For more information about this error, please see
> http://datatables.net/tn/4
> Expected Output:
>
> whenever you click on Datanode tab,it should be display the datanodes
> information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13024:
-
Attachment: HDFS-13024-HDFS-7240.008.patch
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch, HDFS-13024-HDFS-7240.003.patch,
> HDFS-13024-HDFS-7240.004.patch, HDFS-13024-HDFS-7240.005.patch,
> HDFS-13024-HDFS-7240.006.patch, HDFS-13024-HDFS-7240.007.patch,
> HDFS-13024-HDFS-7240.008.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13024:
-
Attachment: HDFS-13024-HDFS-7240.007.patch
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch, HDFS-13024-HDFS-7240.003.patch,
> HDFS-13024-HDFS-7240.004.patch, HDFS-13024-HDFS-7240.005.patch,
> HDFS-13024-HDFS-7240.006.patch, HDFS-13024-HDFS-7240.007.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334345#comment-16334345
]
Mukul Kumar Singh commented on HDFS-13024:
--
Thanks for the review [~ljain], I have addressed the review comments in the v7
patch.
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch, HDFS-13024-HDFS-7240.003.patch,
> HDFS-13024-HDFS-7240.004.patch, HDFS-13024-HDFS-7240.005.patch,
> HDFS-13024-HDFS-7240.006.patch, HDFS-13024-HDFS-7240.007.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334323#comment-16334323
]
genericqa commented on HDFS-13024:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 0
unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
39s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}195m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
|
[jira] [Commented] (HDFS-13024) Ozone: ContainerStateMachine should synchronize operations between createContainer op and writeChunk
[
https://issues.apache.org/jira/browse/HDFS-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334313#comment-16334313
]
Lokesh Jain commented on HDFS-13024:
Thanks [~msingh] for updating the patch! The patch looks very good to me. Below
are few minor comments.
# Please change Prealloced to Preallocated in raftSegmentPreallocedSize.
Similarly for config keys defined.
# We could set the stateMachineData in createContainer call to ByteString.EMPTY.
> Ozone: ContainerStateMachine should synchronize operations between
> createContainer op and writeChunk
>
>
> Key: HDFS-13024
> URL: https://issues.apache.org/jira/browse/HDFS-13024
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13024-HDFS-7240.001.patch,
> HDFS-13024-HDFS-7240.002.patch, HDFS-13024-HDFS-7240.003.patch,
> HDFS-13024-HDFS-7240.004.patch, HDFS-13024-HDFS-7240.005.patch,
> HDFS-13024-HDFS-7240.006.patch
>
>
> This issue happens after HDFS-12853. with HDFS-12853, writeChunk op has been
> divided into two stages 1) the log write phase (here the state machine data
> is written) 2) ApplyTransaction.
> With a 3 node ratis ring, ratis leader will append the log entry to its log
> and forward it to its followers. However there is no guarantee on when the
> followers will apply the log to the state machine in {{applyTransaction}}.
> This issue happens in the following order
> 1) Leader accepts create container
> 2) Leader add entries to its logs and forwards to followers
> 3) Followers append the entry to its log and Ack to the raft leader (Please
> note that the transaction still hasn't been applied)
> 4) Leader applies the transaction and now replies
> 5) write chunk call is sent to the Leader
> 6) Leader now forwards the call to the followers
> 7) Followers try to apply the log by calling {{Dispatcher#dispatch}} however
> the create container call in 3) still hasn't been applied
> 8) write chunk call on followers fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12781) After Datanode down, In Namenode UI Datanode tab is throwing warning message.
[ https://issues.apache.org/jira/browse/HDFS-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307695#comment-16307695 ] Brahma Reddy Battula edited comment on HDFS-12781 at 1/22/18 2:09 PM: -- [~raviprak]/[~wheat9] if you get chance, can you please check this..? thanks. was (Author: brahmareddy): [~raviprak] if you get chance, can you please check this..? thanks. > After Datanode down, In Namenode UI Datanode tab is throwing warning message. > - > > Key: HDFS-12781 > URL: https://issues.apache.org/jira/browse/HDFS-12781 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.9.0, 3.0.0-alpha1 >Reporter: Harshakiran Reddy >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12781-001.patch > > > Scenario: > Stop one Datanode > Refresh or click on the Datanode tab in namenode UI. > Actual Output: > == > it's throwing the warning message. please find the bellow warning message. > DataTables warning: table id=table-datanodes - Requested unknown parameter > '7' for row 2. For more information about this error, please see > http://datatables.net/tn/4 > Expected Output: > > whenever you click on Datanode tab,it should be display the datanodes > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334264#comment-16334264 ] Brahma Reddy Battula commented on HDFS-12733: - TestFailures are unrelated. can anyone review the latest patch? > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, performance >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12733-001.patch, HDFS-12733-002.patch, > HDFS-12733-003.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
