[jira] [Commented] (HDFS-10490) Client may never recovery replica after a timeout during sending packet
[
https://issues.apache.org/jira/browse/HDFS-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316213#comment-15316213
]
Hadoop QA commented on HDFS-10490:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 4s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 9s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestFsDatasetCache |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808232/HDFS-10490.patch |
| JIRA Issue | HDFS-10490 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0bfae3682fd4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 106234d |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15658/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15658/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15658/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15658/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Client may never recovery replica after a timeout during sending packet
>
[jira] [Commented] (HDFS-9805) TCP_NODELAY not set before SASL handshake in data transfer pipeline
[
https://issues.apache.org/jira/browse/HDFS-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316210#comment-15316210
]
Gary Helmling commented on HDFS-9805:
-
[~jojochuang], I have a version that adds in the config already. I'll take a
stab at the unit test and post an update tomorrow.
> TCP_NODELAY not set before SASL handshake in data transfer pipeline
> ---
>
> Key: HDFS-9805
> URL: https://issues.apache.org/jira/browse/HDFS-9805
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Attachments: HDFS-9805.002.patch, HDFS-9805.003.patch
>
>
> There are a few places in the DN -> DN block transfer pipeline where
> TCP_NODELAY is not set before doing a SASL handshake:
> * in {{DataNode.DataTransfer::run()}}
> * in {{DataXceiver::replaceBlock()}}
> * in {{DataXceiver::writeBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9805) TCP_NODELAY not set before SASL handshake in data transfer pipeline
[
https://issues.apache.org/jira/browse/HDFS-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316208#comment-15316208
]
Gary Helmling commented on HDFS-9805:
-
[~jojochuang], I have a version that adds in the config already. I'll take a
stab at the unit test and post an update tomorrow.
> TCP_NODELAY not set before SASL handshake in data transfer pipeline
> ---
>
> Key: HDFS-9805
> URL: https://issues.apache.org/jira/browse/HDFS-9805
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Attachments: HDFS-9805.002.patch, HDFS-9805.003.patch
>
>
> There are a few places in the DN -> DN block transfer pipeline where
> TCP_NODELAY is not set before doing a SASL handshake:
> * in {{DataNode.DataTransfer::run()}}
> * in {{DataXceiver::replaceBlock()}}
> * in {{DataXceiver::writeBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7240) Object store in HDFS
[
https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316202#comment-15316202
]
Lei (Eddy) Xu commented on HDFS-7240:
-
Hi, [~anu] Thanks a lot for organize the meeting.
I also have a few questions that are hopefully be answered in the meeting
* Since Ozone is decided to use range partition, how would key / data
distribution achieve balancing from initial state? For example, a user Foo runs
Hive and creates 10GB of data, these data are distributed to up to 6
(containers) DNs?
* Would you explain what is the benefit of recovering failure pipeline by using
a parallel writes to all 3 containers? It is not very clear in the design.
* It seems to me that in the new pipeline in Ozone, there is no multiple
intermediate states for each chunk?
bq. due to immutability of chunks w rite chunk is an idempotent operation
How does ozone differentiate a recover write from a malicious (or buggy)
re-write?
* You mentioned that KMS/SCM separation is for future scalability. Do KMS / SCM
maintains 1:1, 1:n or n:m relationship? Though it is not in this phase. I'd
like to know whether it is considered. Btw, they are also Raft replicated?
* The raft ring / leader is per-container?
* For pipeline, say if we have a pipeline A->B->C, if the data writes
successfully on A->B, and the metadata Raft writes are succeed on B,C, IIUC,
that is a What would be the result for a read request sent to A or C ?
* How to handle split (merge, migrate) container during writes?
* Since container size is determined by the space usage instead of # of keys,
would that result large performance variants on listing operation, because {{#
of DN reached for a list operation = total # of keys / (# of keys per
container)). And # of keys per container is determined by average object size
in the container.
Thanks.
> Object store in HDFS
>
>
> Key: HDFS-7240
> URL: https://issues.apache.org/jira/browse/HDFS-7240
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf,
> ozone_user_v0.pdf
>
>
> This jira proposes to add object store capabilities into HDFS.
> As part of the federation work (HDFS-1052) we separated block storage as a
> generic storage layer. Using the Block Pool abstraction, new kinds of
> namespaces can be built on top of the storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode
> storage, but independent of namespace metadata.
> I will soon update with a detailed design document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10490) Client may never recovery replica after a timeout during sending packet
[
https://issues.apache.org/jira/browse/HDFS-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated HDFS-10490:
-
Status: Patch Available (was: Open)
> Client may never recovery replica after a timeout during sending packet
> ---
>
> Key: HDFS-10490
> URL: https://issues.apache.org/jira/browse/HDFS-10490
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.6.0
>Reporter: He Tianyi
> Attachments: HDFS-10490.patch
>
>
> For newly created replica, a meta file is created in constructor of
> {{BlockReceiver}} (for {{WRITE_BLOCK}} op). Its header will be written lazily
> (buffered in memory first by {{BufferedOutputStream}}).
> If following packets fail to deliver (e.g. in extreme network condition),
> the header may never get flush until closed.
> However, {{BlockReceiver}} will not call close until block receiving is
> finished or exception(s) encountered. Also in extreme network condition, both
> RST & FIN may not deliver in time.
> In this case, if client tries to initiates a {{transferBlock}} to a new
> datanode (in {{addDatanode2ExistingPipeline}}), existing datanode will see an
> empty meta if its {{BlockReceiver}} did not close in time.
> Then, after HDFS-3429, a default {{DataChecksum}} (NULL, 512) will be used
> during transfer. So when client then tries to recover pipeline after
> completely transferred, it may encounter the following exception:
> {noformat}
> java.io.IOException: Client requested checksum DataChecksum(type=CRC32C,
> chunkSize=4096) when appending to an existing block with different chunk
> size: DataChecksum(type=NULL, chunkSize=512)
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.createStreams(ReplicaInPipeline.java:230)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:226)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:798)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:243)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This will repeat, until exhausted by datanode replacement policy.
> Also to note that, with bad luck (like I), 20k clients are all doing this.
> It's to some extend a DDoS attack to NameNode (because of
> getAdditionalDataNode calls).
> I suggest we flush immediately after header is written, preventing anybody
> from seeing empty meta file for avoiding the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10490) Client may never recovery replica after a timeout during sending packet
[
https://issues.apache.org/jira/browse/HDFS-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated HDFS-10490:
-
Attachment: HDFS-10490.patch
> Client may never recovery replica after a timeout during sending packet
> ---
>
> Key: HDFS-10490
> URL: https://issues.apache.org/jira/browse/HDFS-10490
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.6.0
>Reporter: He Tianyi
> Attachments: HDFS-10490.patch
>
>
> For newly created replica, a meta file is created in constructor of
> {{BlockReceiver}} (for {{WRITE_BLOCK}} op). Its header will be written lazily
> (buffered in memory first by {{BufferedOutputStream}}).
> If following packets fail to deliver (e.g. in extreme network condition),
> the header may never get flush until closed.
> However, {{BlockReceiver}} will not call close until block receiving is
> finished or exception(s) encountered. Also in extreme network condition, both
> RST & FIN may not deliver in time.
> In this case, if client tries to initiates a {{transferBlock}} to a new
> datanode (in {{addDatanode2ExistingPipeline}}), existing datanode will see an
> empty meta if its {{BlockReceiver}} did not close in time.
> Then, after HDFS-3429, a default {{DataChecksum}} (NULL, 512) will be used
> during transfer. So when client then tries to recover pipeline after
> completely transferred, it may encounter the following exception:
> {noformat}
> java.io.IOException: Client requested checksum DataChecksum(type=CRC32C,
> chunkSize=4096) when appending to an existing block with different chunk
> size: DataChecksum(type=NULL, chunkSize=512)
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.createStreams(ReplicaInPipeline.java:230)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:226)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:798)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:243)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This will repeat, until exhausted by datanode replacement policy.
> Also to note that, with bad luck (like I), 20k clients are all doing this.
> It's to some extend a DDoS attack to NameNode (because of
> getAdditionalDataNode calls).
> I suggest we flush immediately after header is written, preventing anybody
> from seeing empty meta file for avoiding the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10490) Client may never recovery replica after a timeout during sending packet
He Tianyi created HDFS-10490:
Summary: Client may never recovery replica after a timeout during
sending packet
Key: HDFS-10490
URL: https://issues.apache.org/jira/browse/HDFS-10490
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: He Tianyi
For newly created replica, a meta file is created in constructor of
{{BlockReceiver}} (for {{WRITE_BLOCK}} op). Its header will be written lazily
(buffered in memory first by {{BufferedOutputStream}}).
If following packets fail to deliver (e.g. in extreme network condition), the
header may never get flush until closed.
However, {{BlockReceiver}} will not call close until block receiving is
finished or exception(s) encountered. Also in extreme network condition, both
RST & FIN may not deliver in time.
In this case, if client tries to initiates a {{transferBlock}} to a new
datanode (in {{addDatanode2ExistingPipeline}}), existing datanode will see an
empty meta if its {{BlockReceiver}} did not close in time.
Then, after HDFS-3429, a default {{DataChecksum}} (NULL, 512) will be used
during transfer. So when client then tries to recover pipeline after completely
transferred, it may encounter the following exception:
{noformat}
java.io.IOException: Client requested checksum DataChecksum(type=CRC32C,
chunkSize=4096) when appending to an existing block with different chunk size:
DataChecksum(type=NULL, chunkSize=512)
at
org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.createStreams(ReplicaInPipeline.java:230)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:226)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:798)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:76)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:243)
at java.lang.Thread.run(Thread.java:745)
{noformat}
This will repeat, until exhausted by datanode replacement policy.
Also to note that, with bad luck (like I), 20k clients are all doing this. It's
to some extend a DDoS attack to NameNode (because of getAdditionalDataNode
calls).
I suggest we flush immediately after header is written, preventing anybody from
seeing empty meta file for avoiding the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316096#comment-15316096
]
Hadoop QA commented on HDFS-10488:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-hdfs-project: The patch generated 3 new + 161
unchanged - 1 fixed = 164 total (was 162) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 38s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808220/HDFS-10488.002.patch |
| JIRA Issue | HDFS-10488 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 12e2a27e107e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 106234d |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15657/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15657/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test
[jira] [Updated] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-10488:
Attachment: HDFS-10488.002.patch
New patch version with checkstyle fixes and changes for
TestWebHdfsFileSystemContract test to now expect 777 permission as default
(same as dfs CLI).
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.002.patch, HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10489) Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones
[ https://issues.apache.org/jira/browse/HDFS-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen reassigned HDFS-10489: Assignee: Xiao Chen > Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones > --- > > Key: HDFS-10489 > URL: https://issues.apache.org/jira/browse/HDFS-10489 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > > When working on HADOOP-13155, we > [discussed|https://issues.apache.org/jira/browse/HADOOP-13155?focusedCommentId=15315117=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15315117] > and concluded that we should use the common config key for key provider uri. > We can depreate the dfs. key for 3.0.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10489) Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones
Xiao Chen created HDFS-10489: Summary: Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones Key: HDFS-10489 URL: https://issues.apache.org/jira/browse/HDFS-10489 Project: Hadoop HDFS Issue Type: Improvement Reporter: Xiao Chen Priority: Minor When working on HADOOP-13155, we [discussed|https://issues.apache.org/jira/browse/HADOOP-13155?focusedCommentId=15315117=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15315117] and concluded that we should use the common config key for key provider uri. We can depreate the dfs. key for 3.0.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315941#comment-15315941
]
Hadoop QA commented on HDFS-10488:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-hdfs-project: The patch generated 23 new + 125
unchanged - 0 fixed = 148 total (was 125) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 38s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 3s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808212/HDFS-10488.patch |
| JIRA Issue | HDFS-10488 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e34a3157025f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 106234d |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15656/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
[jira] [Updated] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-10488:
Affects Version/s: 2.6.0
Status: Patch Available (was: Open)
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-10488:
Attachment: HDFS-10488.patch
Attaching initial patch proposal to make these WebHDFS methods consistent with
dfs CLI. Had also added test cases for checking permissions when calling CREATE
and MDKIR methods.
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
Wellington Chevreuil created HDFS-10488:
---
Summary: WebHDFS CREATE and MKDIRS does not follow same rules as
DFS CLI when creating files/directories without specifying permissions
Key: HDFS-10488
URL: https://issues.apache.org/jira/browse/HDFS-10488
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Reporter: Wellington Chevreuil
Priority: Minor
WebHDFS methods for creating file/directories are always creating it with 755
permissions as default, even ignoring any configured
*fs.permissions.umask-mode* in the case of directories.
Dfs CLI, however, applies the configured umask to 777 permission for
directories, or 666 permission for files.
Example below shows the different behaviour when creating directory via CLI and
WebHDFS:
{noformat}
1) Creating a directory under '/test/' as 'test-user'. Configured
fs.permissions.umask-mode is 000:
$ sudo -u test-user hdfs dfs -mkdir /test/test-user1
$ sudo -u test-user hdfs dfs -getfacl /test/test-user1
# file: /test/test-user1
# owner: test-user
# group: supergroup
user::rwx
group::rwx
other::rwx
4) Doing the same via WebHDFS does not get the proper ACLs:
$ curl -i -X PUT
"http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
$ sudo -u test-user hdfs dfs -getfacl /test/test-user2
# file: /test/test-user2
# owner: test-user
# group: supergroup
user::rwx
group::r-x
other::r-x
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7822) Make webhdfs handling of URI standard compliant
[ https://issues.apache.org/jira/browse/HDFS-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7822: -- Target Version/s: (was: 2.8.0) Not much going on here for a long time, dropping from 2.8.0. Not putting any target-version either anymore, let's target this depending on when there is patch activity. > Make webhdfs handling of URI standard compliant > --- > > Key: HDFS-7822 > URL: https://issues.apache.org/jira/browse/HDFS-7822 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Kihwal Lee >Priority: Critical > > As seen in HDFS-7816, webhdfs client is not encoding URI properly. But since > webhdfs is often used as the compatibility layer, we cannot simply fix it and > break the compatibility. Instead, we should stage the fix so that breakages > caused by incompatibility can be minimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5500) Critical datanode threads may terminate silently on uncaught exceptions
[ https://issues.apache.org/jira/browse/HDFS-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-5500: -- Target Version/s: (was: 2.8.0) Not much going on here for a long time, dropping from 2.8.0. Not putting any target-version either anymore, let's target this depending on when there is patch activity. > Critical datanode threads may terminate silently on uncaught exceptions > --- > > Key: HDFS-5500 > URL: https://issues.apache.org/jira/browse/HDFS-5500 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Priority: Critical > > We've seen refreshUsed (DU) thread disappearing on uncaught exceptions. This > can go unnoticed for a long time. If OOM occurs, more things can go wrong. > On one occasion, Timer, multiple refreshUsed and DataXceiverServer thread had > terminated. > DataXceiverServer catches OutOfMemoryError and sleeps for 30 seconds, but I > am not sure it is really helpful. In once case, the thread did it multiple > times then terminated. I suspect another OOM was thrown while in a catch > block. As a result, the server socket was not closed and clients hung on > connect. If it had at least closed the socket, client-side would have been > impacted less. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7607) Use random rack-local node for webhdfs opens to avoid OOM on DNs
[ https://issues.apache.org/jira/browse/HDFS-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7607: -- Target Version/s: (was: 2.8.0) Not much going on here for a long time, dropping from 2.8.0. Not putting any target-version either anymore, let's target this depending on when there is patch activity. > Use random rack-local node for webhdfs opens to avoid OOM on DNs > > > Key: HDFS-7607 > URL: https://issues.apache.org/jira/browse/HDFS-7607 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > Webhdfs currently redirects a client to the DN that physically has one of the > replicas. Unlike the hdfs data streamer protocol which can easily handle > hundreds or thousands of connections, jetty has poor performance under heavy > load. Webhdfs clients can easily overwhelm the DNs and likely cause OOMs or > excessive GC. > The NN should redirect the client to a rack-local location to distribute the > webhdfs load across multiple hosts. The rack can then use the lightweight > streamer protocol. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6358) WebHdfs DN's DFSClient should not use a retry policy
[ https://issues.apache.org/jira/browse/HDFS-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-6358: -- Target Version/s: (was: 2.8.0) Not much going on here for a long time, dropping from 2.8.0. Not putting any target-version either anymore, let's target this depending on when there is patch activity. > WebHdfs DN's DFSClient should not use a retry policy > > > Key: HDFS-6358 > URL: https://issues.apache.org/jira/browse/HDFS-6358 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > DFSClient retries on the DN are useless. The webhdfs client is going to > timeout before the retries complete. The DFSClient will also continue to run > until it timeouts. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6359) WebHdfs NN servlet issues redirects in safemode or standby
[ https://issues.apache.org/jira/browse/HDFS-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-6359: -- Target Version/s: (was: 2.8.0) Not much going on here for a long time, dropping from 2.8.0. Not putting any target-version either anymore, let's target this depending on when there is patch activity. > WebHdfs NN servlet issues redirects in safemode or standby > -- > > Key: HDFS-6359 > URL: https://issues.apache.org/jira/browse/HDFS-6359 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > Webhdfs does not check for safemode or standby during issuing a redirect for > open/create/checksum calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5946) Webhdfs DN choosing code is flawed
[ https://issues.apache.org/jira/browse/HDFS-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-5946: -- Target Version/s: (was: 2.8.0) Not much going on here for a long time, dropping from 2.8.0. Not putting any target-version either anymore, let's target this depending on when there is patch activity. > Webhdfs DN choosing code is flawed > -- > > Key: HDFS-5946 > URL: https://issues.apache.org/jira/browse/HDFS-5946 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, webhdfs >Affects Versions: 2.4.0, 3.0.0-alpha1 >Reporter: Daryn Sharp >Priority: Critical > > HDFS-5891 improved the performance of redirecting webhdfs clients to a DN. > Instead of attempting a connection with a 1-minute timeout, the NN skips > decommissioned nodes. > The logic appears flawed. It finds the index of the first decommissioned > node, if any, then: > * Throws an exception if index = 0, even if other nodes later in the list are > not decommissioned. > * Else picks a random node prior to the index. Let's say there are 10 > replicas, 2nd location is decommissioned. All clients will be redirected to > the first location even though there are 8 other valid locations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
